Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nan in summary histogram #3

Open
Tejuwi opened this issue Oct 30, 2019 · 1 comment
Open

Nan in summary histogram #3

Tejuwi opened this issue Oct 30, 2019 · 1 comment

Comments

@Tejuwi
Copy link

Tejuwi commented Oct 30, 2019

Thanks for the code. I tried to execute it got the following error. can you please tell me why this problem occurred.

WARNING:tensorflow:From :10: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Starting standard services.
INFO:tensorflow:Starting queue runners.
parameter_count = 223104
progress epoch 1 step 50 image/sec 0.6 remaining 5695m
loss [[[[nan]
[nan]
[nan]
.....
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Nan in summary histogram for: decode_3/batchnorm/offset/values
[[node decode_3/batchnorm/offset/values (defined at :12) ]]

@sugandha11993
Copy link

Hi @YaswinTeja Yaswin/ @RaymondByc, I used this code to train the model from scratch using the data given since in the checkpoint file @RaymondByc you have only mentioned the model12345 something which is not present in the repo, so I passed the checkpoint as None and trained using MODE train. As shown above, by Yaswin I also got the loss as nan. Can you please help. @YaswinTeja if you figured it out please share your edits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants