Q&A
前提・実現したいこと
GoogleColaboratoryでpix2pixを導入し自作の画像で学習を行っています。
学習時に以下のエラーが発生しました。
7,8エポックあたりまでは何も問題なく進み、突然エラー（学習が中止）が発生しました。
発生している問題・エラーメッセージ
Python
12019-11-02 05:37:41.339379: W tensorflow/core/kernels/queue_base.cc:285] _1_batch/fifo_queue: Skipping cancelled dequeue attempt with queue not closed 2Traceback (most recent call last): 3 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call 4 return fn(*args) 5 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn 6 target_list, run_metadata) 7 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun 8 run_metadata) 9tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. 10 (0) Invalid argument: Nan in summary histogram for: generator/encoder_4/batch_normalization/beta/values 11 [[{{node generator/encoder_4/batch_normalization/beta/values}}]] 12 [[generator/encoder_5/conv2d/bias/read/_670]] 13 (1) Invalid argument: Nan in summary histogram for: generator/encoder_4/batch_normalization/beta/values 14 [[{{node generator/encoder_4/batch_normalization/beta/values}}]] 150 successful operations. 160 derived errors ignored. 17 18During handling of the above exception, another exception occurred: 19 20Traceback (most recent call last): 21 File "pix2pix.py", line 803, in <module> 22 main() 23 File "pix2pix.py", line 769, in main 24 results = sess.run(fetches, options=options, run_metadata=run_metadata) 25 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run 26 run_metadata_ptr) 27 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run 28 feed_dict_tensor, options, run_metadata) 29 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run 30 run_metadata) 31 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call 32 raise type(e)(node_def, op, message) 33tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. 34 (0) Invalid argument: Nan in summary histogram for: generator/encoder_4/batch_normalization/beta/values 35 [[node generator/encoder_4/batch_normalization/beta/values (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] 36 [[generator/encoder_5/conv2d/bias/read/_670]] 37 (1) Invalid argument: Nan in summary histogram for: generator/encoder_4/batch_normalization/beta/values 38 [[node generator/encoder_4/batch_normalization/beta/values (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] 390 successful operations. 400 derived errors ignored. 41 42Original stack trace for 'generator/encoder_4/batch_normalization/beta/values': 43 File "pix2pix.py", line 803, in <module> 44 main() 45 File "pix2pix.py", line 700, in main 46 tf.summary.histogram(var.op.name + "/values", var) 47 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/summary/summary.py", line 179, in histogram 48 tag=tag, values=values, name=scope) 49 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_logging_ops.py", line 329, in histogram_summary 50 "HistogramSummary", tag=tag, values=values, name=name) 51 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper 52 op_def=op_def) 53 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func 54 return func(*args, **kwargs) 55 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op 56 attrs, op_def, compute_device) 57 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal 58 op_def=op_def) 59 File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__ 60 self._traceback = tf_stack.extract_stack() 61
該当のソースコード
学習を開始した処理を示します。
GoogleColaboratory
1!python pix2pix.py --mode train --output_dir transition_train --max_epochs 10 --input_dir transition/train --which_direction BtoA
試したこと
サンプルとして実行したfacadesデータセットでは何もエラー無く学習を終えテストまで実行できました。
補足情報（FW/ツールのバージョンなど）
機械学習に関しては初心者です。
使用したpix2pixを以下に示します。
回答1件
0
ベストアンサー
Qiitaの記事 TensorFlowで「Nan in summary histogram for:」というエラーに遭遇した時 に書かれている状況に似ている気がします。
記事に書かれている「costの値を印刷してみて、無限大に向かっているようであれば、学習率を下げてみて」という助言に従ってみてはどうでしょうか？
投稿2019/11/02 05:57
総合スコア6915
あなたの回答
tips
プレビュー
バッドをするには、ログインかつ
こちらの条件を満たす必要があります。
2019/11/02 13:09 編集