前提
Keras-RLのExampleを実行しようとして、以下のエラーに遭遇しました。
表示されたメッセージ「CUBLAS_STATUS_ALLOC_FAILED」から、原因はCuBlasのインストールミスにあるのではないかと考えているのですが、いかがでしょうか。
(GPUメモリ不足も疑い、nvidia-smiを実行しGPUメモリを使用しているプロセスを確認して終了させ、Exampleを再度実行してみました。
しかし、同じメッセージが表示され実行は失敗しました。
GPUメモリ使用量をタスクマネージャーで確認したところ、上限まで使用していませんでした。)
環境は、
Windows10 Home GTX1070 → device:0 Python3.6.6 TensorFlow 1.8.0 keras 2.2.0 keras-RL コミットID: de6a3c25b300e1fba5ceff71f319864661b7507d 時点のもの
となっております。
実際の実行結果を以下に示します。
cmd
1 2(tensorflow) C:\Users\name_\Downloads\work\keras-rl>python examples/dqn_cartpole.py 3Using TensorFlow backend. 4[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.[0m 5_________________________________________________________________ 6Layer (type) Output Shape Param # 7================================================================= 8flatten_1 (Flatten) (None, 4) 0 9_________________________________________________________________ 10dense_1 (Dense) (None, 16) 80 11_________________________________________________________________ 12activation_1 (Activation) (None, 16) 0 13_________________________________________________________________ 14dense_2 (Dense) (None, 16) 272 15_________________________________________________________________ 16activation_2 (Activation) (None, 16) 0 17_________________________________________________________________ 18dense_3 (Dense) (None, 16) 272 19_________________________________________________________________ 20activation_3 (Activation) (None, 16) 0 21_________________________________________________________________ 22dense_4 (Dense) (None, 2) 34 23_________________________________________________________________ 24activation_4 (Activation) (None, 2) 0 25================================================================= 26Total params: 658 27Trainable params: 658 28Non-trainable params: 0 29_________________________________________________________________ 30None 312018-08-05 11:49:18.533194: I C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2 322018-08-05 11:49:18.729365: I C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties: 33name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.683 34pciBusID: 0000:01:00.0 35totalMemory: 8.00GiB freeMemory: 6.63GiB 362018-08-05 11:49:18.735626: I C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0 372018-08-05 11:49:19.400571: I C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: 382018-08-05 11:49:19.403286: I C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0 392018-08-05 11:49:19.405132: I C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N 402018-08-05 11:49:19.407133: I C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6400 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1) 41Training for 50000 steps ... 422018-08-05 11:49:23.103885: E C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\stream_executor\cuda\cuda_blas.cc:462] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 432018-08-05 11:49:23.109865: E C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\stream_executor\cuda\cuda_blas.cc:462] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 442018-08-05 11:49:23.115245: E C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\stream_executor\cuda\cuda_blas.cc:462] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 452018-08-05 11:49:23.118322: E C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\stream_executor\cuda\cuda_blas.cc:462] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 462018-08-05 11:49:23.121745: E C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\stream_executor\cuda\cuda_blas.cc:462] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 472018-08-05 11:49:23.126388: W C:\users\nwani\_bazel_nwani\mmtm6wb6\execroot\org_tensorflow\tensorflow\stream_executor\stream.cc:2001] attempting to perform BLAS operation using StreamExecutor without BLAS support 48Traceback (most recent call last): 49 File "examples/dqn_cartpole.py", line 46, in <module> 50 dqn.fit(env, nb_steps=50000, visualize=True, verbose=2) 51 File "C:\Users\name_\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\rl\core.py", line 169, in fit 52 action = self.forward(observation) 53 File "C:\Users\name_\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\rl\agents\dqn.py", line 228, in forward 54 q_values = self.compute_q_values(state) 55 File "C:\Users\name_\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\rl\agents\dqn.py", line 69, in compute_q_values 56 q_values = self.compute_batch_q_values([state]).flatten() 57 File "C:\Users\name_\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\rl\agents\dqn.py", line 64, in compute_batch_q_values 58 q_values = self.model.predict_on_batch(batch) 59 File "C:\Users\name_\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\keras\engine\training.py", line 1272, in predict_on_batch 60 outputs = self.predict_function(ins) 61 File "C:\Users\name_\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 2666, in __call__ 62 return self._call(inputs) 63 File "C:\Users\name_\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 2636, in _call 64 fetched = self._callable_fn(*array_vals) 65 File "C:\Users\name_\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1454, in __call__ 66 self._session._session, self._handle, args, status, None) 67 File "C:\Users\name_\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 519, in __exit__ 68 c_api.TF_GetCode(self.status.status)) 69tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(1, 4), b.shape=(4, 16), m=1, n=16, k=4 70 [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](flatten_1/Reshape, dense_1/kernel/read)]] 71 [[Node: activation_4/Identity/_113 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_55_activation_4/Identity", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] 72 73(tensorflow) C:\Users\name_\Downloads\work\keras-rl>
以上、よろしくお願いいたします。
あなたの回答
tips
プレビュー