質問編集履歴
2
GTX1080Ti→GTX1080に修正
test
CHANGED
|
File without changes
|
test
CHANGED
|
@@ -36,7 +36,7 @@
|
|
|
36
36
|
|
|
37
37
|
|
|
38
38
|
|
|
39
|
-
研究室にある同じVRAM8GBのGTX1080
|
|
39
|
+
研究室にある同じVRAM8GBのGTX1080はUbuntuで同じKeras実装のYOLOv3を使っており、これはバッチサイズ32で動作しているので疑問に思い質問させていただきました。プログラミングに関する質問ではないのでTeratailでするべきではなかったら申し訳ないです。
|
|
40
40
|
|
|
41
41
|
|
|
42
42
|
|
1
エラー文章2の追記
test
CHANGED
|
File without changes
|
test
CHANGED
|
@@ -49,3 +49,83 @@
|
|
|
49
49
|
remapper failed: Invalid argument: Subshape must have computed start >= end since stride is negative, but is 0 and 2 (computed from start 0 and end 9223372036854775807 over shape with rank 2 and stride-1)
|
|
50
50
|
|
|
51
51
|
```
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
|
|
55
|
+
他にもエラーでストップする場合があったので追記します。
|
|
56
|
+
|
|
57
|
+
```ErrorMessage2
|
|
58
|
+
|
|
59
|
+
OP_REQUIRES failed at transpose_op.cc:199 : Resource exhausted: OOM when allocating tensor with shape[16,104,104,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
|
|
60
|
+
|
|
61
|
+
Traceback (most recent call last):
|
|
62
|
+
|
|
63
|
+
File "train.py", line 208, in <module>
|
|
64
|
+
|
|
65
|
+
_main()
|
|
66
|
+
|
|
67
|
+
File "train.py", line 102, in _main
|
|
68
|
+
|
|
69
|
+
callbacks=[logging, checkpoint, reduce_lr, early_stopping])
|
|
70
|
+
|
|
71
|
+
File "D:\Users\myusername\anaconda3\envs\yolov3_gpu_2\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
|
|
72
|
+
|
|
73
|
+
return func(*args, **kwargs)
|
|
74
|
+
|
|
75
|
+
File "D:\Users\myusername\anaconda3\envs\yolov3_gpu_2\lib\site-packages\keras\engine\training.py", line 1418, in fit_generator
|
|
76
|
+
|
|
77
|
+
initial_epoch=initial_epoch)
|
|
78
|
+
|
|
79
|
+
File "D:\Users\myusername\anaconda3\envs\yolov3_gpu_2\lib\site-packages\keras\engine\training_generator.py", line 217, in fit_generator
|
|
80
|
+
|
|
81
|
+
class_weight=class_weight)
|
|
82
|
+
|
|
83
|
+
File "D:\Users\myusername\anaconda3\envs\yolov3_gpu_2\lib\site-packages\keras\engine\training.py", line 1217, in train_on_batch
|
|
84
|
+
|
|
85
|
+
outputs = self.train_function(ins)
|
|
86
|
+
|
|
87
|
+
File "D:\Users\myusername\anaconda3\envs\yolov3_gpu_2\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
|
|
88
|
+
|
|
89
|
+
return self._call(inputs)
|
|
90
|
+
|
|
91
|
+
File "D:\Users\myusername\anaconda3\envs\yolov3_gpu_2\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
|
|
92
|
+
|
|
93
|
+
fetched = self._callable_fn(*array_vals)
|
|
94
|
+
|
|
95
|
+
File "D:\Users\myusername\anaconda3\envs\yolov3_gpu_2\lib\site-packages\tensorflow\python\client\session.py", line 1458, in __call__
|
|
96
|
+
|
|
97
|
+
run_metadata_ptr)
|
|
98
|
+
|
|
99
|
+
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
|
|
100
|
+
|
|
101
|
+
(0) Resource exhausted: OOM when allocating tensor with shape[16,104,104,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
|
|
102
|
+
|
|
103
|
+
[[{{node leaky_re_lu_9/LeakyRelu}}]]
|
|
104
|
+
|
|
105
|
+
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
|
|
106
|
+
|
|
107
|
+
|
|
108
|
+
|
|
109
|
+
[[loss_1/add_74/_5295]]
|
|
110
|
+
|
|
111
|
+
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
|
|
112
|
+
|
|
113
|
+
|
|
114
|
+
|
|
115
|
+
(1) Resource exhausted: OOM when allocating tensor with shape[16,104,104,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
|
|
116
|
+
|
|
117
|
+
[[{{node leaky_re_lu_9/LeakyRelu}}]]
|
|
118
|
+
|
|
119
|
+
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
|
|
120
|
+
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
0 successful operations.
|
|
124
|
+
|
|
125
|
+
0 derived errors ignored.
|
|
126
|
+
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
これは以下の記事と同じエラーですね。
|
|
130
|
+
|
|
131
|
+
https://qiita.com/enoughspacefor/items/1c09a27877877c56f25a
|