tensorflowでメモリリーク？

症状

ubuntu18.04上で、tensorflowとkerasを用いて縦100横100の画像を学習するニューラルネットワークを作成しようとしています。
データの読み込みまでは何の問題もなく、出来るのですが、いざ学習させようとすると、メモリーの使用量が一気に増え、メモリ使用が著しいプロセスを殺すoomkillerによって強制終了されてしまいます。python側からは特にエラーメッセージは出ていません。

プログラムについての情報

プログラムのバージョン・・・python3.6.9, tensorflow1.14.0, keras2.2.5

インプットさせようとしているデータ・・・100×100(カラー)の画像。バッチサイズは16。

ニューラルネットワークの構造

python
1model = Sequential()
2
3model.add(Dense(3000, activation='sigmoid', input_shape=(30000,)))
4
5model.add(Dense(1000, activation='sigmoid'))
6model.add(Dense(500, activation='sigmoid'))
7model.add(Dense(250, activation='sigmoid'))
8
9model.add(Dense(21, activation='relu'))
10
11model.compile(loss='mean_squared_error', optimizer=Adam(lr=1e-3), metrics=['accuracy'])

model.summary()の結果

python
1Model: "sequential_1"
2_________________________________________________________________
3Layer (type)                 Output Shape              Param #   
4=================================================================
5dense_1 (Dense)              (None, 30000)             900030000 
6_________________________________________________________________
7dense_2 (Dense)              (None, 1000)              30001000  
8_________________________________________________________________
9dense_3 (Dense)              (None, 500)               500500    
10_________________________________________________________________
11dense_4 (Dense)              (None, 250)               125250    
12_________________________________________________________________
13dense_5 (Dense)              (None, 21)                5271      
14=================================================================
15Total params: 930,662,021
16Trainable params: 930,662,021
17Non-trainable params: 0
18_________________________________________________________________
19

パソコン上のメモリ・・・8GB(使用可能なのは実質5GB位)

プロセスがキルされたときのログ

May 10 19:57:57 ****** kernel: [ 6444.801730] JavaFX Applicat invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
May 10 19:57:57 ****** kernel: [ 6444.801733] CPU: 3 PID: 11193 Comm: JavaFX Applicat Tainted: G           OE     5.3.0-46-generic #38~18.04.1-Ubuntu
May 10 19:57:57 ****** kernel: [ 6444.801734] Hardware name: ×××××××××× ×××××××××× ××××××××××
May 10 19:57:57 ****** kernel: [ 6444.801734] Call Trace:
May 10 19:57:57 ****** kernel: [ 6444.801741]  dump_stack+0x6d/0x95
May 10 19:57:57 ****** kernel: [ 6444.801744]  dump_header+0x4f/0x200
May 10 19:57:57 ****** kernel: [ 6444.801745]  oom_kill_process+0xe6/0x120
May 10 19:57:57 ****** kernel: [ 6444.801746]  out_of_memory+0x109/0x510
May 10 19:57:57 ****** kernel: [ 6444.801748]  __alloc_pages_slowpath+0xad1/0xe10
May 10 19:57:57 ****** kernel: [ 6444.801750]  __alloc_pages_nodemask+0x2cd/0x320
May 10 19:57:57 ****** kernel: [ 6444.801752]  alloc_pages_current+0x6a/0xe0
May 10 19:57:57 ****** kernel: [ 6444.801754]  __page_cache_alloc+0x6a/0xa0
May 10 19:57:57 ****** kernel: [ 6444.801755]  pagecache_get_page+0x9c/0x2b0
May 10 19:57:57 ****** kernel: [ 6444.801756]  filemap_fault+0x3a2/0xb00
May 10 19:57:57 ****** kernel: [ 6444.801758]  ? poll_select_finish+0x220/0x220
May 10 19:57:57 ****** kernel: [ 6444.801759]  ? poll_select_finish+0x220/0x220
May 10 19:57:57 ****** kernel: [ 6444.801761]  ? xas_load+0xc/0x80
May 10 19:57:57 ****** kernel: [ 6444.801762]  ? xas_find+0x16f/0x1b0
May 10 19:57:57 ****** kernel: [ 6444.801763]  ? filemap_map_pages+0x18f/0x380
May 10 19:57:57 ****** kernel: [ 6444.801765]  ext4_filemap_fault+0x31/0x44
May 10 19:57:57 ****** kernel: [ 6444.801767]  __do_fault+0x57/0x117
May 10 19:57:57 ****** kernel: [ 6444.801769]  __handle_mm_fault+0xda1/0x1230
May 10 19:57:57 ****** kernel: [ 6444.801770]  handle_mm_fault+0xcb/0x210
May 10 19:57:57 ****** kernel: [ 6444.801773]  __do_page_fault+0x2a1/0x4d0
May 10 19:57:57 ****** kernel: [ 6444.801774]  do_page_fault+0x2c/0xe0
May 10 19:57:57 ****** kernel: [ 6444.801776]  page_fault+0x34/0x40
May 10 19:57:57 ****** kernel: [ 6444.801778] RIP: 0033:0x31eed14bf9
May 10 19:57:57 ****** kernel: [ 6444.801781] Code: Bad RIP value.
May 10 19:57:57 ****** kernel: [ 6444.801782] RSP: 002b:00007fc1287bf720 EFLAGS: 00010293
May 10 19:57:57 ****** kernel: [ 6444.801783] RAX: 0000000000000000 RBX: 00007fc170206c30 RCX: 00000031eed14bf9
May 10 19:57:57 ****** kernel: [ 6444.801784] RDX: 0000000000000010 RSI: 0000000000000003 RDI: 00007fc170206c30
May 10 19:57:57 ****** kernel: [ 6444.801784] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007fc13009ce40
May 10 19:57:57 ****** kernel: [ 6444.801785] R10: 00007fc130300440 R11: 0000000000000293 R12: 0000000000000010
May 10 19:57:57 ****** kernel: [ 6444.801785] R13: 0000000000000010 R14: 00000031f5c5bf50 R15: 0000000000000003
May 10 19:57:57 ****** kernel: [ 6444.801786] Mem-Info:
May 10 19:57:57 ****** kernel: [ 6444.801789] active_anon:1835807 inactive_anon:38759 isolated_anon:0
May 10 19:57:57 ****** kernel: [ 6444.801789]  active_file:48 inactive_file:60 isolated_file:0
May 10 19:57:57 ****** kernel: [ 6444.801789]  unevictable:39081 dirty:35 writeback:0 unstable:0
May 10 19:57:57 ****** kernel: [ 6444.801789]  slab_reclaimable:21539 slab_unreclaimable:16554
May 10 19:57:57 ****** kernel: [ 6444.801789]  mapped:45046 shmem:74778 pagetables:15227 bounce:0
May 10 19:57:57 ****** kernel: [ 6444.801789]  free:25450 free_pcp:98 free_cma:0
May 10 19:57:57 ****** kernel: [ 6444.801791] Node 0 active_anon:7343228kB inactive_anon:155036kB active_file:192kB inactive_file:240kB unevictable:156324kB isolated(anon):0kB isolated(file):0kB mapped:180184kB dirty:140kB writeback:0kB shmem:299112kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
May 10 19:57:57 ****** kernel: [ 6444.801792] Node 0 DMA free:15888kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
May 10 19:57:57 ****** kernel: [ 6444.801794] lowmem_reserve[]: 0 3110 7789 7789 7789
May 10 19:57:57 ****** kernel: [ 6444.801795] Node 0 DMA32 free:45520kB min:26928kB low:33660kB high:40392kB active_anon:3140940kB inactive_anon:1304kB active_file:436kB inactive_file:412kB unevictable:5340kB writepending:0kB present:3293752kB managed:3228216kB mlocked:0kB kernel_stack:32kB pagetables:6196kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
May 10 19:57:57 ****** kernel: [ 6444.801797] lowmem_reserve[]: 0 0 4679 4679 4679
May 10 19:57:57 ****** kernel: [ 6444.801798] Node 0 Normal free:40392kB min:40516kB low:50644kB high:60772kB active_anon:4202252kB inactive_anon:153732kB active_file:572kB inactive_file:336kB unevictable:151156kB writepending:0kB present:4970496kB managed:4799976kB mlocked:13580kB kernel_stack:12368kB pagetables:54712kB bounce:0kB free_pcp:392kB local_pcp:0kB free_cma:0kB
May 10 19:57:57 ****** kernel: [ 6444.801800] lowmem_reserve[]: 0 0 0 0 0
May 10 19:57:57 ****** kernel: [ 6444.801801] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15888kB
May 10 19:57:57 ****** kernel: [ 6444.801806] Node 0 DMA32: 130*4kB (UME) 97*8kB (UME) 216*16kB (UME) 206*32kB (UME) 110*64kB (UME) 46*128kB (UE) 27*256kB (UE) 9*512kB (U) 4*1024kB (UM) 3*2048kB (UM) 0*4096kB = 46032kB
May 10 19:57:57 ****** kernel: [ 6444.801810] Node 0 Normal: 682*4kB (UME) 440*8kB (UME) 546*16kB (UME) 222*32kB (UE) 114*64kB (UME) 46*128kB (UE) 10*256kB (UME) 6*512kB (UME) 0*1024kB 0*2048kB 0*4096kB = 40904kB
May 10 19:57:57 ****** kernel: [ 6444.801815] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
May 10 19:57:57 ****** kernel: [ 6444.801815] 78320 total pagecache pages
May 10 19:57:57 ****** kernel: [ 6444.801816] 0 pages in swap cache
May 10 19:57:57 ****** kernel: [ 6444.801817] Swap cache stats: add 0, delete 0, find 0/0
May 10 19:57:57 ****** kernel: [ 6444.801817] Free swap  = 0kB
May 10 19:57:57 ****** kernel: [ 6444.801817] Total swap = 0kB
May 10 19:57:57 ****** kernel: [ 6444.801818] 2070059 pages RAM
May 10 19:57:57 ****** kernel: [ 6444.801818] 0 pages HighMem/MovableOnly
May 10 19:57:57 ****** kernel: [ 6444.801818] 59035 pages reserved
May 10 19:57:57 ****** kernel: [ 6444.801819] 0 pages cma reserved
May 10 19:57:57 ****** kernel: [ 6444.801819] 0 pages hwpoisoned
May 10 19:57:57 ****** kernel: [ 6444.801819] Tasks state (memory values in pages):
May 10 19:57:57 ****** kernel: [ 6444.801820] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
~(プロセス一覧は省略)~
May 10 19:57:57 ****** kernel: [ 6444.801988] [   3282]  1000  3282  2303067  1394164 12517376        0             0 python3
May 10 19:57:57 ****** kernel: [ 6444.801988] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=python3,pid=3282,uid=1000
May 10 19:57:57 ****** kernel: [ 6444.802036] Out of memory: Killed process 3282 (python3) total-vm:9212268kB, anon-rss:5576656kB, file-rss:0kB, shmem-rss:0kB
May 10 19:57:57 ****** kernel: [ 6444.938021] oom_reaper: reaped process 3282 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

試したこと

バッチサイズを1にしてみたが、効果なし
ニューラルネットワークのニューロンや層の数を減らしてみたが、効果なし

まだまだプログラミング初心者で至らない点や情報不足があるかもしれません。ご指摘いただけるとありがたいです。どうかよろしくお願いします。

fiveHundred

2020/05/10 11:52

- 最初のレイヤーにinput_shapeが入っていないですが、これは意図的なものですか？ - model.compile()の前に「model.summary()」を実行すると、モデルの詳細が見られますが、それはどうなっておりますか？（データの読み込み方次第では、(16, 100, 100, 3000)とかいう膨大なサイズになっていることも考えられます）