Jupyterでtensorflowを使用し，CNNのモデルfit時にカーネルが異常終了

Question

機械学習初心者です．JypyterでTensorflow(keras)を用いCNNを構成し，fittingのコードを実行させたタイミングでカーネルが異常終了してしまいます．

今回はjupyterでは結果の出力枠にWarningなど表示せず，ウィンドウにて以下メッセージを表示してお亡くなりになります．

カーネルは再起動中です
カーネルが異常終了した様です。自動的に再起動します。

これだけだと原因は分からなかったので，スクリプトにおこして実行すると，今度は次のエラーが出ます

2021-01-01 16:20:37.030154: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.
2021-01-01 16:20:37.030423: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 8. Tune using inter_op_parallelism_threads for best performance.
Train on 22967 samples, validate on 5742 samples
Epoch 1/50
python(14456,0x7000076fc000) malloc: Incorrect checksum for freed object 0x7fed99d3d868: probably modified after being freed.
Corrupt value: 0x10400076fa040
python(14456,0x7000076fc000) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: abort      python test.py

mallocなどを調べてみたのですが解決策にはいたらず．．．

コードは以下になります．

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import matplotlib.image as mpimg

import tensorflow as tf
from tensorflow import keras
from sklearn.model_selection import train_test_split

import os
os.environ['KMP_DUPLICATE_LIB_OK']='TRUE'


label_map = ['Anger', 'Disgust', 'Fear', 'Happy', 'Sad', 'Surprise', 'Neutral']
names_train=['emotion','pixels']
names_test=['pixels']
df_test=pd.read_csv('./hoge/test.csv',names=names_test, na_filter=False)
df_test=df_test.drop([0],axis=0)
df_train=pd.read_csv('./hoge/train.csv',names=names_train, na_filter=False)
df_train=df_train.drop([0],axis=0)


def gray_to_rgb(im):
  w, h = im.shape
  ret = np.empty((w, h, 3), dtype=np.uint8)
  ret[:, :, 2] =  ret[:, :, 1] =  ret[:, :, 0] =  im
  return ret

def convert_to_image(pixels, mode="save", t="gray"):
  if type(pixels) == str:
      pixels = np.array([int(i) for i in pixels.split()])
  if mode == "show":
    if t == "gray":
      return pixels.reshape(48,48)
    else:
      return gray_to_rgb(pixels.reshape(48,48))
  else:
      return pixels


df_train["pixels"] = df_train["pixels"].apply(lambda x : convert_to_image(x, mode="show", t="gray"))
df_test["pixels"] = df_test["pixels"].apply(lambda x : convert_to_image(x, mode="show", t="gray"))


X_train, X_val, y_train, y_val = train_test_split(df_train["pixels"],  df_train["emotion"], test_size=0.2, random_state=1)

X_train = np.array(list(X_train[:]), dtype=np.float)
X_val = np.array(list(X_val[:]), dtype=np.float)

y_train = np.array(list(y_train[:]), dtype=np.float)
y_val = np.array(list(y_val[:]), dtype=np.float)

X_train = X_train.reshape(X_train.shape[0], 48, 48, 1) 
X_val = X_val.reshape(X_val.shape[0], 48, 48, 1)


X_test=np.array(list(df_test['pixels']), dtype=np.float)


X_test = X_test.reshape(X_test.shape[0], 48, 48, 1)

IMG_SIZE=48

model = keras.models.Sequential([
    keras.layers.Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(IMG_SIZE,IMG_SIZE,1)),
    keras.layers.BatchNormalization(axis=1),
    keras.layers.MaxPooling2D(pool_size=(2, 2)),

    keras.layers.Conv2D(64, kernel_size=(3, 3), activation='relu'),
    keras.layers.BatchNormalization(axis=1),
    keras.layers.MaxPooling2D(pool_size=(2, 2)),

    keras.layers.Conv2D(64, kernel_size=(3, 3), activation='relu'),
    keras.layers.BatchNormalization(axis=1),
    keras.layers.MaxPooling2D(pool_size=(2, 2)),

    keras.layers.Conv2D(32, kernel_size=(3, 3), activation='relu'),
    keras.layers.BatchNormalization(axis=1),
    keras.layers.MaxPooling2D(pool_size=(2, 2)),

    keras.layers.Flatten(),
    keras.layers.Dense(512, activation='relu'),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(512, activation='relu'),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(7, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

model.fit(X_train,y_train,epochs=50,batch_size=64,validation_data=(X_val,y_val))

初歩的な質問で申し訳ありませんが，よろしくお願いいたします．

Comments

[jbpb0]

コードが無いと、他人には何が起きてるか分かりません

[meg_]

メモリは足りていますか？

[velonica74]

すみません．修正しました．

[jbpb0]

csvはこれ？
https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data

[jbpb0]

https://stackoverflow.com/questions/41179495/encountering-a-malloc-incorrect-checksum-for-freed-object-error-in-tensorflow
に、メモリーが少ないPCで同様な現象が起きた、と書かれてます

model.fit()で、batch_size=2 で実行してみてください
もしそれでエラーが出なくなったら、数字を4, 8, 16... と2倍ずつ変えてみて、どこまで増やせるか確認してください

Answer 1

conda installの環境(2.0.0)からpipの環境(2.3.1)に変えて稼働させてみたところ，無事動作しました．ご回答くださった皆さま，ありがとうございました．