ValueError: x and y must have same first dimension, but have shapes (200,) and (5,)の解決法

Question

前提・実現したいこと

Resnet50で転移学習行い、モデル作成しGradcamで特徴領域の抽出を試みています。
学習は完了しましたが下記のようなエラーメッセージが表示されます。
解決法などご教示いただけませんでしょうか。よろしくお願いいたします。

発生している問題・エラーメッセージ

 C:\Users\orala\Desktop\new>python train3.py
Using TensorFlow backend.
2021-05-12 10:57:06.498117: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2421
39
Found 2421 images belonging to 2 classes.
Found 39 images belonging to 2 classes.
2021-05-12 10:57:08.799557: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2021-05-12 10:57:08.830175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1660 Ti computeCapability: 7.5
coreClock: 1.59GHz coreCount: 24 deviceMemorySize: 6.00GiB deviceMemoryBandwidth: 268.26GiB/s
2021-05-12 10:57:08.830324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-05-12 10:57:08.836867: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-05-12 10:57:08.840921: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-05-12 10:57:08.842442: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-05-12 10:57:08.847273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-05-12 10:57:08.849772: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-05-12 10:57:08.858227: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-05-12 10:57:08.858429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2021-05-12 10:57:08.858718: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2021-05-12 10:57:08.859551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1660 Ti computeCapability: 7.5
coreClock: 1.59GHz coreCount: 24 deviceMemorySize: 6.00GiB deviceMemoryBandwidth: 268.26GiB/s
2021-05-12 10:57:08.859703: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-05-12 10:57:08.859833: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-05-12 10:57:08.859954: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-05-12 10:57:08.860088: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-05-12 10:57:08.860236: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-05-12 10:57:08.860386: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-05-12 10:57:08.860498: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-05-12 10:57:08.860631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2021-05-12 10:57:09.340333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-05-12 10:57:09.340452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0
2021-05-12 10:57:09.340708: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N
2021-05-12 10:57:09.340914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4755 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
C:\Users\orala\anaconda3\envs\envtest2\lib\site-packages\keras_applications\resnet50.py:265: UserWarning: The output shape of `ResNet50(include_top=False)` has been changed since Keras 2.2.0.
  warnings.warn('The output shape of `ResNet50(include_top=False)` '
train3.py:76: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor("in..., outputs=Tensor("se...)`
  model = Model(input=ResNet50.input, output=top_model(ResNet50.output))
Epoch 1/200
2021-05-12 10:57:27.358295: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-05-12 10:57:27.650637: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-05-12 10:57:28.705641: W tensorflow/stream_executor/gpu/redzone_allocator.cc:312] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation. This message will be only logged once.
2421/2421 [==============================] - 2049s 846ms/step - loss: 8.0491 - accuracy: 0.8244 - val_loss: 0.0262 - val_accuracy: 0.8718
Epoch 2/200
2421/2421 [==============================] - 2158s 891ms/step - loss: 1.9205 - accuracy: 0.8887 - val_loss: 0.0023 - val_accuracy: 0.8205
Epoch 3/200
2421/2421 [==============================] - 2112s 873ms/step - loss: 0.8607 - accuracy: 0.9081 - val_loss: 0.0530 - val_accuracy: 0.9231
Epoch 4/200
2421/2421 [==============================] - 2128s 879ms/step - loss: 0.2938 - accuracy: 0.9360 - val_loss: 0.1249 - val_accuracy: 0.9231
Epoch 5/200
2421/2421 [==============================] - 2131s 880ms/step - loss: 0.2834 - accuracy: 0.9441 - val_loss: 0.5585 - val_accuracy: 0.9231
{'val_loss': [0.026193205267190933, 0.0023010552395135164, 0.05301950126886368, 0.12488315254449844, 0.5584982633590698], 'val_accuracy': [0.8717948794364929, 0.8205128312110901, 0.9230769276618958, 0.9230769276618958, 0.9230769276618958], 'loss': [8.012906371525457, 1.923668423558113, 0.86216264440948, 0.2938869503103723, 0.277316237373691], 'accuracy': [0.82441044, 0.888705, 0.9080679, 0.9359537, 0.9441456]}
Traceback (most recent call last):
  File "train3.py", line 103, in <module>
    plt.plot(range(1,epochs+1),acc,'b',label = 'traning accracy')
  File "C:\Users\orala\anaconda3\envs\envtest2\lib\site-packages\matplotlib\pyplot.py", line 2796, in plot
    is not None else {}), **kwargs)
  File "C:\Users\orala\anaconda3\envs\envtest2\lib\site-packages\matplotlib\axes\_axes.py", line 1665, in plot
    lines = [*self._get_lines(*args, data=data, **kwargs)]
  File "C:\Users\orala\anaconda3\envs\envtest2\lib\site-packages\matplotlib\axes\_base.py", line 225, in __call__
    yield from self._plot_args(this, kwargs)
  File "C:\Users\orala\anaconda3\envs\envtest2\lib\site-packages\matplotlib\axes\_base.py", line 391, in _plot_args
    x, y = self._xy_from_xy(x, y)
  File "C:\Users\orala\anaconda3\envs\envtest2\lib\site-packages\matplotlib\axes\_base.py", line 270, in _xy_from_xy
    "have shapes {} and {}".format(x.shape, y.shape))
ValueError: x and y must have same first dimension, but have shapes (200,) and (5,)

該当のソースコード

import os
import random
import glob
import matplotlib.pyplot as plt 
from keras.preprocessing.image import ImageDataGenerator
from keras.applications.resnet50 import ResNet50
from keras.models import Sequential, Model
from keras.layers import Input, Flatten, Dense
from keras import optimizers
from keras.callbacks import ModelCheckpoint, EarlyStopping
from keras import backend


classes = ['normal','others']
nb_classes = len(classes)

#train val dir
train_dir = './dataset/train'
val_dir = './dataset/valid'
model_dir = './saved_model'

#num samples
train_samples = glob.glob(train_dir + '/*/*.jpg')
val_samples = glob.glob(val_dir + '/*/*.jpg')
train_samples = len(train_samples)
val_samples = len(val_samples)
print(train_samples)
print(val_samples)

#img size
img_w, img_h = 224,224

#keras image data generator
train_datagen = ImageDataGenerator(rescale = 1.0/255, zoom_range=0.2,horizontal_flip=True)
val_datagen = ImageDataGenerator(rescale=1.0 / 255)

train_generator = train_datagen.flow_from_directory(
    train_dir,
    target_size=(img_w,img_h),
    color_mode = 'rgb',
    classes = classes,
    class_mode = 'categorical',
    batch_size = 5
)

val_generator = val_datagen.flow_from_directory(
    val_dir,
    target_size=(img_w,img_h),
    color_mode = 'rgb',
    classes = classes,
    class_mode = 'categorical',
    batch_size = 5
)


#train 
epoch = 200
history = model.fit_generator(
    train_generator,
    steps_per_epoch=train_samples,
    epochs=epoch,
    validation_data=val_generator,
    callbacks=[checkpoint,early_stopping]
)
print(history.history)

#plot 
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = epoch

plt.figure()
plt.plot(range(1,epochs+1),acc,'b',label = 'traning accracy')
plt.plot(range(1,epochs+1),val_acc,'r',label='validation accracy')
plt.title('Training and validation accuracy')
plt.legend()
plt.savefig('result_acc')


plt.figure()
plt.plot(range(1,epochs+1), loss, 'bo', label='Training loss')
plt.plot(range(1,epochs+1), val_loss, 'ro', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
plt.savefig('result_loss')

plt.show()

Answer 1

直接のエラーの原因は、EarlyStoppingが機能してepochが5回で終了したため、以下のrange(1,epochs+1)=サイズ200（当初予定のepoch回数）、と、acc=サイズ5（実際のepoch回数）が食い違ったためです。

epochs = epoch

plt.figure()
plt.plot(range(1,epochs+1),acc,'b',label = 'traning accracy')
plt.plot(range(1,epochs+1),val_acc,'r',label='validation accracy')

epochs = len(acc) などとすれば、エラーは発生しないでしょう。
もしくは、以下のように、plotのxパラメータを省略してしまってもよいでしょう。

plt.plot(acc,'b',label = 'traning accracy')
plt.plot(val_acc,'r',label='validation accracy')

上記に加え、EarlyStoppingとModelCheckpointが定義されているにも関わらず、コードが書かれていない（そのためデフォルト動作になっている）のが気になります。

また、生Kerasを使うのはトラブルの元ですのでやめましょう。

これらは別の話題になりますので、質問ありましたら新規にお願いします。

Comments

[shujiu]

ご返信ありがとうございました。
200、5という数字が何を指すのかが全く理解できていませんでした。
お恥ずかしいことですが、Early stoppingとmodel checkpoint の記載がない点に関しては文字数の関係で一部コードを省略しなければならず、自己判断でこの部分が今回の質問には関係ないと思い込み除外して質問していたというところです。epochs = len(acc)と変更したところ、errorなく動きました。