要素数がなぜか減っている

Question

実現したいこと

要因がどこにあるのかを把握したい。

発生している問題・分からないこと

表を作る際にいつの間にか項目ごとの要素数が減っていることに気づいたが、原因がわからない。

エラーメッセージ

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[14], line 3
      1 x = train_val3.loc[ : ,['duration', 'housing_yes', 'campaign', 'contact_sending _document']]
      2 t = train_val3[['y']]
----> 3 s1, s2 = learn(x, t)
      4 print(s1, s2)

Cell In[9], line 12, in learn(x, t)
      9 sc_y_train = sc_model_y.transform(y_train)
     11 model = LinearRegression()
---> 12 model.fit(sc_x_train, sc_y_train)
     13 sc_x_val = sc_model_x.transform(x_val)
     14 sc_y_val = sc_model_y.transform(y_val)

File ~\anaconda3\Lib\site-packages\sklearn\base.py:1389, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1382     estimator._validate_params()
   1384 with config_context(
   1385     skip_parameter_validation=(
   1386         prefer_skip_nested_validation or global_skip_validation
   1387     )
   1388 ):
-> 1389     return fit_method(estimator, *args, **kwargs)

File ~\anaconda3\Lib\site-packages\sklearn\linear_model\_base.py:601, in LinearRegression.fit(self, X, y, sample_weight)
    597 n_jobs_ = self.n_jobs
    599 accept_sparse = False if self.positive else ["csr", "csc", "coo"]
--> 601 X, y = validate_data(
    602     self,
    603     X,
    604     y,
    605     accept_sparse=accept_sparse,
    606     y_numeric=True,
    607     multi_output=True,
    608     force_writeable=True,
    609 )
    611 has_sw = sample_weight is not None
    612 if has_sw:

File ~\anaconda3\Lib\site-packages\sklearn\utils\validation.py:2961, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   2959         y = check_array(y, input_name="y", **check_y_params)
   2960     else:
-> 2961         X, y = check_X_y(X, y, **check_params)
   2962     out = X, y
   2964 if not no_val_X and check_params.get("ensure_2d", True):

File ~\anaconda3\Lib\site-packages\sklearn\utils\validation.py:1389, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1370 X = check_array(
   1371     X,
   1372     accept_sparse=accept_sparse,
   (...)
   1384     input_name="X",
   1385 )
   1387 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
-> 1389 check_consistent_length(X, y)
   1391 return X, y

File ~\anaconda3\Lib\site-packages\sklearn\utils\validation.py:475, in check_consistent_length(*arrays)
    473 uniques = np.unique(lengths)
    474 if len(uniques) > 1:
--> 475     raise ValueError(
    476         "Found input variables with inconsistent numbers of samples: %r"
    477         % [int(l) for l in lengths]
    478     )

ValueError: Found input variables with inconsistent numbers of samples: [17360, 4340]

該当のソースコード

import pandas as pd
from sklearn.linear_model import LinearRegression
df = pd.read_csv('Bank.csv')
df.head(2)
get_dummies = ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'month']
GET_DUMMIES = pd.get_dummies(df[get_dummies], drop_first = True, dtype = int)
df2 = pd.concat([df, GET_DUMMIES], axis = 1)
df2 = df2.drop(get_dummies, axis = 1)
df2.head(2)
from sklearn.model_selection import train_test_split
train_val, test = train_test_split(df2, test_size = 0.2, random_state = 0)
train_val.isnull().sum()
train_val_mean = train_val.mean(numeric_only = True)
train_val2 = train_val.fillna(train_val_mean)
colname = train_val2.columns
for name in colname:
train_val2.plot(kind = 'scatter', x = name, y = 'y')
outline1 = train_val2[(train_val2['duration'] > 1400) & (train_val2['y'] < 0.2)]
outline2 = train_val2[(train_val2['amount'] > 100000) & (train_val2['y'] < 0.2)]
outline3 = train_val2[(train_val2['previous'] > 250) & (train_val2['y'] < 0.2)]
print(outline1, outline2, outline3)
train_val3 = train_val2.drop([3140, 16312], axis = 0)
from sklearn.preprocessing import StandardScaler
def learn(x, t):
x_train, y_train, x_val, y_val = train_test_split(x, t, test_size = 0.2, random_state = 0)
sc_model_x = StandardScaler()
sc_model_y = StandardScaler()
sc_model_x.fit(x_train)
sc_x_train = sc_model_x.transform(x_train)
sc_model_y.fit(y_train)
sc_y_train = sc_model_y.transform(y_train)

model = LinearRegression()
model.fit(sc_x_train, sc_y_train)
sc_x_val = sc_model_x.transform(x_val)
sc_y_val = sc_model_y.transform(y_val)
train_score = model.score(sc_x_train, sc_y_train)
val_score = model.score(sc_x_val, sc_y_train)
return train_score, val_score
train_cor = train_val3.corr()['y']
train_cor
abs_cor = train_cor.map(abs)
abs_cor.sort_values(ascending = False)

x = train_val3.loc[ : ,['duration', 'housing_yes', 'campaign', 'contact_sending _document']]
t = train_val3[['y']]
s1, s2 = learn(x, t)
print(s1, s2)

試したこと・調べたこと

• teratailやGoogle等で検索した
• ソースコードを自分なりに変更した
• 知人に聞いた
• その他

上記の詳細・結果

Google,ChatGPTなどで様々な提案を受けて改善しようとしたのですが、エラーの種類が若干変わるだけで自分の力では何がどうなっているのかすらわかりません。
参考情報としては薄いですが、おそらくネット検索では解決法は見つからないのではないかと思います。

補足

全体のプログラム