Vocabulary not fitted or providedエラー

Question

自然言語処理データセットfetch_20newsgroupsから学習させたモデルをpickle保存した。pickle loadして文章を学習に基づいて分類させようとしたが、Vocabulary not fitted or providedというエラー発生。
以下エラーメッセージ全行です。
Traceback (most recent call last):

File "<ipython-input-2-694698f5959c>", line 1, in <module>
runfile('C:/pyfile3/AI_nlp_categories3.py', wdir='C:/pyfile3')

File "C:\Users\BDS001\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)

File "C:\Users\BDS001\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/pyfile3/AI_nlp_categories3.py", line 28, in <module>
nlpcat.model_run(input_data)

File "C:/pyfile3/AI_nlp_categories3.py", line 13, in model_run
input_tc=count_vectorizer.transform(data)

File "C:\Users\BDS001\Anaconda3\lib\site-packages\sklearn\feature_extraction\text.py", line 1267, in transform
self._check_vocabulary()

File "C:\Users\BDS001\Anaconda3\lib\site-packages\sklearn\feature_extraction\text.py", line 490, in _check_vocabulary
raise NotFittedError("Vocabulary not fitted or provided")

NotFittedError: Vocabulary not fitted or provided

＃学習プログラム
from sklearn.datasets import fetch_20newsgroups
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer
tfidf=TfidfTransformer()
count_vectorizer=CountVectorizer()
import pickle

class Nlp_categories:
    def get_train_data(self,categories_):
        training_data=fetch_20newsgroups(subset="train",
                                         categories=categories_.keys(),
                                         shuffle=True,random_state=5)
        self.categ=categories_
        self.training_data=training_data
    def count_terms(self):
        train_tc=count_vectorizer.fit_transform(self.training_data.data)
        print("Dimensions of training data;",train_tc.shape)
        self.train_tc=train_tc
    def train_data(self):
        train_tfidf=tfidf.fit_transform(self.train_tc)
        classifier=MultinomialNB().fit(train_tfidf,self.training_data.target)
        self.classifier=classifier
    def file_train_data(self,model_name):
        with open(model_name,mode='wb') as mfs:
            pickle.dump(self.classifier,mfs)
        
if __name__ == "__main__":

    category_map={"talk.politics.misc":"Politics","rec.autos":"Autos",
                 "rec.sport.hockey":"Hockey","sci.electronics":"Electronics",
                 "sci.med":"Medicine"} コード
    
    category_model="C:/pyfile3/model_nlp_categories.pkl"
    
    nlp_cat=Nlp_categories()
    nlp_cat.get_train_data(category_map)
    nlp_cat.count_terms()   
    nlp_cat.train_data()
    nlp_cat.file_train_data(category_model)

#pickle loadプログラム
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer
tfidf=TfidfTransformer()
count_vectorizer=CountVectorizer()
import pickle

class NlpCategoriesModelRun:
    def model_load(self,model_file):
        with open(model_file,mode="rb") as mfl:
            self.analyzer_model=pickle.load(mfl)
    def model_run(self,data): 
        input_tc=count_vectorizer.transform(data)
        input_tfidf=tfidf.transform(input_tc)
        predictions=self.analyzer_model.predict(input_tfidf)
      
if __name__ == "__main__":

    model_name="C:/pyfile3/model_nlp_categories.pkl"
    
    input_data=[
            "A lot of devices can operate wireless",
            "Political debates help us understand the perspectives of both sides"
            ]
    
    nlpcat=NlpCategoriesModelRun()
    nlpcat.model_load(model_name)
    nlpcat.model_run(input_data)

Comments

[kabayan55]

何行目に問題があるのか知りたいので、エラー文を全て貼っていただけますか？

[oono]

以下エラーメッセージ全行です。
Traceback (most recent call last):

File "<ipython-input-2-694698f5959c>", line 1, in <module>
runfile('C:/pyfile3/AI_nlp_categories3.py', wdir='C:/pyfile3')

File "C:\Users\BDS001\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)

File "C:\Users\BDS001\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/pyfile3/AI_nlp_categories3.py", line 28, in <module>
nlpcat.model_run(input_data)

File "C:/pyfile3/AI_nlp_categories3.py", line 13, in model_run
input_tc=count_vectorizer.transform(data)

File "C:\Users\BDS001\Anaconda3\lib\site-packages\sklearn\feature_extraction\text.py", line 1267, in transform
self._check_vocabulary()

File "C:\Users\BDS001\Anaconda3\lib\site-packages\sklearn\feature_extraction\text.py", line 490, in _check_vocabulary
raise NotFittedError("Vocabulary not fitted or provided")

NotFittedError: Vocabulary not fitted or provided

[oono]

どうも、以下サイトなぞみると、https://www.reddit.com/r/LanguageTechnology/comments/a67z07/getting_error_on_running_the_trained_machine/
classifierモデルだけではなく、count_vectorizerもpickleしなければならないようですが、それでもうまくいきません。

[kabayan55]

ありがとうございます！質問の方にも追記していただけると、他の回答者の方も答えやすいかなと思います。

NotFittedError: Vocabulary not fitted or provided
の not fitted の方なのではないかと思っているのですが、
.transform を .fit_transform にするとどうですか？
おそらく、count_vectorizerをpickleして読みこまないとfitができていないということなんじゃないかと思います。

[oono]

pickle load側、即ち2番目のプログラムの　.transformを .fit_transformに変えてもみましたが、
ValueError: dimension mismatchとなります。
以下エラー全文
runfile('C:/pyfile3/AI_nlp_categories3.py', wdir='C:/pyfile3')
Traceback (most recent call last):

File "<ipython-input-3-694698f5959c>", line 1, in <module>
runfile('C:/pyfile3/AI_nlp_categories3.py', wdir='C:/pyfile3')

File "C:\Users\06245\AppData\Local\Continuum\anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)

File "C:\Users\06245\AppData\Local\Continuum\anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/pyfile3/AI_nlp_categories3.py", line 28, in <module>
nlpcat.model_run(input_data)

File "C:/pyfile3/AI_nlp_categories3.py", line 15, in model_run
predictions=self.analyzer_model.predict(input_tfidf)

File "C:\Users\06245\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\naive_bayes.py", line 77, in predict
jll = self._joint_log_likelihood(X)

File "C:\Users\06245\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\naive_bayes.py", line 770, in _joint_log_likelihood
return (safe_sparse_dot(X, self.feature_log_prob_.T) +

File "C:\Users\06245\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\extmath.py", line 151, in safe_sparse_dot
ret = a @ b

File "C:\Users\06245\AppData\Local\Continuum\anaconda3\lib\site-packages\scipy\sparse\base.py", line 564, in __matmul__
return self.__mul__(other)

File "C:\Users\06245\AppData\Local\Continuum\anaconda3\lib\site-packages\scipy\sparse\base.py", line 520, in __mul__
raise ValueError('dimension mismatch')

ValueError: dimension mismatch

Answer 1

classifier,count_vectorizer,tfidfの3つをpickleして、
pickle load側のプログラムにて、
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer
をimportせず、pickleしたcount_vectorizerとtfidfをclassifierと共にloadしました。
上手く動きました。
からくりはよく理解できていませんが、pickleは学習モデルだけでなく、TfidfTransformer、CountVectorizerも必要のようです。