Scrapyのpipeline.py内でMySQLdbを使用した際のAttributeError

Question

サイトから記事をクローリング、スクレイピングしてMySQLに格納

ScrapyでクローリングしてMySQLに格納するためにMySQLdbを使用したのですが、AttributeErrorがでてしまいます。。SpiderのソースコードとSetting.pyは以下です。

from scrapy.spiders import SitemapSpider


class WiredjpSpider(SitemapSpider):
    name = "wiredjp"
    allowed_domains = ["wired.jp"]
    sitemap_urls = [
        'https://wired.jp/sitemap.xml',
    ]

    sitemap_follow = [
        r'post-2015-',
    ]

    sitemap_rules = [
        (r'/2015/\d\d/\d\d/', 'parse_post'),
    ]

    def parse_post(self, response):
        yield{
            'title': response.css('h1.post-title::text').extract_first(),
        }

# -*- coding: utf-8 -*-

# Scrapy settings for myproject project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'myproject'

SPIDER_MODULES = ['myproject.spiders']
NEWSPIDER_MODULE = 'myproject.spiders'

ROBOTSTXT_OBEY = True

DOWNLOAD_DELAY = 1

ITEM_PIPELINES = {
    'myproject.pipelines.ValidationPipeline': 300,
    'myproject.pipelines.MySQLPipeline': 800,
}

発生している問題・エラーメッセージ

(menv_scrapy) □□□□@home:/mnt/c/Users/□□□□/python.crawling&scraping/myproject$ scrapy crawl wiredjp
2018-10-21 05:29:15 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: myproject)
2018-10-21 05:29:15 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.7.0 (default, Oct  9 2018, 10:31:47) - [GCC 7.3.0], pyOpenSSL 18.0.0 (OpenSSL 1.0.2p  14 Aug 201
8), cryptography 2.3.1, Platform Linux-4.4.0-17134-Microsoft-x86_64-with-debian-buster-sid
2018-10-21 05:29:15 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'myproject', 'DOWNLOAD_DELAY': 1, 'FEED_EXPORT_ENCODING': 'utf-8', 'NEWSPIDER_MODULE': 'myproject.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['myproject.spiders']}

2018-10-21 05:29:15 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2018-10-21 05:29:15 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-10-21 05:29:15 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-10-21 05:29:15 [scrapy.middleware] INFO: Enabled item pipelines:
['myproject.pipelines.ValidationPipeline', 'myproject.pipelines.MySQLPipeline']
2018-10-21 05:29:15 [scrapy.core.engine] INFO: Spider opened
2018-10-21 05:29:15 [scrapy.core.engine] INFO: Closing spider (shutdown)


ここから↓エラー


2018-10-21 05:29:15 [scrapy.core.engine] ERROR: Scraper close failure
Traceback (most recent call last):
  File "/home/□□□□/.conda/envs/menv_scrapy/lib/python3.7/site-packages/scrapy/crawler.py", line 82, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
AttributeError: 'WiredjpSpider' object has no attribute 'setting'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/□□□□/.conda/envs/menv_scrapy/lib/python3.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/mnt/c/Users/□□□□/python.crawling&scraping/myproject/myproject/pipelines.py", line 43, in close_spider
    self.conn.close()
AttributeError: 'MySQLPipeline' object has no attribute 'conn'
2018-10-21 05:29:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'shutdown',
 'finish_time': datetime.datetime(2018, 10, 20, 20, 29, 15, 747971),
 'log_count/ERROR': 1,
 'log_count/INFO': 6}
2018-10-21 05:29:15 [scrapy.core.engine] INFO: Spider closed (shutdown)
Unhandled error in Deferred:
2018-10-21 05:29:15 [twisted] CRITICAL: Unhandled error in Deferred:

2018-10-21 05:29:15 [twisted] CRITICAL:
Traceback (most recent call last):
  File "/home/□□□□/.conda/envs/menv_scrapy/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/□□□□/.conda/envs/menv_scrapy/lib/python3.7/site-packages/scrapy/crawler.py", line 82, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
AttributeError: 'WiredjpSpider' object has no attribute 'setting'

pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
from scrapy.exceptions import DropItem
import MySQLdb


class MyprojectPipeline(object):
    def process_item(self, item, spider):
        return item


class MySQLPipeline(object):

    def open_spider(self, spider):

        setting = spider.setting
        params = {
            'host': setting.get('MYSQL_HOST', 'localhost'),
            'db': setting.get('MYSQL_DATABASE', 'scraping'),
            'user': setting.get('MYSQL_USER', ''),
            'passwd': setting.get('MYSQL_PASSWORD', ''),
            'charset': setting.get('MYSQL_CHARSET', 'utf8mb4'),
        }

        self.conn = MySQLdb.connect(**params)
        self.c = self.conn.cursor()
        self.c.execute('''
        create table if no exists items (
        id integer not null auto_increment,
        title char(200) not null,
        primary key (id)
        )
        ''')

        self.conn.commit()

    def close_spider(self, spider):

        self.conn.close()

    def process_item(self, item, spider):

        self.c.execute('insert into items (title) values (%(title)s)', dict(item))
        self.conn.commit()
        return item



class ValidationPipeline(object):

    def process_item(self, item, spider):
        if not item['title']:
            raise DropItem('Missing title')

        return item

環境

windows10
WSL：Ubuntu 18.04.1 LTS

Anaconda3-5.3.0
Python3.7

Answer 1

自己解決いたしました。