前提・実現したいこと

リクナビNEXTからデータをクローリングしたい

発生している問題・エラーメッセージ

start_urlsに定義しているURLが同じレスポンスしか返さない（すべてのURLが同じURLとして扱われる）

# 文字数制限によりOverridden settings, Enabled extensions, Enabled downloader middlewares, Enabled spider middlewares省略

2022-06-03 15:46:39 [scrapy.core.engine] INFO: Spider opened
2022-06-03 15:46:40 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-06-03 15:46:40 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024
in start_requests https://next.rikunabi.com/company/cmi1897571005/   # ここは違うURLが入っている
in start_requests https://next.rikunabi.com/company/cmi4040731001/   # ここは違うURLが入っている
2022-06-03 15:46:40 [filelock] DEBUG: Attempting to acquire lock 4661134720 on /Users/xxx/.cache/python-tldextract/3.9.1.final__3.9.1__b4e8ab__tldextract-3.3.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-03 15:46:40 [filelock] DEBUG: Lock 4661134720 acquired on /Users/xxx/.cache/python-tldextract/3.9.1.final__3.9.1__b4e8ab__tldextract-3.3.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-03 15:46:40 [filelock] DEBUG: Attempting to release lock 4661134720 on /Users/xxx/.cache/python-tldextract/3.9.1.final__3.9.1__b4e8ab__tldextract-3.3.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-03 15:46:40 [filelock] DEBUG: Lock 4661134720 released on /Users/xxx/.cache/python-tldextract/3.9.1.final__3.9.1__b4e8ab__tldextract-3.3.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-03 15:46:40 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://point.recruit.co.jp/member/OIDCLogin/?scope=openid+r_email&response_type=code&redirect_uri=https%3A%2F%2Fnext.rikunabi.com%2Frnc%2Fdocs%2Fcp_i01960_r.jsp%3F__m%3D1&state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&prompt=none&client_id=34ad9466433d0a1b5c7fae56090ee605619a5ae452be1b41555da255f2714e54> from <GET https://next.rikunabi.com/company/cmi1897571005/>
2022-06-03 15:46:41 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://point.recruit.co.jp/member/OIDCLogin/?scope=openid+r_email&response_type=code&redirect_uri=https%3A%2F%2Fnext.rikunabi.com%2Frnc%2Fdocs%2Fcp_i01960_r.jsp%3F__m%3D1&state=__f%3D16542388012344027442142945188417%2Csso%3D1&prompt=none&client_id=34ad9466433d0a1b5c7fae56090ee605619a5ae452be1b41555da255f2714e54> from <GET https://next.rikunabi.com/company/cmi4040731001/>
2022-06-03 15:46:42 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?__m=1&state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&error=login_required> from <GET https://point.recruit.co.jp/member/OIDCLogin/?scope=openid+r_email&response_type=code&redirect_uri=https%3A%2F%2Fnext.rikunabi.com%2Frnc%2Fdocs%2Fcp_i01960_r.jsp%3F__m%3D1&state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&prompt=none&client_id=34ad9466433d0a1b5c7fae56090ee605619a5ae452be1b41555da255f2714e54>
2022-06-03 15:46:43 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?__m=1&state=__f%3D16542388012344027442142945188417%2Csso%3D1&error=login_required> from <GET https://point.recruit.co.jp/member/OIDCLogin/?scope=openid+r_email&response_type=code&redirect_uri=https%3A%2F%2Fnext.rikunabi.com%2Frnc%2Fdocs%2Fcp_i01960_r.jsp%3F__m%3D1&state=__f%3D16542388012344027442142945188417%2Csso%3D1&prompt=none&client_id=34ad9466433d0a1b5c7fae56090ee605619a5ae452be1b41555da255f2714e54>
2022-06-03 15:46:44 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&error=login_required> from <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?__m=1&state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&error=login_required>
2022-06-03 15:46:45 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?state=__f%3D16542388012344027442142945188417%2Csso%3D1&error=login_required> from <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?__m=1&state=__f%3D16542388012344027442142945188417%2Csso%3D1&error=login_required>
2022-06-03 15:46:47 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_s01810.jsp?cntct_pnt_cd=001&corp_cd=4040731> from <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&error=login_required>
2022-06-03 15:46:48 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_s01810.jsp?cntct_pnt_cd=001&corp_cd=4040731> from <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?state=__f%3D16542388012344027442142945188417%2Csso%3D1&error=login_required>
2022-06-03 15:46:49 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://next.rikunabi.com/company/cmi4040731001/> from <GET https://next.rikunabi.com/rnc/docs/cp_s01810.jsp?cntct_pnt_cd=001&corp_cd=4040731>
2022-06-03 15:46:50 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://next.rikunabi.com/company/cmi4040731001/> from <GET https://next.rikunabi.com/rnc/docs/cp_s01810.jsp?cntct_pnt_cd=001&corp_cd=4040731>
2022-06-03 15:46:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://next.rikunabi.com/company/cmi4040731001/> (referer: None)
in parse https://next.rikunabi.com/company/cmi4040731001/    # 1種類のURLのみレスポンスとして返ってくる
2022-06-03 15:46:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://next.rikunabi.com/company/cmi4040731001/> (referer: None)
in parse https://next.rikunabi.com/company/cmi4040731001/    # 1種類のURLのみレスポンスとして返ってくる
2022-06-03 15:46:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://next.rikunabi.com/company/cmi4040731001/> (referer: https://next.rikunabi.com/company/cmi4040731001/)
--->>>parse_com
2022-06-03 15:46:54 [scrapy.core.scraper] DEBUG: Scraped from <200 https://next.rikunabi.com/company/cmi4040731001/>
{クローリングデータ}
2022-06-03 15:46:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://next.rikunabi.com/company/cmi4040731001/> (referer: https://next.rikunabi.com/company/cmi4040731001/)
--->>>parse_com 
2022-06-03 15:46:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://next.rikunabi.com/company/cmi4040731001/>
{クローリングデータ}
2022-06-03 15:46:55 [scrapy.core.engine] INFO: Closing spider (finished)
2022-06-03 15:46:55 [rikunabinext.pipelines] INFO: postgresql+psycopg2://xxx:postgres@localhost:5432/postgres
2022-06-03 15:46:56 [scrapy.extensions.feedexport] INFO: Stored json feed (2 items) in: rikunabinext.json
2022-06-03 15:46:56 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 5677,
 'downloader/request_count': 14,
 'downloader/request_method_count/GET': 14,
 'downloader/response_bytes': 59085,
 'downloader/response_count': 14,
 'downloader/response_status_count/200': 4,
 'downloader/response_status_count/301': 2,
 'downloader/response_status_count/302': 8,
 'elapsed_time_seconds': 16.386866,
 'feedexport/success_count/FileFeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2022, 6, 3, 6, 46, 56, 442776),
 'httpcompression/response_bytes': 226875,
 'httpcompression/response_count': 4,
 'item_scraped_count': 2,
 'log_count/DEBUG': 21,
 'log_count/INFO': 12,
 'memusage/max': 344211456,
 'memusage/startup': 344211456,
 'request_depth_max': 1,
 'response_received_count': 4,
 'scheduler/dequeued': 14,
 'scheduler/dequeued/memory': 14,
 'scheduler/enqueued': 14,
 'scheduler/enqueued/memory': 14,
 'start_time': datetime.datetime(2022, 6, 3, 6, 46, 40, 55910)}
2022-06-03 15:46:56 [scrapy.core.engine] INFO: Spider closed (finished)

該当のソースコード（概略）

spider.py
1import scrapy
2
3class RikunabiNextSpider(scrapy.Spider):
4    name = "rikunabinext_add"
5
6    custom_settings = {
7        "ITEM_PIPELINES": {
8            # 処理順序の優先度
9            "rikunabinext.pipelines.RikunabiNextCrawlerPipeline": 350,
10        }
11    }
12
13    start_urls = [
14    "https://next.rikunabi.com/company/cmi1897571005/",
15    "https://next.rikunabi.com/company/cmi4040731001/",
16    ]
17
18
19    def start_requests(self):
20        """
21        URLに対し順次スクレイピングを行う
22        """
23        for url in self.start_urls:
24            print("in start_requests", url)
25            yield scrapy.Request(url=url, callback=self.parse)
26
27
28    def parse(self, response):
29        """
30        企業・求人のスクレイピングを行う
31
32        Args:
33            response: スクレイピング対象のサイトデータ
34        """
35        url = response.url
36        print("in parse",url)
37
38        # エラー
39        if response.status >= 400:
40            yield scrapy.Request(url=url, callback=self.error_url)
41
42        # 求人データ
43        if "nx1_rq" in url:
44            # 通常求人
45            yield scrapy.Request(url, callback=self.parse_job)
46            return
47        if "end_detail" in url:
48            # 掲載終了求人
49            yield scrapy.Request(url, callback=self.parse_end_job)
50            return
51        elif "rqmt_id" in url:
52            # リクルートエージェントからの求人
53            yield scrapy.Request(url, callback=self.parse_ra_job)
54            return
55        else:
56            # 企業データ
57            yield scrapy.Request(url, callback=self.parse_com)
58            return
59
60
61    def parse_com(self, response):
62        """
63        企業データを抽出
64
65        Args:
66            response: スクレイピング対象のサイトデータ
67        """
68        print("--->>>parse_com")
69        pass

自分で調べたことや試したこと

リクナビNEXT以外のサイトのURLを適当に入れたところ通常通りに動作しました（start_urlsの全種類のURLのレスポンスを取得できました）

使っているツールのバージョンなど補足情報

Python 3.9.1
Scrapy 2.6.1

どなたか分かる方いらっしゃったらよろしくお願い致します

行動規範の内容に同意します

回答1件

自己解決

302番の多重リダイレクトが問題だったようでspider.pyにhandle_httpstatus_listを追加したらできました！

    start_urls = [...]

    handle_httpstatus_list = [302]

    def start_requests(self):
        ...

この際になぜかparse先でresponse.bodyの中身がなくなっていたので，以下を追加してbodyを取得してresponseを入れ直しました

def parse(self, response):
        req = requests.get(response.url)
        soup = BeautifulSoup(req.content, 'html.parser')
        response = response.replace(body=soup.encode('cp1252'))

投稿2022/06/08 04:37

mu-ro

総合スコア20