常連

# 前提・実現したいこと
リクナビNEXTからデータをクローリングしたい

# 発生している問題・エラーメッセージ
start_urlsに定義しているURLが同じレスポンスしか返さない（すべてのURLが同じURLとして扱われる）
```
# 文字数制限によりOverridden settings, Enabled extensions, Enabled downloader middlewares, Enabled spider middlewares省略

2022-06-03 15:46:39 [scrapy.core.engine] INFO: Spider opened
2022-06-03 15:46:40 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-06-03 15:46:40 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024
in start_requests https://next.rikunabi.com/company/cmi1897571005/   # ここは違うURLが入っている
in start_requests https://next.rikunabi.com/company/cmi4040731001/   # ここは違うURLが入っている
2022-06-03 15:46:40 [filelock] DEBUG: Attempting to acquire lock 4661134720 on /Users/xxx/.cache/python-tldextract/3.9.1.final__3.9.1__b4e8ab__tldextract-3.3.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-03 15:46:40 [filelock] DEBUG: Lock 4661134720 acquired on /Users/xxx/.cache/python-tldextract/3.9.1.final__3.9.1__b4e8ab__tldextract-3.3.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-03 15:46:40 [filelock] DEBUG: Attempting to release lock 4661134720 on /Users/xxx/.cache/python-tldextract/3.9.1.final__3.9.1__b4e8ab__tldextract-3.3.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-03 15:46:40 [filelock] DEBUG: Lock 4661134720 released on /Users/xxx/.cache/python-tldextract/3.9.1.final__3.9.1__b4e8ab__tldextract-3.3.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-03 15:46:40 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://point.recruit.co.jp/member/OIDCLogin/?scope=openid+r_email&response_type=code&redirect_uri=https%3A%2F%2Fnext.rikunabi.com%2Frnc%2Fdocs%2Fcp_i01960_r.jsp%3F__m%3D1&state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&prompt=none&client_id=34ad9466433d0a1b5c7fae56090ee605619a5ae452be1b41555da255f2714e54> from <GET https://next.rikunabi.com/company/cmi1897571005/>
2022-06-03 15:46:41 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://point.recruit.co.jp/member/OIDCLogin/?scope=openid+r_email&response_type=code&redirect_uri=https%3A%2F%2Fnext.rikunabi.com%2Frnc%2Fdocs%2Fcp_i01960_r.jsp%3F__m%3D1&state=__f%3D16542388012344027442142945188417%2Csso%3D1&prompt=none&client_id=34ad9466433d0a1b5c7fae56090ee605619a5ae452be1b41555da255f2714e54> from <GET https://next.rikunabi.com/company/cmi4040731001/>
2022-06-03 15:46:42 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?__m=1&state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&error=login_required> from <GET https://point.recruit.co.jp/member/OIDCLogin/?scope=openid+r_email&response_type=code&redirect_uri=https%3A%2F%2Fnext.rikunabi.com%2Frnc%2Fdocs%2Fcp_i01960_r.jsp%3F__m%3D1&state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&prompt=none&client_id=34ad9466433d0a1b5c7fae56090ee605619a5ae452be1b41555da255f2714e54>
2022-06-03 15:46:43 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?__m=1&state=__f%3D16542388012344027442142945188417%2Csso%3D1&error=login_required> from <GET https://point.recruit.co.jp/member/OIDCLogin/?scope=openid+r_email&response_type=code&redirect_uri=https%3A%2F%2Fnext.rikunabi.com%2Frnc%2Fdocs%2Fcp_i01960_r.jsp%3F__m%3D1&state=__f%3D16542388012344027442142945188417%2Csso%3D1&prompt=none&client_id=34ad9466433d0a1b5c7fae56090ee605619a5ae452be1b41555da255f2714e54>
2022-06-03 15:46:44 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&error=login_required> from <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?__m=1&state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&error=login_required>
2022-06-03 15:46:45 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?state=__f%3D16542388012344027442142945188417%2Csso%3D1&error=login_required> from <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?__m=1&state=__f%3D16542388012344027442142945188417%2Csso%3D1&error=login_required>
2022-06-03 15:46:47 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_s01810.jsp?cntct_pnt_cd=001&corp_cd=4040731> from <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?state=__f%3D1654238800231-9162751980079751129%2Csso%3D1&error=login_required>
2022-06-03 15:46:48 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://next.rikunabi.com/rnc/docs/cp_s01810.jsp?cntct_pnt_cd=001&corp_cd=4040731> from <GET https://next.rikunabi.com/rnc/docs/cp_i01960_r.jsp?state=__f%3D16542388012344027442142945188417%2Csso%3D1&error=login_required>
2022-06-03 15:46:49 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://next.rikunabi.com/company/cmi4040731001/> from <GET https://next.rikunabi.com/rnc/docs/cp_s01810.jsp?cntct_pnt_cd=001&corp_cd=4040731>
2022-06-03 15:46:50 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://next.rikunabi.com/company/cmi4040731001/> from <GET https://next.rikunabi.com/rnc/docs/cp_s01810.jsp?cntct_pnt_cd=001&corp_cd=4040731>
2022-06-03 15:46:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://next.rikunabi.com/company/cmi4040731001/> (referer: None)
in parse https://next.rikunabi.com/company/cmi4040731001/    # 1種類のURLのみレスポンスとして返ってくる
2022-06-03 15:46:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://next.rikunabi.com/company/cmi4040731001/> (referer: None)
in parse https://next.rikunabi.com/company/cmi4040731001/    # 1種類のURLのみレスポンスとして返ってくる
2022-06-03 15:46:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://next.rikunabi.com/company/cmi4040731001/> (referer: https://next.rikunabi.com/company/cmi4040731001/)
--->>>parse_com
2022-06-03 15:46:54 [scrapy.core.scraper] DEBUG: Scraped from <200 https://next.rikunabi.com/company/cmi4040731001/>
{クローリングデータ}
2022-06-03 15:46:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://next.rikunabi.com/company/cmi4040731001/> (referer: https://next.rikunabi.com/company/cmi4040731001/)
--->>>parse_com 
2022-06-03 15:46:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://next.rikunabi.com/company/cmi4040731001/>
{クローリングデータ}
2022-06-03 15:46:55 [scrapy.core.engine] INFO: Closing spider (finished)
2022-06-03 15:46:55 [rikunabinext.pipelines] INFO: postgresql+psycopg2://xxx:postgres@localhost:5432/postgres
2022-06-03 15:46:56 [scrapy.extensions.feedexport] INFO: Stored json feed (2 items) in: rikunabinext.json
2022-06-03 15:46:56 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 5677,
 'downloader/request_count': 14,
 'downloader/request_method_count/GET': 14,
 'downloader/response_bytes': 59085,
 'downloader/response_count': 14,
 'downloader/response_status_count/200': 4,
 'downloader/response_status_count/301': 2,
 'downloader/response_status_count/302': 8,
 'elapsed_time_seconds': 16.386866,
 'feedexport/success_count/FileFeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2022, 6, 3, 6, 46, 56, 442776),
 'httpcompression/response_bytes': 226875,
 'httpcompression/response_count': 4,
 'item_scraped_count': 2,
 'log_count/DEBUG': 21,
 'log_count/INFO': 12,
 'memusage/max': 344211456,
 'memusage/startup': 344211456,
 'request_depth_max': 1,
 'response_received_count': 4,
 'scheduler/dequeued': 14,
 'scheduler/dequeued/memory': 14,
 'scheduler/enqueued': 14,
 'scheduler/enqueued/memory': 14,
 'start_time': datetime.datetime(2022, 6, 3, 6, 46, 40, 55910)}
2022-06-03 15:46:56 [scrapy.core.engine] INFO: Spider closed (finished)
```

# 該当のソースコード（概略）
```spider.py
import scrapy

class RikunabiNextSpider(scrapy.Spider):
    name = "rikunabinext_add"

    custom_settings = {
        "ITEM_PIPELINES": {
            # 処理順序の優先度
            "rikunabinext.pipelines.RikunabiNextCrawlerPipeline": 350,
        }
    }

    start_urls = [
    "https://next.rikunabi.com/company/cmi1897571005/",
    "https://next.rikunabi.com/company/cmi4040731001/",
    ]


    def start_requests(self):
        """
        URLに対し順次スクレイピングを行う
        """
        for url in self.start_urls:
            print("in start_requests", url)
            yield scrapy.Request(url=url, callback=self.parse)


    def parse(self, response):
        """
        企業・求人のスクレイピングを行う

        Args:
            response: スクレイピング対象のサイトデータ
        """
        url = response.url
        print("in parse",url)

        # エラー
        if response.status >= 400:
            yield scrapy.Request(url=url, callback=self.error_url)

        # 求人データ
        if "nx1_rq" in url:
            # 通常求人
            yield scrapy.Request(url, callback=self.parse_job)
            return
        if "end_detail" in url:
            # 掲載終了求人
            yield scrapy.Request(url, callback=self.parse_end_job)
            return
        elif "rqmt_id" in url:
            # リクルートエージェントからの求人
            yield scrapy.Request(url, callback=self.parse_ra_job)
            return
        else:
            # 企業データ
            yield scrapy.Request(url, callback=self.parse_com)
            return


    def parse_com(self, response):
        """
        企業データを抽出

        Args:
            response: スクレイピング対象のサイトデータ
        """
        print("--->>>parse_com")
        pass
```

# 自分で調べたことや試したこと
リクナビNEXT以外のサイトのURLを適当に入れたところ通常通りに動作しました（start_urlsの全種類のURLのレスポンスを取得できました）

# 使っているツールのバージョンなど補足情報
Python 3.9.1
Scrapy 2.6.1 

どなたか分かる方いらっしゃったらよろしくお願い致します




	@@ -1 +1 @@
1	- Scrapyでクローリング時にstart_urlsのresponseがすべて同じになる
1	+ ScrapyでリクナビNEXTのクローリング時にstart_urlsのresponseがすべて同じになる

質問編集履歴