やりたいこと

Scrapy の利用検討で、Googleのサイトをスクレイピングしてみています。

Googleにアクセス
Googleについてというリンクを見つける
Googleについてに飛ぶ
プロダクトというリンクを見つける
プロダクトに飛ぶ

困っていること

「Googleについて」のページに飛べません。もしよろしければ、解決の糸口をご教授下さい。
そもそも、スパイダーの使い方を間違えているのでしょうか？

コード

python
1import scrapy
2
3class GoogleSpider(scrapy.Spider):
4    name = 'google'
5    allowed_domains = ['']
6    start_urls = ['https://www.google.com']
7
8    def parse(self, response:scrapy.Selector):
9        self.log("page1")
10        a:scrapy.Selector
11        for a in response.css('a'):
12            try:
13                if "Googleについて" in a.css("::text").get():
14                    self.log("Googleについて")
15                    next_page=response.urljoin(a.attrib['href'])
16                    print(next_page)
17                    scrapy.Request(next_page,callback=self.parse_page2)
18                    break
19            except:
20                print("error")
21        self.log("finish")
22    
23    def parse_page2(self, response):
24        self.log("page2")
25        for a in response.css('a'):
26            if "プロダクト" in a.css("div::text").get():
27                pass
28
29    def log(self,text):
30        f = open('log.txt', 'a')
31        f.write(text+"\n")
32        f.close()

テキトーなLog

log
1page1
2Googleについて
3finish

terminal
1 cd /home/who/ScrapyPE ; /usr/bin/env /home/who/.local/share/virtualenvs/ScrapyPE-1yXy5pp_/bin/python /home/who/.vscode-server/extensions/ms-python.python-2021.4.765268190/pythonFiles/lib/python/debugpy/launcher 35893 -- crawl.py google 
22021-04-26 17:06:58 [scrapy.utils.log] INFO: Scrapy 2.5.0 started (bot: ScrapyPE)
32021-04-26 17:06:58 [scrapy.utils.log] INFO: Versions: lxml 4.6.3.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.2.0, Python 3.9.4 (default, Apr  4 2021, 18:23:51) - [GCC 8.3.0], pyOpenSSL 20.0.1 (OpenSSL 1.1.1k  25 Mar 2021), cryptography 3.4.7, Platform Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-glibc2.28
42021-04-26 17:06:58 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
52021-04-26 17:06:58 [scrapy.crawler] INFO: Overridden settings:
6{'BOT_NAME': 'ScrapyPE',
7 'DOWNLOAD_DELAY': 1,
8 'NEWSPIDER_MODULE': 'ScrapyPE.spiders',
9 'ROBOTSTXT_OBEY': True,
10 'SPIDER_MODULES': ['ScrapyPE.spiders'],
11 'USER_AGENT': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) '
12               'AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 '
13               'Mobile/15E148 Safari/604.1'}
142021-04-26 17:06:58 [scrapy.extensions.telnet] INFO: Telnet Password: c9730deda600e37f
152021-04-26 17:06:58 [scrapy.middleware] INFO: Enabled extensions:
16['scrapy.extensions.corestats.CoreStats',
17 'scrapy.extensions.telnet.TelnetConsole',
18 'scrapy.extensions.memusage.MemoryUsage',
19 'scrapy.extensions.logstats.LogStats']
202021-04-26 17:06:58 [scrapy.middleware] INFO: Enabled downloader middlewares:
21['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
22 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
23 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
24 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
25 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
26 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
27 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
28 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
29 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
30 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
31 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
32 'scrapy.downloadermiddlewares.stats.DownloaderStats']
332021-04-26 17:06:58 [scrapy.middleware] INFO: Enabled spider middlewares:
34['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
35 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
36 'scrapy.spidermiddlewares.referer.RefererMiddleware',
37 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
38 'scrapy.spidermiddlewares.depth.DepthMiddleware']
392021-04-26 17:06:58 [scrapy.middleware] INFO: Enabled item pipelines:
40[]
412021-04-26 17:06:58 [scrapy.core.engine] INFO: Spider opened
422021-04-26 17:06:58 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
432021-04-26 17:06:58 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
442021-04-26 17:06:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.google.com/robots.txt> (referer: None)
452021-04-26 17:07:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.google.com> (referer: None)
46error
47error
48error
49error
50error
51https://about.google/?utm_source=google-JP&utm_medium=referral&utm_campaign=hp-footer&fg=1
522021-04-26 17:07:04 [scrapy.core.engine] INFO: Closing spider (finished)
532021-04-26 17:07:04 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
54{'downloader/request_bytes': 648,
55 'downloader/request_count': 2,
56 'downloader/request_method_count/GET': 2,
57 'downloader/response_bytes': 47881,
58 'downloader/response_count': 2,
59 'downloader/response_status_count/200': 2,
60 'elapsed_time_seconds': 6.347248,
61 'finish_reason': 'finished',
62 'finish_time': datetime.datetime(2021, 4, 26, 8, 7, 4, 946759),
63 'httpcompression/response_bytes': 151383,
64 'httpcompression/response_count': 2,
65 'log_count/DEBUG': 2,
66 'log_count/INFO': 10,
67 'memusage/max': 71077888,
68 'memusage/startup': 71077888,
69 'response_received_count': 2,
70 'robotstxt/request_count': 1,
71 'robotstxt/response_count': 1,
72 'robotstxt/response_status_count/200': 1,
73 'scheduler/dequeued': 1,
74 'scheduler/dequeued/memory': 1,
75 'scheduler/enqueued': 1,
76 'scheduler/enqueued/memory': 1,
77 'start_time': datetime.datetime(2021, 4, 26, 8, 6, 58, 599511)}
782021-04-26 17:07:04 [scrapy.core.engine] INFO: Spider closed (finished)