ページ遷移を繰り返しスクレイピングしたい

Question

URLが　https://kakaku.com/pricedown/#pc になります。
「デスクトップパソコン」、「Mac デスクトップ」などリンク先の
URL全てをdetail_urls_listに格納されております。

from bs4 import BeautifulSoup
import requests
import pandas as pd
import time

detail_urls_list = ['https://kakaku.com/pc/desktop-pc/ranking_0010/pricedown/', 'https://kakaku.com/pc/mac-desktop-pc/ranking_0019/pricedown/', 'https://kakaku.com/pc/note-pc/ranking_0020/pricedown/', 'https://kakaku.com/pc/mac-note-pc/ranking_0029/pricedown/', 'https://kakaku.com/pc/pda/ranking_0030/pricedown/', 'https://kakaku.com/pc/stick-pc/ranking_V066/pricedown/', 'https://kakaku.com/pc/2in1-tablet/ranking_V076/pricedown/', 'https://kakaku.com/pc/gaming-pc/ranking_V077/pricedown/', 'https://kakaku.com/pc/gaming-note/ranking_V078/pricedown/', 'https://kakaku.com/pc/mobile-note/ranking_V079/pricedown/', 'https://kakaku.com/pc/tablet-case/ranking_0032/pricedown/', 'https://kakaku.com/pc/scanner/ranking_0040/pricedown/', 'https://kakaku.com/pc/card-reader/ranking_0057/pricedown/', 'https://kakaku.com/pc/printer/ranking_0060/pricedown/', 'https://kakaku.com/pc/printer-option/ranking_0061/pricedown/', 'https://kakaku.com/pc/3d-printer/ranking_0062/pricedown/', 'https://kakaku.com/pc/wireless-access-point/ranking_0063/pricedown/', 'https://kakaku.com/pc/wireless-adapter/ranking_0064/pricedown/', 'https://kakaku.com/pc/lan-adapter/ranking_0065/pricedown/', 'https://kakaku.com/pc/network-others/ranking_0066/pricedown/', 'https://kakaku.com/pc/bluetooth-adapter/ranking_0067/pricedown/', 'https://kakaku.com/pc/hub/ranking_0074/pricedown/', 'https://kakaku.com/pc/router/ranking_0075/pricedown/', 'https://kakaku.com/pc/wireless-router/ranking_0077/pricedown/', 'https://kakaku.com/pc/crt-monitor/ranking_0080/pricedown/', 'https://kakaku.com/pc/lcd-monitor/ranking_0085/pricedown/', 'https://kakaku.com/pc/monitor-arm/ranking_0086/pricedown/', 'https://kakaku.com/pc/vr-headset/ranking_0087/pricedown/', 'https://kakaku.com/pc/projector/ranking_0088/pricedown/', 'https://kakaku.com/pc/scan-converter/ranking_0089/pricedown/', 'https://kakaku.com/pc/dvd-drive/ranking_0125/pricedown/', 'https://kakaku.com/pc/blu-ray-drive/ranking_0126/pricedown/', 'https://kakaku.com/pc/mp3-player/ranking_0130/pricedown/', 'https://kakaku.com/pc/mp3-accessories/ranking_0131/pricedown/', 'https://kakaku.com/pc/lcd-film/ranking_0132/pricedown/', 'https://kakaku.com/pc/player-others/ranking_0139/pricedown/', 'https://kakaku.com/pc/ups/ranking_0140/pricedown/', 'https://kakaku.com/pc/note-pc-battery/ranking_0145/pricedown/', 'https://kakaku.com/pc/note-pc-cooler/ranking_0146/pricedown/', 'https://kakaku.com/pc/keyboard/ranking_0150/pricedown/', 'https://kakaku.com/pc/numeric-key/ranking_0151/pricedown/', 'https://kakaku.com/pc/mouse/ranking_0160/pricedown/', 'https://kakaku.com/pc/pen-tablet/ranking_0161/pricedown/', 'https://kakaku.com/pc/headset/ranking_0162/pricedown/', 'https://kakaku.com/pc/pc-speaker/ranking_0170/pricedown/', 'https://kakaku.com/pc/bluetooth-speaker/ranking_0171/pricedown/', 'https://kakaku.com/pc/projector-screen/ranking_0180/pricedown/', 'https://kakaku.com/pc/usb-hub/ranking_0190/pricedown/', 'https://kakaku.com/pc/usb-memory/ranking_0526/pricedown/', 'https://kakaku.com/pc/ic-card-reader/ranking_0528/pricedown/', 'https://kakaku.com/pc/pc-1seg/ranking_0556/pricedown/', 'https://kakaku.com/pc/kvm/ranking_0557/pricedown/', 'https://kakaku.com/pc/dvd-media/ranking_0710/pricedown/', 'https://kakaku.com/pc/blu-ray-media/ranking_0711/pricedown/', 'https://kakaku.com/pc/toner/ranking_0720/pricedown/', 'https://kakaku.com/pc/printer-ink/ranking_0730/pricedown/', 'https://kakaku.com/pc/lan-cable/ranking_0810/pricedown/', 'https://kakaku.com/pc/usb-cable/ranking_0811/pricedown/', 'https://kakaku.com/pc/wda/ranking_7611/pricedown/', 'https://kakaku.com/pc/ink-jet-printer/ranking_V061/pricedown/', 'https://kakaku.com/pc/wireless-mouse/ranking_V062/pricedown/', 'https://kakaku.com/pc/laser-printer/ranking_V063/pricedown/', 'https://kakaku.com/pc/wired-mouse/ranking_V064/pricedown/', 'https://kakaku.com/pc/trackball/ranking_V065/pricedown/', 'https://kakaku.com/pc/gaming-monitor/ranking_V085/pricedown/', 'https://kakaku.com/pc/gaming-keyboard/ranking_V086/pricedown/', 'https://kakaku.com/pc/gaming-mouse/ranking_V087/pricedown/', 'https://kakaku.com/pc/gaming-headset/ranking_V088/pricedown/', 'https://kakaku.com/pc/os-soft/ranking_0310/pricedown/', 'https://kakaku.com/pc/programming-soft/ranking_0312/pricedown/', 'https://kakaku.com/pc/office-soft/ranking_0314/pricedown/', 'https://kakaku.com/pc/business-soft/ranking_0320/pricedown/', 'https://kakaku.com/pc/graphic-soft/ranking_0330/pricedown/', 'https://kakaku.com/pc/clipart-soft/ranking_0332/pricedown/', 'https://kakaku.com/pc/font-soft/ranking_0334/pricedown/', 'https://kakaku.com/pc/multimedia-soft/ranking_0340/pricedown/', 'https://kakaku.com/pc/music-soft/ranking_0342/pricedown/', 'https://kakaku.com/pc/antivirus-soft/ranking_0350/pricedown/', 'https://kakaku.com/pc/utility-soft/ranking_0360/pricedown/', 'https://kakaku.com/pc/backup-soft/ranking_0361/pricedown/', 'https://kakaku.com/pc/greetingcard-soft/ranking_0362/pricedown/', 'https://kakaku.com/pc/dictionary-soft/ranking_0363/pricedown/', 'https://kakaku.com/pc/map-soft/ranking_0364/pricedown/', 'https://kakaku.com/pc/translation-soft/ranking_0365/pricedown/', 'https://kakaku.com/pc/certification-soft/ranking_0366/pricedown/', 'https://kakaku.com/pc/typing-soft/ranking_0367/pricedown/', 'https://kakaku.com/pc/education-soft/ranking_0370/pricedown/', 'https://kakaku.com/pc/account-soft/ranking_3740/pricedown/', 'https://kakaku.com/pc/cad-3d-soft/ranking_3750/pricedown/', 'https://kakaku.com/pc/cpu/ranking_0510/pricedown/', 'https://kakaku.com/pc/cpu-cooler/ranking_0512/pricedown/', 'https://kakaku.com/pc/pc-memory/ranking_0520/pricedown/', 'https://kakaku.com/pc/removable-case/ranking_0529/pricedown/', 'https://kakaku.com/pc/hdd-35inch/ranking_0530/pricedown/', 'https://kakaku.com/pc/hdd-scsi/ranking_0533/pricedown/', 'https://kakaku.com/pc/hdd-18inch/ranking_0535/pricedown/', 'https://kakaku.com/pc/hdd-25inch/ranking_0536/pricedown/', 'https://kakaku.com/pc/ssd/ranking_0537/pricedown/', 'https://kakaku.com/pc/external-hdd/ranking_0538/pricedown/', 'https://kakaku.com/pc/hdd-case/ranking_0539/pricedown/', 'https://kakaku.com/pc/motherboard/ranking_0540/pricedown/', 'https://kakaku.com/pc/videocard/ranking_0550/pricedown/', 'https://kakaku.com/pc/video-capture/ranking_0555/pricedown/', 'https://kakaku.com/pc/sound-card/ranking_0560/pricedown/', 'https://kakaku.com/pc/interface-card/ranking_0567/pricedown/', 'https://kakaku.com/pc/pc-case/ranking_0580/pricedown/', 'https://kakaku.com/pc/case-fan/ranking_0581/pricedown/', 'https://kakaku.com/pc/fan-controller/ranking_0582/pricedown/', 'https://kakaku.com/pc/barebone/ranking_0585/pricedown/', 'https://kakaku.com/pc/power-supply/ranking_0590/pricedown/', 'https://kakaku.com/pc/nas/ranking_7731/pricedown/', 'https://kakaku.com/pc/cpu_intel/ranking_V057/pricedown/', 'https://kakaku.com/pc/cpu_amd/ranking_V058/pricedown/', 'https://kakaku.com/pc/portable-hdd/ranking_V059/pricedown/', 'https://kakaku.com/pc/water_cpu-cooler/ranking_V060/pricedown/', 'https://kakaku.com/pc/portable-ssd/ranking_V090/pricedown/', 'https://kakaku.com#top']

def get_page(next_url):
    r = requests.get(next_url)
    soup = BeautifulSoup(r.text)
    time.sleep(1)
    get_next = soup.find('li',class_ = ('next'))
     
#再帰関数で「次へ」が無くなるまでURLを抽出させてます。
    if get_next != None:
        next_pages2 = get_next.select('a[href]')
        for next_pages in next_pages2:
            next_page = 'https://kakaku.com' + next_pages.get('href')
            print(get_page(next_page))

for detail_url in detail_urls_list[0:2]:
    get_page(detail_url)

#####問題点
動作確認の為「デスクトップパソコン」のみで試しております。
デスクトップパソコンは「次へ」が5回（計6ページ）あります。
ちゃんと5回分動作は行われているようですが、全てNoneになって出力されます。
デスクトップパソコンの2~6ページのURLがちゃんと出力できるようにするには
どうすればよろしいでしょうか？

また、再帰関数以外でリスト内のURLから「次」ページを全て抽出する良い方法
があればご教授頂きたいです。

よろしくお願いします。

Comments

[meg_]

収集したデータの利用目的は何でしょうか？業務利用は禁止されているようなので念のための確認です。

[Zarathustra]

ご指摘ありがとうございます。個人利用になります。

[meg_]

> ちゃんと5回分動作は行われているようですが、全てNoneになって出力されます。
get_page()は値を返さないのでNoneが出力されるのは正常な動作だと思います。

[Zarathustra]

値を返すようにするにはどうすればよろしいでしょうか？

Answer 1

デスクトップパソコンの2~6ページのURLがちゃんと出力できるようにするには

どうすればよろしいでしょうか？

下記でどうでしょうか？

def get_page(next_url):
    r = requests.get(next_url)
    soup = BeautifulSoup(r.text)
    time.sleep(1)
    get_next = soup.find('li',class_ = ('next'))

    #再帰関数で「次へ」が無くなるまでURLを抽出させてます。
    if get_next != None:
        next_pages2 = get_next.select('a[href]')
        for next_pages in next_pages2:
            next_page = 'https://kakaku.com' + next_pages.get('href')
            print(next_page)
            get_page(next_page)