スクレイピングにてurlを取得しました。
取得後のurlに重複が生じているので、重複を無くしたいのですが、urlとして残したまま、重複を消したいです。
全体のコード
from bs4 import BeautifulSoup import requests html= requests.get("https://www.springfieldspringfield.co.uk/latest-movie-scripts.php?page=1").text soup=BeautifulSoup(html,"html.parser") links=soup.find_all("a") urls=[] for link in links: url=link.get("href") if "/movie_script.php?movie" in url : urls.append ("https://www.springfieldspringfield.co.uk"+url) for url in urls: print(url)
現状
https://www.springfieldspringfield.co.uk/movie_script.php?movie=cedric-the-entertainer-live-from-the-ville https://www.springfieldspringfield.co.uk/movie_script.php?movie=cedric-the-entertainer-live-from-the-ville https://www.springfieldspringfield.co.uk/movie_script.php?movie=im-brent-morin https://www.springfieldspringfield.co.uk/movie_script.php?movie=im-brent-morin https://www.springfieldspringfield.co.uk/movie_script.php?movie=chelsea-peretti-one-of-the-greats https://www.springfieldspringfield.co.uk/movie_script.php?movie=chelsea-peretti-one-of-the-greats 、、、
と言った形で2つずつ重複したurlがある
求めている結果
https://www.springfieldspringfield.co.uk/movie_script.php?movie=cedric-the-entertainer-live-from-the-ville https://www.springfieldspringfield.co.uk/movie_script.php?movie=im-brent-morin https://www.springfieldspringfield.co.uk/movie_script.php?movie=chelsea-peretti-one-of-the-greats
試したこと
dict.fromkey()の使用
for url in urls: l=[url] l_unique_order = list(dict.fromkeys(l)) print(l_unique_order)
結果
['https://www.springfieldspringfield.co.uk/movie_script.php?movie=cedric-the-entertainer-live-from-the-ville'] ['https://www.springfieldspringfield.co.uk/movie_script.php?movie=cedric-the-entertainer-live-from-the-ville'] ['https://www.springfieldspringfield.co.uk/movie_script.php?movie=im-brent-morin'] ['https://www.springfieldspringfield.co.uk/movie_script.php?movie=im-brent-morin'] ['https://www.springfieldspringfield.co.uk/movie_script.php?movie=chelsea-peretti-one-of-the-greats'] ['https://www.springfieldspringfield.co.uk/movie_script.php?movie=chelsea-peretti-one-of-the-greats']
変わらず、、。
よろしくお願いします。
回答4件
あなたの回答
tips
プレビュー
バッドをするには、ログインかつ
こちらの条件を満たす必要があります。
2018/12/18 10:25