Ruby openuri スクレイピング

Question

Rubyで画像ダウンロードを行うコードを書いています。
chromedriverを使いbing検索を行い画像URLを所得し、open可能なURLを厳選するコードなのですが、厳選する部分でいつも落ちてしまいます。文法のミスかなとは思うのですが、試行錯誤してもうまくいかず。

require 'selenium-webdriver'
require 'uri'
require 'open-uri'
require 'openssl'
require 'fileutils'

client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 300

def valid_url?(url)
    open(url)
    rescue SocketError, OpenURI::HTTPError,OpenSSL::SSL::SSLError,Timeout::Error,URI::InvalidURIError
    false
end




Names = [
"name1",
"name2",
"name3",]




Names.each do | names |
    
    driver = Selenium::WebDriver.for :chrome, :http_client => client
    driver.navigate.to "https://www.bing.com/?scope=images&FORM=Z9LH1"
    element = driver.find_element(:name, 'q')
    element.send_keys names
    element = driver.find_element(:name, 'go')
    element.submit
    path = "./"+names
    FileUtils.mkdir_p(path) unless FileTest.exist?(path)
    

    element = driver.find_elements(:xpath,'//a[@class="iusc"]')
    links = element.map{|a| a.attribute('href')}
    links.uniq! #重複削除

    el = []
    allurls = []
    
    links.each.with_index(1) do |url,i|
        
        n = 2
        begin
            driver.navigate.to url
            rescue Timeout::Error,ECONNREFUSED::Error
            if n==0 then
                next
                else
                n=n-1
                retry
            end
        end
        sleep 2
        
        n = 2
        begin
            tab2.click
            rescue Timeout::Error
            if n==0 then
                next
                else
                n=n-1
                retry
            end
        end
        sleep 2
        el = driver.find_elements(:xpath, '//a[@class="view_image"]')
        src = el.map{|a| a.attribute('href')}
        src.uniq!#重複削除
        allurls.push(src[0])
        //700枚いったら終わり
        if i >= 700 then
            break
        end
    
    end

    valid_urls = allurls.select{|url| valid_url?(url) }
    
    puts valid_urls.length
    
    valid_urls.each.with_index(1) do |urls,i|
        if File.extname(urls).empty? == false then
                filename = File.basename(urls)
                file_path = path + "/" + filename
                open(file_path, 'wb') do |file|
                    open(urls) do |data|
                        file.write(data.read)
                    end
                end
        end
        sleep 1
    end
    driver.quit
end

in `__read_nonblock': Connection reset by peer (Errno::ECONNRESET)
	from <internal:prelude>:77:in `read_nonblock'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/net/protocol.rb:172:in `rbuf_fill'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/net/protocol.rb:154:in `readuntil'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/net/protocol.rb:164:in `readline'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/net/http/response.rb:40:in `read_status_line'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/net/http/response.rb:29:in `read_new'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/net/http.rb:1446:in `block in transport_request'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/net/http.rb:1443:in `catch'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/net/http.rb:1443:in `transport_request'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/net/http.rb:1416:in `request'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/open-uri.rb:329:in `block in open_http'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/net/http.rb:877:in `start'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/open-uri.rb:323:in `open_http'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/open-uri.rb:741:in `buffer_open'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/open-uri.rb:212:in `block in open_loop'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/open-uri.rb:210:in `catch'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/open-uri.rb:210:in `open_loop'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/open-uri.rb:151:in `open_uri'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/open-uri.rb:721:in `open'
	from .rbenv/versions/2.4.0/lib/ruby/2.4.0/open-uri.rb:35:in `open'
	from url.rb:29:in `valid_url?'
	from url.rb:118:in `block (2 levels) in <main>'
	from url.rb:118:in `select'
	from url.rb:118:in `block in <main>'
	from url.rb:34:in `each'
	from url.rb:34:in `<main>'

コードのご指摘があれば頂きたいです。

Answer 1

まず、Errno::ECONNRESETは、エラーメッセージに「Connection reset by peer」とあるように、通信でのエラーで、文法の問題ではありません。

1つの原因としては、open(url)したものをどこでも閉じていないので、開きっぱなしの接続が溜まっておかしくなっていることが考えられます。開いたIOを受け取る必要はなさそうなので、ブロックで閉じてしまえば解決するかもしれません。

def valid_url?(url)
    open url do
        return true
    end
    rescue SocketError, OpenURI::HTTPError,OpenSSL::SSL::SSLError,Timeout::Error,URI::InvalidURIError
    false
end

Comments

[daigakuse-]

とても参考になる指摘をありがとうございます。
参考にいただいたコードを実施しましたが同じエラーが出てしまいます。
end の位置を変えたりしてみたのですが、うまくいかないです。
rescueにErrno::ECONNRESETをセットする方法はありでしょうか？