title: Speeding up webscraping with Nokogiri & Ruby
body: <p>I'm brand new to using Nokogiri and Ruby, so bear with me here. I wrote a script to access a bunch of different pages on a website, each of which contains some links, which I want to extract. It works as intended, but I was wondering if theres a faster, more efficient way of accomplishing this. As it stand, the <b>get_urls</b> method collects the name used for the url of the pages where the links are located, so it will get 'bitcoin', 'ethereum', etc, then in the <b>get_websites</b> method it puts that string into the url where the links are located and scrapes each page.</p>
  
  <p>I'm wondering if the way I'm doing this is inefficient and if there is a better way to do it. It takes a fairly long time to run, but as I said, this is my first foray into web scraping so I'm not sure if that is normal. </p>
  
  <pre><code>require 'open-uri'
  require 'nokogiri'
  require 'csv'
  
  def get_urls    
    url               = "https://coinmarketcap.com/"
    document          = open(url)
    content           = document.read
    parsed_content    = Nokogiri::HTML(content)
    cointable         = parsed_content.css('.col-lg-10').css('.row').css('.col-xs-12').css('#currencies')
  
    cointable.css("tbody").css("tr").each do |row|
      currency_name   = row.css('.currency-name-container').inner_text
      currency_link   = row.css('.currency-name').css('.currency-symbol').css("a").first.values
      link_name       = currency_link[0].gsub("/currencies/", "").gsub("/","")
      $url_list &lt;&lt; [link_name, currency_name]
    end
  end
  
  def get_websites
    $url_list.each do |url|
      coin_arr        = Array.new
      url_name        = url[0]
      coin_name       = url[1]
      url             = "https://coinmarketcap.com/currencies/#{url_name}"
      document        = open(url)
      content         = document.read
      parsed_content  = Nokogiri::HTML(content)
      links           = parsed_content.css('.container').css('.list-unstyled')
      puts "Adding #{url_name}..."
  
      links.css("li").css("a").each do |row|
          website = row.first[1]
          website_name = row.inner_text
          coin_arr.push([website_name, website])
          CSV.open('crypto_websites.csv', 'a+') do |csv|
              csv &lt;&lt; [coin_name, website_name, website]
          end
      end
      $list_of_sites[url_name] = coin_arr 
    end
  end
  
  $url_list = Array.new
  $list_of_sites = {}
  get_urls
  get_websites
  </code></pre>