title: Fast way to import 24000 single variation products to Shopify
body: <p>I want to import ~24000 (single variation) products to Shopify every night at 04:00. </p>
  
  <p>The problem with the default <code>Admin &gt; Products &gt; Import &gt; CSV</code> method is that it takes an unacceptably long time to finish. So I started working with the <a href="http://docs.shopify.com/api/product" rel="nofollow">Shopify API</a> using their <a href="https://github.com/Shopify/shopify_api" rel="nofollow">Ruby gem</a>.</p>
  
  <p>The Shopify API has an API call limit of 2 calls per second, with room for a burst of 40 calls at once. You can read more about the API call limit <a href="http://docs.shopify.com/api/introduction/api-call-limit" rel="nofollow">here</a>. I don't understand the "room for a burst of 40 calls at once" part.</p>
  
  <p>A Shopify Employee told me the following about importing products via the API:</p>
  
  <blockquote>
    <p>If you were to make Product creation API requests that includes the
    variant information in them (assuming 3 variants per product) and
    using a default API limit (refresh of 2 requests per second, or 2
    concurrent requests) you could probably finish creating all those
    products in about 67 minutes.</p>
  </blockquote>
  
  <p>I wrote the following Ruby script to import all products to Shopify:</p>
  
  <pre><code>require 'shopify_api'
  require 'open-uri'
  require 'json'
  
  begin_time = Time.now
  
  shop_url  = "https://&lt;API_KEY&gt;:&lt;PASSWORD&gt;@&lt;SHOPNAME&gt;.myshopify.com/admin"
  
  ShopifyAPI::Base.site = shop_url
  
  raw_product_data = JSON.parse(open('&lt;LINK&gt;') {|f| f.read }.force_encoding('UTF-8'))
  raw_product_data_size = raw_product_data.size
  
  puts '========================================================================='
  puts "#{raw_product_data_size} products loaded. Starting import now..."
  puts '-------------------------------------------------------------------------'
  
  raw_product_data.each_with_index do |item, index|
    single_product_begin_time = Time.now
  
    # Store item data in variables
    &lt;DATA&gt;
  
    # Create new product
    new_product = ShopifyAPI::Product.new
    new_product.title = title
    new_product.body_html = body
    new_product.product_type = type
    new_product.vendor = vendor
    new_product.tags = tags
  
    new_product.variants = [
      {
        "option_1" =&gt; "First",
        "price" =&gt; variant_price,
        "sku" =&gt; variant_sku
      }
    ]
  
    new_product.images = [
      {
        src: image_src
      }
    ]
  
    new_product.save
  
    creation_time = Time.now - single_product_begin_time
  
    puts "#{((index / raw_product_data_size.to_f) * 100).round(2)}% - #{index} - #{variant_sku} added in #{creation_time.round(2)} seconds"
  end
  
  puts '-------------------------------------------------------------------------'
  puts "Done. It took #{begin_time - Time.now} minutes."
  puts '========================================================================='
  </code></pre>
  
  <p>But I'm not getting anywhere near the 67 minutes. </p>
  
  <p>It currently takes around 3 seconds to add a single product. At this rate it would take 20 hours to add 24000 products. That's unacceptable. Everything under 2 hours is acceptable.</p>
  
  <p>What could I try to speed things up?</p>