title: Quickly adding multiple items (1000/sec) to a sidekiq queue?
body: <p>I realize there is a push_bulk option for sidekiq but I'm currently being limited by latency to redis, so passing multiple items via push_bulk still isn't going quickly enough (only about 50/s).</p>
  
  <p>I've tried to increase the number of redis connections like so:</p>
  
  <pre><code>redis_conn = proc {
    Redis.new({ :url =&gt; Rails.configuration.redis.url })
  }
  
  Sidekiq.configure_client do |config|
    Sidekiq.configure_client do |config|
      config.redis = ConnectionPool.new(size: 50, &amp;redis_conn)
    end
    config.client_middleware do |chain|
      chain.add Sidekiq::Status::ClientMiddleware
    end
  end
  </code></pre>
  
  <p>And then fire off separate threads (Thread.new) to actually perform_async on the various objects.  What is interesting is any thread that isn't the first thread NEVER gets thrown into the sidekiq queue, it's like they're ignored entirely.</p>
  
  <p>Does anyone know of a better way to do this?</p>
  
  <p>Edit: Here is the push_bulk method I was trying which is actually slower:</p>
  
  <pre><code>  user_ids = User.need_scraping.pluck(:id)
    bar = ProgressBar.new(user_ids.count)
    user_ids.in_groups_of(10000, false).each do |user_id_group|
      Sidekiq::Client.push_bulk(
        'args'  =&gt; user_id_group.map{ |user_id| [user_id] },
        'class' =&gt; ScrapeUser,
        'queue' =&gt; 'scrape_user',
        'retry' =&gt; true
      )
    end
  </code></pre>
  
  <p>Thanks!</p>