title: Rails/Heroku - How to create a background job for process that requires file upload
body: <p>I run my Rails app on Heroku. I have an admin dashboard that allows for creating new objects in bulk through a custom CSV uploader. Ultimately I'll be uploading CSVs with 10k-35k rows. The parser works perfectly on my dev environment and 20k+ entries are successfully created through uploading the CSV. On Heroku, however, I run into H12 errors (request timeout). This obviously makes sense since the files are so large and so many objects are being created. To get around this I tried some simple solutions, amping up the dyno power on Heroku and reducing the CSV file to 2500 rows. Neither of these did the trick.</p>
  
  <p>I tried to use my <code>delayed_job</code> implementation in combination with adding a worker dyno to my procfile to <code>.delay</code> the file upload and process so that the web request wouldn't timeout waiting for the file to process. This fails, though, because this background process relies on a CSV upload which is held in memory at the time of the web request so the background job doesn't have the file when it executes.</p>
  
  <p><strong>It seems like what I might need to do is</strong>:</p>
  
  <ol>
  <li>Execute the upload of the CSV to S3 as a background process</li>
  <li>Schedule the processing of the CSV file as a background job</li>
  <li>Make sure the CSV parser knows how to find the file on S3</li>
  <li>Parse and finish</li>
  </ol>
  
  <p>This solution isn't 100% ideal as the admin user who uploads the file will essentially get an "ok, you sent the instructions" confirmation without good visibility into whether or not the process is executing properly. But I can handle that and fix later if it gets the job done.</p>
  
  <p><strong>tl;dr question</strong></p>
  
  <p>Assuming the above-mentioned solution is the right/recommended approach, how can I structure this properly? I am mostly unclear on how to schedule/create a delayed_job entry that knows where to find a CSV file uploaded to S3 via Carrierwave. Any and all help much appreciated.</p>
  
  <p>Please request any code that's helpful.</p>