title: Uploading Large File to S3 with Ruby Fails with Out of Memory Error, How to Read and Upload in Chunks?
body: <p>We are uploading various files to S3 via the Ruby AWS SDK (v2) from a Windows machine. We have tested with Ruby 1.9. Our code works fine except when large files are encountered, when an out of memory error is thrown.</p>
  
  <p>At first we were reading the whole file into memory with this code:</p>
  
  <pre><code>:body =&gt; IO.binread(filepath),
  </code></pre>
  
  <p>Then after Googling we found that there were ways to read the file in chunks with Ruby:</p>
  
  <pre><code>:body =&gt;  File.open(filepath, 'rb') { |io| io.read },
  </code></pre>
  
  <p>This code did not resolve the issue though, and we can't find a specific S3 (or related) example which shows how the file can be read and passed to S3 in chunks. The whole file is still loaded into memory and throws an out of memory error with large files.</p>
  
  <p>We know we can split the file into chunks and upload to S3 using the AWS multi part upload, however the preference would be to avoid this if possible (although it's fine if it's the only way).</p>
  
  <p>Our code sample is below. What is the best way to read the file in chunks, avoiding the out of memory errors, and upload to S3? </p>
  
  <pre><code>require 'aws-sdk'
  
  filepath = 'c:\path\to\some\large\file.big'
  bucket = 's3-bucket-name'
  s3key = 'some/s3/key/file.big'
  accesskeyid = 'ACCESSKEYID'
  accesskey = 'ACCESSKEYHERE'
  region = 'aws-region-here'
  
  s3 = Aws::S3::Client.new(
    :access_key_id =&gt; accesskeyid,
    :secret_access_key =&gt; accesskey,
    :region =&gt; region
    )
  
  resp = s3.put_object(
    :bucket =&gt; bucket,
    :key =&gt; s3key,
    :body =&gt;  File.open(filepath, 'rb') { |io| io.read },
    )
  </code></pre>
  
  <p>Note that we are not hitting the S3 5GB limit, this is happening for files for example of 1.5GB.</p>