title: Uploading photos from Google Drive to AWS S3 using Node.js
body: <p>Requirement is to upload photos from a public google drive folder to s3.</p>
  <p>Code to fetch images list from google drive</p>
  <pre><code>module.exports.fetchImageDetails = async (fileId) =&gt; new Promise(async (res, rej) =&gt; 
  {
    const token = await googleUtil();
    const drive = google.drive('v2');
    drive.files.get(
      {
        auth: token,
        fileId: `${fileId}`,
      },
      (err, response) =&gt; {
        if (err) {
          console.log(`The API returned an error: ${err}`);
          rej(err);
        }
        res(response.data);
      },
    );
  });
  </code></pre>
  <p>Iterating over images</p>
  <pre><code>for (file of fetchedFilesInFolder.files) {
      uploadStatus = await s3Helper.uploadImage(
           process.env.MEDIA_S3_BUCKET,
           url,
           fileDetails.webContentLink
      );
  }
  </code></pre>
  <p>Upload to S3:
  const options = {
  uri: url,
  encoding: null,
  };
  const body = await request(options);</p>
  <pre><code>const uploadResult = await s3
    .upload({
      Bucket: bucket,
      Key: path,
      Body: body,
    })
    .promise();
  </code></pre>
  <p>This code sometime works and sometime not. Many a times, image uploaded on S3 is not a valid image.</p>
  <p>We are really not sure, what's happening.</p>
  <p>Please Note: Images are around 6 MB in size. Also, google drive <em>webContentLink</em> by default download the image when we access the URL (<em><a href="https://drive.google.com/uc?id=" rel="nofollow noreferrer">https://drive.google.com/uc?id=</a>&lt;file_id&gt;&amp;export=download</em>). Can this be a root cause of problem for <code>const body = await request(options);</code>? Also, removing <em>&amp;export=download</em> from the link did no wonder as <em><a href="https://drive.google.com/uc?id=" rel="nofollow noreferrer">https://drive.google.com/uc?id=</a>&lt;file_id&gt;</em> is redirect to another link.</p>
  <p>We have already spent enough time on this but couldn't fix it.</p>