title: Splitting Large CSV file from S3 in AWS Lambda function to read
body: <p>I have a huge CSV file which is of 2GB to be uploaded to an AWS lambda function. The Lambda function has the maximum time out of 15 mins and that can not be exceeded. When the code hits the part where it needs to read that huge file from S3 and convert it into a data frame in python code, it just keeps processing and I face exit time error. 
  I can not create multiple lambda functions to read it but in the single lambda function to read that big file. I understood the concept of splitting the file into smaller chunks and reading each as a file and then I want to append all the chunks into a single data frame, I can work at. Apparently my further function would need the entire data frame created, instead of small data frames.</p>
  
  <p>I am using this code to read the file from s3:</p>
  
  <pre class="lang-py prettyprint-override"><code>
      s3 = boto3.client('s3')
      Bucket_Dest = buckets['DbBucket']
      file_name = "/Temp_File/" + BlastFolder + fileName
      file_merge = s3.get_object(Bucket = Bucket_Dest, Key = file_name)
      file_content = file_merge['Body'].read()
      read_csv_data = io.BytesIO(file_content)
      df = pd.read_csv(read_csv_data)
  
  </code></pre>
  
  <p>The data frame named "df" is not getting created as the file from s3 is not loaded because of its large size.
  Please help me in how can I load files through splitting and appending.</p>
  
  <p>Thanks</p>