title: How to process multiple CSV files from an Amazon S3 bucket in a lambda function?
body: <p>In an Amazon S3 bucket, event logs are sent as a CSV file every hour. I would like to perform some brief descriptive analysis on 1 weeks worth of data, every week (e.g. 168 files every week). The point of the analysis is to output a list of trending products for each week. I have a python script written out on my local machine which retrieves the latest 168 files from S3 using boto3, and does all the necessary wrangling etc.</p>
  <p>But now I need to put this into a lambda function. I will set up an eventbridge to trigger the lambda function every monday. But, is it possible to call multiple files into a lambda function using the standard boto3, or do I need to do something special when defining the lambda handler function?</p>
  <p>Here is the code from my local machine for getting the 168 files:</p>
  <pre><code># import modules
  import boto3
  import pandas as pd
  from io import StringIO
  
  # set up aws credentials
  s3 = boto3.resource('s3')
  client = boto3.client('s3', aws_access_key_id='XXXXXXXXXXXXXXXXXXXX',
                        aws_secret_access_key='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')
  
  # name s3 bucket
  my_bucket = s3.Bucket('bucket_with_data')
  
  # get names of last weeks files from s3 bucket
  files = []
  
  for file in my_bucket.objects.all():
      files.append(file.key)
  
  files = files[-168:]  # all files from last 7 days (24 files * 7 days per week)
  
  bucket_name = 'bucket_with_data'
  
  data = []
  
  col_list = [&quot;user_id&quot;, &quot;event_type&quot;, &quot;event_time&quot;, &quot;session_id&quot;, &quot;event_properties.product_id&quot;]
  
  for obj in files:
      csv_obj = client.get_object(Bucket=bucket_name, Key=obj)
      body = csv_obj['Body']
      csv_string = body.read().decode('utf-8')
      temp = pd.read_csv(StringIO(csv_string))
      final_list = list(set(col_list) &amp; set(temp.columns))
      temp = temp[final_list]
      data.append(temp)
  
  # combining all dataframes into one
  event_data = pd.concat(data, ignore_index=True)
  </code></pre>
  <p>So, my question is, can I just put this code into a lambda function and it should work, or do I need to incorporate a lambda_handler function? If I need to use a lambda_handler function, how would I handle multiple files? Because the lambda is being triggered by a schedule, rather than one event taking place.</p>