title: Retrieving data from s3 bucket in pyspark
body: <p>I am reading data from s3 bucket in pyspark . I need to parallelize read operation and doing some transformation on the data. But its throwing error. Below is the code.</p>
  
  <pre><code>s3 = boto3.resource('s3',aws_access_key_id=access_key,aws_secret_access_key=secret_key)
  bucket = s3.Bucket(bucket)
  
  prefix = 'clickEvent-2017-10-09'
  files = bucket.objects.filter(Prefix = prefix)
  keys=[k.key for k in files]
  pkeys = sc.parallelize(keys)
  </code></pre>
  
  <p>I have a global variable <code>d</code> which is an empty list. And I am appending <code>deviceId</code> data into this.</p>
  
  <p><code>applying flatMap on the keys</code> </p>
  
  <pre><code>pkeys.flatMap(map_func)
  </code></pre>
  
  <p>This the function      </p>
  
  <pre><code> def map_func(key):
     print "in map func"
     for line in key.get_contents_as_string().splitlines():
      # parse one line of json
       content = json.loads(line)
       d.append(content['deviceID'])
  </code></pre>
  
  <p>But the above code gives me error.
  Can anyone help!</p>