title: Loading data from AWS S3 through Apache-Spark
body: <p>I have written a python code to load files from Amazon Web Service (AWS) S3 through Apache-Spark. Specifically, the code creates RDD and load all csv files from the directory <code>data</code> in my bucket <code>ruofan-bucket</code> on AWS S3 using <code>SparkContext().wholeTextFiles("s3n://ruofan-bucket/data")</code>. The code shows below:</p>
  
  <pre><code>import os, sys, inspect
  
  ### Current directory path.
  curr_dir = os.path.split(inspect.getfile(inspect.currentframe()))[0]
  
  ### Setup the environment variables
  spark_home_dir = os.path.realpath(os.path.abspath(os.path.join(curr_dir, "../spark-1.4.0")))
  python_dir = os.path.realpath(os.path.abspath(os.path.join(spark_home_dir, "./python")))
  os.environ["SPARK_HOME"] = spark_home_dir
  os.environ["PYTHONPATH"] = python_dir
  
  ### Setup pyspark directory path
  pyspark_dir = os.path.realpath(os.path.abspath(os.path.join(spark_home_dir, "./python")))
  sys.path.append(pyspark_dir)
  
  ### Import the pyspark
  from pyspark import SparkConf, SparkContext
  
  def main():
      ### Initialize the SparkConf and SparkContext
      conf = SparkConf().setAppName("ruofan").setMaster("local")
      sc = SparkContext(conf = conf)
  
      ### Create a RDD containing metadata about files in directory "data"
      datafile = sc.wholeTextFiles("s3n://ruofan-bucket/data")    ### Read data directory from S3 storage.
  
      ### Collect files from the RDD
      datafile.collect()
  
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>Before I run my code, I've already exported the environment variables: <code>AWS_SECRET_ACCESS_KEY</code> and <code>AWS_ACCESS_KEY_ID</code>. But when I run my code, it shows up the error: </p>
  
  <p><code>IOError: [Errno 2] No such file or directory: 's3n://ruofan-bucket/data/test1.csv'</code></p>
  
  <p>I'm sure I have the directory as well as the files on AWS S3, and I have no idea about the error. I really appreciate if anyone helps me solve the problem.</p>