title: How to read huge number of .gz S3 files into RDD?
body: <pre><code>aws s3api list-objects-v2 --bucket cw-milenko-tests | grep 'tick_c'
  </code></pre>
  
  <p>output shows </p>
  
  <pre><code>    "Key": "Json_gzips/tick_calculated_3_2020-05-27T11-50-22.json.gz",
      "Key": "Json_gzips/tick_calculated_3_2020-05-27T11-52-59.json.gz",
      "Key": "Json_gzips/tick_calculated_3_2020-05-27T11-55-08.json.gz",
      "Key": "Json_gzips/tick_calculated_3_2020-05-27T11-57-30.json.gz",
      "Key": "Json_gzips/tick_calculated_3_2020-05-27T11-59-59.json.gz",
      "Key": "Json_gzips/tick_calculated_4_2020-05-27T09-14-28.json.gz",
      "Key": "Json_gzips/tick_calculated_4_2020-05-27T11-35-38.json.gz",  
  </code></pre>
  
  <p>With <code>wc -l</code></p>
  
  <pre><code>aws s3api list-objects-v2 --bucket cw-milenko-tests | grep 'tick_c' | wc -l
  457
  </code></pre>
  
  <p>I can read one file into data frame.</p>
  
  <pre><code>val path ="tick_calculated_2_2020-05-27T00-01-21.json"
  scala&gt; val tick1DF = spark.read.json(path)
  tick1DF: org.apache.spark.sql.DataFrame = [aml_barcode_canc: string, aml_barcode_payoff: string ... 70 more fields]
  </code></pre>
  
  <p>I was surprised to see negative votes. 
  What I want to know is how to load 457 files into RDD? I saw <a href="https://stackoverflow.com/questions/27477730/how-to-read-multiple-gzipped-files-from-s3-into-a-single-rdd">this SO</a> question.
  Is it possible at all? What are the limitations?
  This is what I tried so far.</p>
  
  <pre><code>val rdd1 = sc.textFile("s3://cw-milenko-tests/Json_gzips/tick_calculated*.gz")
  </code></pre>
  
  <p>If I go for s3a</p>
  
  <pre><code>val rdd1 = sc.textFile("s3a://cw-milenko-tests/Json_gzips/tick_calculated*.gz")
  rdd1: org.apache.spark.rdd.RDD[String] = s3a://cw-milenko-tests/Json_gzips/tick_calculated*.gz MapPartitionsRDD[3] at textFile at &lt;console&gt;:27
  </code></pre>
  
  <p>Doesn't work either.</p>
  
  <p>Try to inspect my RDD.</p>
  
  <pre><code>scala&gt; rdd1.take(1)
  java.io.IOException: No FileSystem for scheme: s3
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
  </code></pre>
  
  <p>FileSytem was not recognized.</p>
  
  <p>My GOAL:</p>
  
  <pre><code>s3://json.gz -&gt; rdd -&gt; parquet
  </code></pre>