title: How to read multiple gzipped files from S3 into a single RDD with http request?
body: <p>I have to download many gzipped files stored on S3 like this:</p>
  
  <pre><code>crawl-data/CC-MAIN-2018-43/segments/1539583508988.18/robotstxt/CC-MAIN-20181015080248-20181015101748-00000.warc.gz
  crawl-data/CC-MAIN-2018-43/segments/1539583508988.18/robotstxt/CC-MAIN-20181015080248-20181015101748-00001.warc.gz
  </code></pre>
  
  <p>to download them you must add the prefix <a href="https://commoncrawl.s3.amazonaws.com/" rel="nofollow noreferrer">https://commoncrawl.s3.amazonaws.com/</a></p>
  
  <p>I have to download and decompress the files,then assemble the content as a single RDD.</p>
  
  <p>Something similar to this:</p>
  
  <pre><code>JavaRDD&lt;String&gt; text = 
      sc.textFile("https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2018-43/segments/1539583508988.18/robotstxt/CC-MAIN-20181015080248-20181015101748-00000.warc.gz");
  </code></pre>
  
  <p>I want to do this code with spark:</p>
  
  <pre><code>    for (String key : keys) {
          object = s3.getObject(new GetObjectRequest(bucketName, key));
  
          gzipStream = new GZIPInputStream(object.getObjectContent());
          decoder = new InputStreamReader(gzipStream);
          buffered = new BufferedReader(decoder);
  
          sitemaps = new ArrayList&lt;&gt;();
  
          String line = buffered.readLine();
  
          while (line != null) {
              if (line.matches("Sitemap:.*")) {
                  sitemaps.add(line);
              }
              line = buffered.readLine();
          }
  </code></pre>