title: Read files along with subdirectories from s3 using sc.textFile
body: <p>Trying to read the s3 data from java spark context:</p>
  
  <pre><code>"mapreduce.input.fileinputformat.input.dir.recursive", "true"
  jsc.textFile(filePath);
  </code></pre>
  
  <p>It was working for me when I have only files insides hour folders:</p>
  
  <pre><code>s3://&lt;year&gt;/&lt;month&gt;/&lt;day&gt;/&lt;hour&gt;/&lt;files&gt;
  jsc.textFile("s3://&lt;year&gt;/&lt;month&gt;/&lt;day&gt;");
  </code></pre>
  
  <p>Now, in S3 parallel to hour folder we may have new_folder as well</p>
  
  <pre><code>s3://&lt;year&gt;/&lt;month&gt;/&lt;day&gt;/&lt;hour&gt;/&lt;files&gt;
  s3://&lt;year&gt;/&lt;month&gt;/&lt;day&gt;/&lt;hour&gt;/&lt;new_folder&gt;/&lt;files&gt;
  </code></pre>
  
  <p>Below code ignoring the files under new_folder</p>
  
  <pre><code>jsc.textFile("s3://&lt;year&gt;/&lt;month&gt;/&lt;day&gt;");
  </code></pre>
  
  <p>Tried with multiple regular expression, but my method "isPathExist" always return false</p>
  
  <pre><code>jsc.textFile("s3n://&lt;year&gt;/&lt;month&gt;/&lt;day&gt;/*/&lt;regular_expression&gt;");
  </code></pre>
  
  <p>Checked S3 path using below method, which returns false</p>
  
  <pre><code>private static boolean isPathExists(String folderPath, String bucket, String accessKey, String secret) {
      AWSCredentials cred = new BasicAWSCredentials(accessKey, secret);
      AmazonS3 s3 = new AmazonS3Client(cred);
      ObjectListing objectListing = s3
              .listObjects(new ListObjectsRequest().withBucketName(bucket).withPrefix(folderPath));
      return !objectListing.getObjectSummaries().isEmpty();
  }
  </code></pre>