title: Spark - Reading JSON from Partitioned Folders using Firehose
body: <p>Kinesis firehose manages the persistence of files, in this case time series JSON, into a folder hierarchy that is partitioned by YYYY/MM/DD/HH (down to the hour in 24 numbering)...great.</p>
  
  <p>How using Spark 2.0 then can I read these nested sub folders and create a static Dataframe from all the leaf json files?  Is there an 'option' to the dataframe reader?</p>
  
  <p>My next goal is for this to be a streaming DF, where new files persisted by Firehose into s3 naturally become part of the streaming dataframe using the new structured streaming in Spark 2.0.  I know this is all experimental - hoping someone has used S3 as a streaming file source before, where the data is paritioned into folders as described above.  Of course would prefer straight of a Kinesis stream but there is no date on this connector for 2.0 so Firehose->S3 is the interim.</p>
  
  <p>ND:  I am using databricks, which mounts S3 into DBFS, but could easily be EMR of course or other Spark providers. Be great to see a notebook too if one is shareable that gives an example.</p>
  
  <p>Cheers!</p>