title: Read HDFS file as stream
body: <p>I have a file in my DataLake (HDFS) <code>adl://datalake.azuredatalakestore.net/Data/prod/2018/2018-02/AHA-2018-02-13.json</code>.</p>
  
  <p>This file receive continously data from ESB. Therefore, I would like to read this file like a stream and perform action on lines. I tried this : </p>
  
  <pre><code>from pyspark import SparkContext
  from pyspark.streaming import StreamingContext
  sc = pyspark.SparkContext()
  ssc = StreamingContext(sc, batchDuration=60)
  a = 'adl://datalake.azuredatalakestore.net/Data/prod/2018/2018-02/AHA-2018-02-13.csv'
  b = "adl://datalake.azuredatalakestore.net/Data/dev/test.out"
  lines = ssc.textFileStream(a)
  words = lines.flatMap(lambda x : x.split(' '))
  wordCounts = words.map(lambda x :  (x, 1)).reduceByKey(lambda x,y : x+y)
  wordCounts.saveAsTextFiles(b)
  ssc.start()
  ssc.awaitTermination()
  </code></pre>
  
  <p>This code runs without error but in my output directory <code>adl://datalake.azuredatalakestore.net/Data/dev/test.out</code>, I can see several folders created every minute, all empty despite the fact that I know I received data. </p>
  
  <p>This code above is just an example. I am currently trying to understand how to perform a stream read on my files. The first thing for me is to count words of each new line in my file. Could you please tell me what is wrong with my current code ? </p>