title: Handling symlinks in pyspark textFileStream
body: <p>I have a textFileStream which reads files coming into a folder <code>'hl/'</code> every <code>3</code> seconds</p>
  <p>This works fine when I put actual <code>.csv</code> files inside the folder. However when putting symlinks to <code>.csv</code> files that are located elsewhere on the drive it does not work.</p>
  <p>Is it possible to make it work, what am I missing ?</p>
  <p>Here's my code for the textFileStream</p>
  <pre><code>from pyspark.streaming import StreamingContext
  from pyspark.sql import SparkSession
  def testprint(rdd):
    if not rdd.isEmpty():
      print('Empty')
    else:
      rdd.toDF().pprint()
  
  spark = SparkSession.builder.getOrCreate()
  
  sc = spark.sparkContext
  ssc = StreamingContext(sc, 3)
  files = ssc.textFileStream('hl/')
  files.foreachRDD(testprint)
  ssc.start()
  time.sleep(30)
  ssc.stop()
  </code></pre>
  <p>While It runs I just copy a csv or make a symlink inside the folder.
  Here's the output while copying :</p>
  <pre><code>Empty
  ----------------------------------
  Time: 2022-04-25 12:12:03
  ----------------------------------
  
  A, B
  0, 1
  1, 0
  0, 0
  ----------------------------------
  Time: 2022-04-25 12:12:06
  ----------------------------------
  </code></pre>
  <p>And Here's the output using symlinks :</p>
  <pre><code>Empty
  ----------------------------------
  Time: 2022-04-25 12:14:03
  ----------------------------------
  
  Empty 
  ----------------------------------
  Time: 2022-04-25 12:14:06
  ----------------------------------
  </code></pre>
  <p>Edit :
  reading symlinks works when using the usual non-streaming way i.e. :</p>
  <pre><code>&gt; df = spark.read.option('multiline', 'true').option('header', 'true').csv('hl/')
  &gt; df.show()
  +-----+
  | A| B|
  | 0| 1|
  +-----+
  </code></pre>
  <p>Any help appreciated !</p>