title: pyspark writeStream: Each Data Frame row in a separate json file
body: <p>I am using pyspark to read data from a Kafka topic as a streaming dataframe as follows:</p>
  
  <pre><code>spark = SparkSession.builder \
    .appName("Spark Structured Streaming from Kafka") \
    .getOrCreate()
  
  sdf = spark \
    .readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "test") \
    .option("startingOffsets", "latest") \
    .option("failOnDataLoss", "false") \
    .load() \
    .select(from_json(col("value").cast("string"), json_schema).alias("parsed_value"))
  
  sdf_ = sdf.select("parsed_value.*")
  </code></pre>
  
  <p>My goal is to write each of the <code>sdf_</code> rows as <strong>seperate</strong> json files.
  The following code:</p>
  
  <pre><code>writing_sink = sdf_.writeStream \
      .format("json") \
      .option("path", "/Desktop/...") \
      .option("checkpointLocation", "/Desktop/...") \
      .start()
  
  writing_sink.awaitTermination()
  </code></pre>
  
  <p>will write several rows of the dataframe within the same json, depending on the size of the micro-batch (or this is my hypothesis at least).
  What I need is to tweak the above so that each row of the dataframe is written in a separate json file. </p>
  
  <p>I have also tried using <code>partitionBy('column')</code>, but still this will not do exactly what I need, but instead create folders within which the json files might still have multiple rows written within them (if they have the same id).</p>
  
  <p>Any ideas that could help out here? Thanks in advance.</p>