title: Is is possible to parse JSON string from Kafka topic in real time using Spark Streaming SQL?
body: <p>I have a Pyspark notebook that connects to kafka broker and creates a spark writeStream called temp. The data values in Kafka topic are in json format but I'm not sure how to go about creating a spark sql table that can parse this data in real time. The only way I know is to create a copy of the table convert it into RDD or DF and parse the value into another RDD and DF. Is is possible to have this done in real time processing as the stream is being written?</p>
  
  <p><strong>Code:</strong></p>
  
  <pre><code>df = spark \
      .readStream \
      .format("kafka") \
      .option("kafka.bootstrap.servers","localhost:9092") \
      .option("subscribe","hoteth") \
      .option("startingOffsets", "earliest") \
      .load()
  
  ds = df.selectExpr("CAST (key AS STRING)", "CAST(value AS STRING)", "timestamp")
  ds.writeStream.queryName("temp").format("memory").start()
  spark.sql("select * from temp limit 5").show()
  </code></pre>
  
  <p><strong>Output:</strong></p>
  
  <pre><code>+----+--------------------+--------------------+
  | key|               value|           timestamp|
  +----+--------------------+--------------------+
  |null|{"e":"trade","E":...|2018-09-18 15:41:...|
  |null|{"e":"trade","E":...|2018-09-18 15:41:...|
  |null|{"e":"trade","E":...|2018-09-18 15:41:...|
  |null|{"e":"trade","E":...|2018-09-18 15:41:...|
  |null|{"e":"trade","E":...|2018-09-18 15:41:...|
  +----+--------------------+--------------------+
  </code></pre>