title: Pyspark Structured streaming locally with Kafka-Jupyter
body: <p>After looking at the other answers i still cant figure it out. </p>
  
  <p>I am able to use kafkaProducer and kafkaConsumer to send and receive a messages from within my notebook. </p>
  
  <pre><code>    producer = KafkaProducer(bootstrap_servers=['127.0.0.1:9092'],value_serializer=lambda m: json.dumps(m).encode('ascii'))
      consumer = KafkaConsumer('hr',bootstrap_servers=['127.0.0.1:9092'],group_id='abc' )
  </code></pre>
  
  <p>I've tried to connect to the stream with both spark context and spark session. </p>
  
  <pre><code>    from pyspark.streaming.kafka import KafkaUtils
      sc = SparkContext("local[*]", "stream")
      ssc = StreamingContext(sc, 1)
  </code></pre>
  
  <p>Which gives me this error</p>
  
  <pre><code>    Spark Streaming's Kafka libraries not found in class path. Try one 
      of the following.
  
      1. Include the Kafka library and its dependencies with in the
      spark-submit command as
  
      $ bin/spark-submit --packages org.apache.spark:spark-streaming- 
      kafka-0-8:2.3.2 ...
  </code></pre>
  
  <p>It seems that i needed to add the JAR to my </p>
  
  <pre><code>    !/usr/local/bin/spark-submit   --master local[*]  /usr/local/Cellar/apache-spark/2.3.0/libexec/jars/spark-streaming-kafka-0-8-assembly_2.11-2.3.2.jar pyspark-shell
  </code></pre>
  
  <p>which returns </p>
  
  <pre><code>    Error: No main class set in JAR; please specify one with --class
      Run with --help for usage help or --verbose for debug output
  </code></pre>
  
  <p>What class do i put in?
  How do i get Pyspark to connect to the consumer?</p>