title: Spark : UnpicklingError: NEWOBJ class argument has NULL tp_new when trying to push the data to Kinesis Firehose
body: <p>I am trying to data to kinesis firehose through spark : </p>
  
  <p>I am getting an error like 
  UnpicklingError : NEWOBJ Class argument has NULL tp_new </p>
  
  <p>I am using python version : 2.10 (Anaconda version) 
  Spark - 2.4.6-hadoop version. 
  I am using Boto3 - to push the data to Amazon Kinesis. </p>
  
  <p>When I tried to push the data to kinesis with sample data with python script - it works well and I get the resultant output like shown below.. </p>
  
  <pre><code>In [21]: aws_client.put_record_batch(DeliveryStreamName='streamname',Records=Records)
  
  And I am getting a success message that it is pushed to kinesis : 
  
  Out[16]: 
  {u'FailedPutCount': 0,
   u'RequestResponses': [{u'RecordId': u'gfdgfdgdtgrt4rt4trfg'},
    {u'RecordId': u'hggfhhy7686ghytryhfgfdgdgtrged'}],
   'ResponseMetadata': {'HTTPStatusCode': 200,
    'RequestId': 'u'ytutujguiuuuiiuuu'}}
  </code></pre>
  
  <p><a href="http://boto3.readthedocs.org/en/latest/guide/configuration.html" rel="nofollow noreferrer">http://boto3.readthedocs.org/en/latest/guide/configuration.html</a>
  <a href="http://boto3.readthedocs.org/en/latest/reference/services/kinesis.html#client" rel="nofollow noreferrer">http://boto3.readthedocs.org/en/latest/reference/services/kinesis.html#client</a></p>
  
  <p>similarly if i try in spark - (Doing processing for enrichment of tweets from twitter and pushing to kinesis).. </p>
  
  <pre><code>sc = SparkContext(appName="PythonStreamingKafkaWordCount")
  ssc = StreamingContext(sc, 1)
  
  zkQuorum, topic = sys.argv[1:]
  kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer", {topic: 1})
  lines = kvs.map(lambda x: x[1])
  enrichedData = lines.flatMap(f1)
  enrichedData.foreachRDD(lambda rdd: rdd.foreachPartition(sendToKinesis))
  # simplejson.l1oad()
  ssc.start()
  ssc.awaitTermination()
  </code></pre>
  
  <p>I am getting an error like 
  UnpicklingError : NEWOBJ Class argument has NULL tp_new </p>
  
  <p><a href="https://i.stack.imgur.com/pLmdI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pLmdI.png" alt="enter image description here"></a></p>
  
  <p>Can anyone help me to figure out why it is not working with Spark? </p>