title: Spark Streaming using Tweepy
body: <p>I'm trying to stream twitter data using python library Tweepy. I have setup working environment googled about the stuff but i'm not getting how things are working. I want to use spark streaming (DStream - Batch processing) with python (tweepy). I have go through at least following links:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/27243819/how-to-get-tweets-of-a-particular-hashtag-in-a-location-in-a-tweepy">How to get tweets of a particular hashtag in a location in a tweepy?</a></li>
  <li><a href="http://spark.apache.org/docs/latest/streaming-programming-guide.html" rel="nofollow noreferrer">http://spark.apache.org/docs/latest/streaming-programming-guide.html</a></li>
  <li><a href="http://docs.tweepy.org/en/v3.5.0/streaming_how_to.html" rel="nofollow noreferrer">http://docs.tweepy.org/en/v3.5.0/streaming_how_to.html</a></li>
  <li><a href="https://stackoverflow.com/questions/26701311/retrieving-twitter-data-using-tweepy">Retrieving Twitter data using Tweepy</a></li>
  <li><a href="http://www.awesomestats.in/spark-twitter-stream" rel="nofollow noreferrer">http://www.awesomestats.in/spark-twitter-stream</a></li>
  </ul>
  
  <p>Following tweepy code is working fine for me:</p>
  
  <pre><code>import tweepy
  
  consumer_key = ''
  consumer_secret = ''
  access_token = ''
  access_secret = ''
  
  auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
  auth.set_access_token(access_token, access_secret)
  api = tweepy.API(auth)
  
  politicsTweets = tweepy.Cursor(api.search, q='#GONAWAZGO').items(100)
  
  for tweet in politicsTweets:
      print tweet.created_at, tweet.text, tweet.lang
  </code></pre>
  
  <p>but it's not using spark streaming. How should I update the aforementioned code to use Spark Streaming? I'm not getting why do I need two separate files? Overall I'm trying to do the followings:</p>
  
  <ol>
  <li>Get top 10 hashtags from 1st May, 2017. (Tweepy search function accepts parameter 'since_id', not getting how to use it [<a href="http://docs.tweepy.org/en/latest/api.html#help-methods" rel="nofollow noreferrer">http://docs.tweepy.org/en/latest/api.html#help-methods</a> ]?)</li>
  <li>Count how many times #GONAWAZGO found since 11th May, 2013.</li>
  <li>Count how many #gonawazgo were done by people outside of Pakistan. (Without any date limit, Tweepy cursor method accepts geocode but I want tweets from locations other than the provided geocode.)</li>
  <li>Observe the trend about France Elections on Twitter.</li>
  <li>Find the most recent tweets done by [<a href="https://twitter.com/imrankhanpti" rel="nofollow noreferrer">https://twitter.com/imrankhanpti</a> ] twitter account. (Tweepy search method accepts userID, how I can get that?)</li>
  </ol>
  
  <p>Above all I'm a bit confused about when to use Twitter REST/Streaming API. I think for 1st and 2nd point REST API should be used as we are processing past data till date and for remaining Streaming API should be used.</p>