title: How to remove duplicate tweets in Python?
body: <p>I am trying to retrieve about 1000 tweets from a search term like 'NFL' using tweepy and storing the tweets into a DataFrame using pandas. My issue is I can't find a way to remove duplicated tweets, I have tried df.drop_duplicates but it only gives me about 100 tweets to work with. Help would be appreciated!</p>
  <pre><code>num_needed = 1000
  tweet_list = [] # Lists to be added as columns( Tweets, usernames, and screen names) in our dataframe 
  user_list = []
  screen_name_list = []
  last_id = -1 # ID of last tweet seen
  while len(tweet_list) &lt; num_needed:
      try:
          new_tweets = api.search(q = 'NFL', count = num_needed, max_id = str(last_id - 1), lang = 'en', tweet_mode = 'extended') # This is the criteria for collecting the tweets that I want. I want to make sure the results are as accurate as possible when making a final analysis.
      except tweepy.TweepError as e:
          print(&quot;Error&quot;, e)
          break
      else:
          if not new_tweets:
              print(&quot;Could not find any more tweets!&quot;)
              break
          else:
              for tweet in new_tweets:
  
                  # Fetching the screen name and username
  
                  screen_name = tweet.author.screen_name
                  user_name = tweet.author.name
    
                  tweet_text = tweet.full_text
                  tweet_list.append(tweet_text)
                  user_list.append(user_name)
                  screen_name_list.append(screen_name)
  
          
  df = pd.DataFrame()  #Create a new dataframe (df) with new columns 
  df['Screen name'] = screen_name_list
  df['Username'] = user_list
  df['Tweets'] = tweet_list
  
  </code></pre>