title: Text data preprocessing in python
body: <p>I am working on extraction of positive, negative &amp; neutral keyword in python.There are 10,000 comments in my comments remarks.txt file(encoded UTF-8).I want to import the text file, read the individual row of comments &amp; extract words(tokenize) from the comments mentioned in column c2 &amp; store it in a next adjacent column. I have written a small program calling get_keywords function in Python.I have created get_keywords() function but facing issues passing each row of the dataframe as argument &amp; calling it using iterations  to provide keywords &amp; store it in adjacent columns.</p>
  
  <p>Codes are not providing expected column "tokens" with all the processed words in the df dataframe. </p>
  
  <pre><code>    import nltk
      import pandas as pd
      import re
      import string
      from nltk import sent_tokenize, word_tokenize
      from nltk.corpus import stopwords
      from nltk.stem.porter import PorterStemmer
      remarks = pd.read_csv('/Users/ZKDN0YU/Desktop/comments/New 
      comments/ccomments.txt')
      df = pd.DataFrame(remarks, columns= ['c2'])
      df.head(50)
      df.tail(50)
  
      filename = 'ccomments.txt'
      file = open(filename, 'rt', encoding="utf-8")
      text = file.read()
      file.close()
  
      def get_keywords(row):     
      # split into tokens by white space
        tokens = text.split(str(row))
      # prepare regex for char filtering
        re_punc = re.compile('[%s]' % re.escape(string.punctuation))
      # remove punctuation from each word
        tokens = [re_punc.sub('', w) for w in tokens]
      # remove remaining tokens that are not alphabetic
        tokens = [word for word in tokens if word.isalpha()]
      # filter out stop words
        stop_words = set(stopwords.words('english'))
        tokens = [w for w in tokens if not w in stop_words]
      # stemming of words
        porter = PorterStemmer()
        stemmed = [porter.stem(word) for word in tokens]
      # filter out short tokens
        tokens = [word for word in tokens if len(word) &gt; 1]
        return tokens
        df['tokens'] = df.c2.apply(lambda row: get_keywords(row['c2']), 
         axis=1)
        for index, row in df.iterrows():
        print(index, row['c2'],"tokens : {}".format(row['tokens']))
  </code></pre>
  
  <p>Expected Output:- A Comments_modified file containing columns 1)index,2) c2(Comments) &amp; 3)tokenized words for all rows of the dataframe having  10,000 comments.</p>