<p>Firstly, you open at once a large file that needs to fit into your RAM. If you do not have a really good computer, this might be the first bottleneck for perfomance. Read each line separately, or try to use some IO buffer.
What CPU do you have? If you have enough cores, you can get a lot of extra performance when parallelizing the program with an async Pool from Multiprocessing library because you really use the full power of all cores (choose the number of processes according to the thread number. With this method, I reduced a model on 2500 data sets from ~5 minutes to ~17 seconds on 12 threads). You would have to implement the processes to return a dict each, updating them after the processes have finished.</p>

<p>Otherwise, there are machine learning approches for text summarization (sequence to sequence RNNs). With a tensorflow implementation, you can use a dedicated GPU on your local machine (even a decent 10xx or a 2060 from Nvidia will help) to speed up your model. </p>

<p><a href="https://docs.python.org/2/library/multiprocessing.html" rel="nofollow noreferrer">https://docs.python.org/2/library/multiprocessing.html</a> 
<a href="https://arxiv.org/abs/1602.06023" rel="nofollow noreferrer">https://arxiv.org/abs/1602.06023</a></p>

<p>hope this helps</p>
Improve efficiency for ranking in Bag of words model
<p>I am creating a text summarizer and using a basic model to work with using Bag of words approach.<br>
the code i am performing is using the nltk library. 
the file read is a large file with over 2500000 words. 
below is the loop i am working on with but this takes over 2 hours to run and complete. is there a way to optimize this code    </p>

<pre><code>f= open('Complaints.csv', 'r')
raw = f.read()
len(raw)
tokens = nltk.word_tokenize(raw)
len(tokens)
freq = nltk.FreqDist(text)
top_words = [] # blank dictionary 
top_words = freq.most_common(100)
print(top_words)
sentences = sent_tokenize(raw)
print(raw)
ranking = defaultdict(int)
for i, sent in enumerate(raw):
for word in word_tokenize(sent.lower()):
    if word in freq:
        ranking[i]+=freq[word]
top_sentences = nlargest(10, ranking, ranking.get)
print(top_sentences)
</code></pre>

<p>This is only one one file and the actual deployment has more than 10-15 files of similar size. 
How we can improve this.<br>
Please note these are the text from a chat bot and are actual sentences hence there was no requirement to remove whitespaces, stemming and other text pre processing methods</p>