Take the 2-minute tour ×

I am a python newibie and I am writing for expert advice

Of course this question is very basic and was asked in stackoverflow before. I have a specific requirement in addition to just reading line by line.

For each line in the file, I want to iterate over entire file. On way to do this is by reading entire file, save it to list, then go over line of interest. This method uses lot of memory, so I am looking for an alternative

Code so far:

for each_line in fileinput.input(input_file):

    for each_line_again in fileinput.input(input_file):

Executing this code gives an error message "device active" :(

Any suggestions ?



EDIT: The purpose is to calculate pair-wise string similarity. Meaning, for each line in file, I want to calculate levenshtein distance with every other line

share|improve this question
Is that your real code indenting? The second for loop runs within the first one? Use four spaces for indentation, please. –  eumiro Nov 4 '11 at 13:29
Why do you need to read the entire file again for each line? Maybe if you told what you're trying to accomplish someone might be able to suggest a better approach. –  Juhana Nov 4 '11 at 13:32
add comment

5 Answers

up vote 147 down vote accepted

Nobody has given the correct, fully Pythonic way to read a file. It's the following:

with open(...) as f:
    for line in f:
        <do something with line>

The with statement handles opening and closing the file, including if an exception is raised in the inner block. The for line in f treats the file object f as an iterable, which automatically uses buffered IO and memory management so you don't have to worry about large files.


share|improve this answer
yep, this is the best version with python 2.6 and above –  Simon Nov 4 '11 at 13:54
I personally prefer generators & coroutines for dealing with data pipelines. –  jldupont Nov 4 '11 at 14:30
I am sorry to bother you, but the correct version would be with open(...) as f =) –  newtover Nov 15 '11 at 18:46
@newtover: oopsie, thanks! –  katrielalex Nov 15 '11 at 21:35
what would be the best strategy if a file is a huge text file but with one line and the idea is to process words? –  mfcabrera Dec 18 '13 at 14:48
show 3 more comments

Not clear with your code but I will give pointers to read a huge file in python.

Best method is to use iter & yield.

def readInChunks(fileObj, chunkSize=2048):
    Lazy function to read a file piece by piece.
    Default chunk size: 2kB.
    while True:
        data = fileObj.read(chunkSize)
        if not data:
        yield data

f = open('bigFile')
for chuck in readInChunks(f):

But since your file is line based it might even be better to go with

for line in open('myfile'):

For the sake of completeness - (the below methods are probably not good for reading large files)

In Python, the most common way to read lines from a file is to do the following:

for line in open('myfile','r').readlines():

When this is done, however, the readlines() function loads the entire file into memory as it runs. A better approach for large files is to use the fileinput module, as follows:

import fileinput

for line in fileinput.input(['myfile']):

the fileinput.input() call reads lines sequentially, but doesn't keep them in memory after they've been read or even simply so this, since file in python is iterable.

share|improve this answer
-1 It's basically never a good idea to do for line in open(...).readlines(): <do stuff>. Why would you?! You've just lost all the benefit of Python's clever buffered iterator IO for no benefit. –  katrielalex Nov 4 '11 at 13:44
dude! that's what i go about saying in my answer.No need to downvote. I wanted my answer to be comprehensive to have all the good & bad of reading file in python. read the answer fully before downvoting. –  Srikar Appal Nov 4 '11 at 13:49
it is best to avoid giving bad exemples even if you state clearly that this is bad. Especially if your answer is long and OP is a beginner. Still +1 for completness –  Simon Nov 4 '11 at 13:58
@Srikar: there is a time and a place for giving all the possible solutions to a problem; teaching a beginner how to do file input is not it. Having the correct answer buried at the bottom of a long post full of wrong answers does not good teaching make. –  katrielalex Nov 4 '11 at 14:21
@Srikar: You could make your post significantly better by putting the right way at the top, then mentioning readlines and explaining why it's not a good thing to do (because it reads the file into memory), then explaining what the fileinput module does and why you might want to use it over the other methods, then explaining how chunking the file makes the IO better and giving an example of the chunking function (but mentioning that Python does this already for you so you don't need to). But just giving five ways to solve a simple problem, four of which are wrong in this case, is not good. –  katrielalex Nov 4 '11 at 14:23
show 3 more comments

Katrielalex provided the way to open & read one file.

However the way your algorithm goes it reads the whole file for each line of the file. That means the overall amount of reading a file - and computing the Levenshtein distance - will be done N*N if N is the amount of lines in the file. Since you're concerned about file size and don't want to keep it in memory, I am concerned about the resulting quadratic runtime. Your algorithm is in the O(n^2) class of algorithms which often can be improved with specialization.

I suspect that you already know the tradeoff of memory versus runtime here, but maybe you would want to investigate if there's an efficient way to compute multiple Levenshtein distances in parallel. If so it would be interesting to share your solution here.

How many lines do your files have, and on what kind of machine (mem & cpu power) does your algorithm have to run, and what's the tolerated runtime?

Code would look like:

with f_outer as open(input_file, 'r'):
    for line_outer in f_outer:
        with f_inner as open(input_file, 'r'):
            for line_inner in f_inner:
                compute_distance(line_outer, line_inner)

But the questions are how do you store the distances (matrix?) and can you gain an advantage of preparing e.g. the outer_line for processing, or caching some intermediate results for reuse.

share|improve this answer
-1 This is not an answer! –  katrielalex Nov 15 '11 at 21:37
@katriealex: What is your point? Please be constructive. –  cfi Nov 16 '11 at 6:47
My point is that this post does not contain an answer to the question, just some more questions! IMO it would be better suited as a comment. –  katrielalex Nov 16 '11 at 9:00
@katriealex: Err. Strange. You did see the nested loops, expanding your own answer to fit the actual question? I can remove my questions here from my answer, and there's yet enough content to warrant providing this as a - albeit partial - answer. I could also accept if you'd edit your own answer to include the nested loop example - which was explicitly asked by the question - and then I can remove my own answer happily. But a downvote is something I don't get at all. –  cfi Nov 16 '11 at 9:30
Fair enough; I don't really see demonstrating the nested for loops as an answer to the question but I guess it's pretty strongly targeted at beginners. Downvote removed. –  katrielalex Nov 16 '11 at 17:32
show 2 more comments

this is the canonical way of reading a file in python:

f = open(input_file)
for line in f:

it does not allocate a full list. It iterates over the lines.

share|improve this answer
Thanks ! this works –  384X21 Nov 4 '11 at 13:44
add comment

From the python documentation for fileinput.input():


further, the definition of the function is:

fileinput.FileInput([files[, inplace[, backup[, mode[, openhook]]]]])

reading between the lines, this tells me that files can be a list so you could have something like:

for each_line in fileinput.input([input_file, input_file]):

See here for more information

share|improve this answer
add comment

protected by Srikar Appal Sep 21 '13 at 6:00


Would you like to answer one of these unanswered questions instead?
