Take the 2-minute tour ×

I have to read a file from a particular line number and i know the line number say "n": i have been thinking of two choice:

1)for i in range(n) fname.readline() k=readline() print k

2)i=0 for line in fname: dictionary[i]=line i=i+1

but i want to know faster alternative as i might have to perform this on different files 20000 times. is there is any other better alternatives??

i even want to know if there are other performance enchancement methods for simple looping,as my code has nested loops

thanking u

share|improve this question
Do you only need the nth line of the file, or do you need all lines up to n? –  danben Jun 10 '10 at 4:52
need only the nth line –  kaushik Jun 10 '10 at 5:26
add comment

3 Answers

up vote 4 down vote accepted

If the files aren't too huge, the linecache module of the standard library is pretty good -- it lets you very directly ask for the Nth line of such-and-such file.

If the files are huge, I recommend something like (warning, untested code):

def readlinenum(filepath, n, BUFSIZ=65536):
  bufs = [None] * 2
  previous_lines = lines_so_far = 0
  with open(filepath, 'b') as f
    while True:
      bufs[0] = f.read(BUFSIZ)
      if not bufs[0]:
        raise ValueError('File %s has only %d lines, not %d',
                         filepath, lines_so_far, n)
      lines_this_block = bufs[0].count('\n')
      updated_lines_count = lines_so_far + lines_this_block
      if n < updated_lines_count:
      previous_lines = lines_so_far
      lines_so_far = updated_lines_count
      bufs[1] = bufs[0]
    if n == lines_so_far:
      # line split between blocks
      buf = bufs[1] + bufs[0]
      delta = n - previous_lines
    else:  # normal case
      buf = bufs[0]
      delta = n = lines_so_far
    f = cStringIO.StringIO(buf)
    for i, line in enumerate(f):
      if i == delta: break
    return line.rstrip()

The general idea is to read in the file as binary, in large blocks (at least as large as the longest possible line) -- the processing (on Windows) from binary to "text" is costly on huge files -- and use the fast .count method of strings on most blocks. At the end we can do the line parsing on a single block (two at most in the anomalous case where the line being sought spans block boundaries).

This kind of code requires careful testing and checking (which I haven't performed in this case), being prone to off-by-one and other boundary errors, so I'd recommend it only for truly huge files -- ones that would essentially bust memory if using linecache (which just sucks up the whole file into memory instead of working by blocks). On a typical modern machine with 4GB bytes of RAM, for example, I'd start thinking about such techniques for text files that are over a GB or two.

Edit: a commenter does not believe that binary reading a file is much faster than the processing required by text mode (on Windows only). To show how wrong this is, let's use the 'U' ("universal newlines") option that forces the line-end processing to happen on Unix machines too (as I don't have a Windows machine to run this on;-). Using the usual kjv.txt file:

$ wc kjv.txt
  114150  821108 4834378 kjv.txt

(4.8 MB, 114 Klines) -- about 1/1000th of the kind of file sizes I was mentioning earlier:

$ python -mtimeit 'f=open("kjv.txt", "rb")' 'f.seek(0); f.read()'
100 loops, best of 3: 13.9 msec per loop
$ python -mtimeit 'f=open("kjv.txt", "rU")' 'f.seek(0); f.read()'
10 loops, best of 3: 39.2 msec per loop

i.e., just about exactly a factor of 3 cost for the line-end processing (this is on an old-ish laptop, but the ratio should be pretty repeatable elsewhere, too).

Reading by a loop on lines, of course, is even slower:

$ python -mtimeit 'f=open("kjv.txt", "rU")' 'f.seek(0)' 'for x in f: pass'
10 loops, best of 3: 54.6 msec per loop

and using readline as the commented mentioned (with less efficient buffering than directly looping on the file) is worst:

$ python -mtimeit 'f=open("kjv.txt", "rU")' 'f.seek(0); x=1' 'while x: x=f.readline()'
10 loops, best of 3: 81.1 msec per loop

If, as the question mentions, there are 20,000 files to read (say they're all small-ish, on the order of this kjv.txt), the fastest approach (reading each file in binary mode in a single gulp) should take about 260 seconds, 4-5 minutes, while the slowest one (based on readline) should take about 1600 seconds, almost half an hour -- a pretty significant difference for many, I'd say most, actual applications.

share|improve this answer
linecache helped me,thank u –  kaushik Jun 10 '10 at 5:26
hmmm, this claim is weird to me - why would text files be slow in Windows? i never had such problems in c or pascal or python... –  Nas Banov Jun 10 '10 at 5:56
@EnTerr, how much experience do you have processing rt vs rb multi-GB files? Reading the file as t means the runtime must turn each \r\n into a \n (and "shift" all subsequent text down 1 byte in the buffer) -- over the gigabytes, it sure mounts up (plus, locating 2-character sequences like \r\n is not as fast, at machine level, as locating the single character \n). I have no Windows machine at hand to confirm, but, try yourself: make up a 4GB text file with 1000 concatenated copies of the King James' Bible text (each 4MB), read it as rb vs rt many times, measure. –  Alex Martelli Jun 10 '10 at 14:21
@Alex, what's 'rt' here? I almost just open file with r or rb mode, and I can't find the right place in the python's library manual for the rt. –  sunqiang Jun 10 '10 at 15:43
@sunqiang, 'rt' is "read, text-mode", and 'rb' is "read, binary-mode". 'r' defaults to the same as 'rt', but "explicit is better than implicit" (especially when one is deliberately contrasting binary vs text modes as I was here;-). I did just check the online docs and you're right that they mistakenly omit mentioning the 't' part (!) but see Python in a Nutshell p. 188 "Binary and text modes" for all details (link to Google Books' online scan follows in my next comment) -- BTW, patches and corrections to the Python online docs are always very welcome! –  Alex Martelli Jun 10 '10 at 17:25
show 11 more comments

Unless you know, or can figure out, the offset of line n in your file (for example, if every line were of a fixed width), you will have to read lines until you get to the nth one.

Regarding your examples:

  • xrange is faster than range since range has to generate a list, whereas xrange uses a generator
  • if you only need line n, why are you storing all of the lines in a dictionary?
share|improve this answer
add comment

Caching a list of offsets of every end-of-line character in the file would cost a lot of memory, but caching roughly one per memory page (generally 4KB) gives mostly the same reduction in I/O, and the cost of scanning a couple KB from a known offset is negligible. So, if your average line length is 40 characters, you only have to cache a list of every 100th end-of-line in the file. Exactly where you draw the line depends on how much memory you have and how fast your I/O is. You might even be able to get away with caching a list of the offsets of every 1000th end-of-line character without a noticeable difference in performance from indexing every single one.

share|improve this answer
add comment

Your Answer


