Working efficiently with millions of files

Below, I’m providing relevant parts of my code:



import sqlite3, time
import tarfile, zlib, cStringIO
# lookup function
def tar_lookup(dbpath, file_name):
    cur = sqlite3.connect(dbpath).cursor()
                FROM offset_data as o JOIN file_data as f ON o.file_id=f.file_id
    return cur.fetchall()
# saving to archive
    # open tarfile
    tar =, "w")
    # save files to tar
    for fname, txt in files_generator:
        # compress file content (optionally)
        gztxt = zlib.compress(txt)
        # get tarinfo
        ti = tarfile.TarInfo(fname)
        ti.size  = len(gztxt)
        ti.mtime = time.time()
        # add to tar
        tar.addfile(ti, cStringIO.StringIO(gztxt))
# reading from indexed archive(s)
# NOTE: before you need to run on your archives
    tarfnames = tar_lookup(index_path, file_name)
        tarf = open(tarfn)
        # move pointer to right archive place
        # read tar fragment & uncompress
        txt = zlib.decompress(

TAR random access

I was often challenged with accessing thousands/millions files from network file system (NFS). As I update some of the stored files once in a while, I have decided to store these files in multiple TAR archives.  The data complexity was therefore reduced. But still, there was an issue with random access to the files within each archive.

First, I had a look at tar indexer. Its simplicity is brilliant. Yet, it stores index in raw text file and it can handle only single tar file. Therefore, I have ended up writing my own tar_indexer tool using sqlite3 for index storing and allowing indexing of multiple tar archives. This can be easily incorporated into any Python project.

Note, only raw (uncompressed) tar files are accepted as native tar.gz cannot be random accessed. But you can compress each file using zlib before adding it to tar. At least, this is what I do.

Hope, someone will find it useful:)