Take the 2-minute tour ×

I have a very large directory of files and folders. Currently, I scan the entire directory for JPGs and store them in a list. This is really slow due to the size of the directory. Is there a faster, more efficient way to do this? Perhaps without scanning everything?

My directory looks like this:



share|improve this question
Depends. What does the directory layout look like? –  larsmans Jun 5 '12 at 13:07
I don't know your constraints, but you could perhaps use the updatedb command once and then use the output of locate /library/Modified as the input for your program. –  Lauritz V. Thaulow Jun 5 '12 at 13:18
Since it seems to be mostly historic data, you could save the list for previous years to a file so that it only has be to scanned once, then append a folder scan for the current year. –  Junuxx Jun 5 '12 at 13:22
+1 for caching approach –  schlamar Jun 5 '12 at 13:55

3 Answers 3

See Generator Tricks for System Programmers for a bunch of neat stuff. But specifically, see the gen-find example. This is as efficient as you are going to get, without making a bunch of assumptions about your file structure layout.

share|improve this answer
And how does this help when randomness comes into play? –  schlamar Jun 5 '12 at 15:40
@ms4py what kind of randomness? Hardware failure? –  Spencer Rathbun Jun 5 '12 at 17:23
See the title! How do you want to select 100 items randomly from a generator? –  schlamar Jun 6 '12 at 6:09

Assuming that you application is the only one changing directory and that you have control over the directory names/structure and that you have to do the operation described in your question more than once:

Rename all the files once so you can access them in predictable order. Say, give all files numeric name from 1 to N (where N is the number of files in directory) and have a special file ".count" which will hold the N for each directory. Then access them directly with their names generated by random generator.

share|improve this answer

I don't know where the slowness occurs, but to scan directories and files I found it much faster the dump the directories/files into a text file first using a batch file then get python to read the file. This worked well on our server system with 7 servers and many thousands of directories. Python could, of course, run the batch file.

share|improve this answer

Your Answer


