Take the 2-minute tour ×

I work (create, read, copy, delete, count the number of) with millions of files under Windows XP (and NTFS). Are there any other filesystems or OS better optimized for dealing with millions of files? Can you provide any suggestions on how I can significantly speed up things (say, copy/delete/count)? Any useful utilities?

share|improve this question
Millions of files in a directory will kill pretty much any OS and/or filesystem. –  Ignacio Vazquez-Abrams Jun 13 '11 at 22:25
So maybe the approach should be to store data in fewer files? –  Leo Jun 13 '11 at 22:46
Or to use a hierarchy of folders to partition the files. –  Ignacio Vazquez-Abrams Jun 13 '11 at 22:47
add comment

3 Answers

I use FAR Manager when dealing with large numbers of files. It's ideally suited for this sort of task because you can set up filters to limit the number of files you see at one time:

  FAR Manager (free and open source)

Both 32-bit and 64-bit editions are available, which is particularly important when dealing with really large numbers of files since 64-bit processing will yield a definite advantage for you.

enter image description here

share|improve this answer
add comment

Check DiskBoss.


share|improve this answer
add comment
  1. It sounds a design issue. You should revise your solution.
  2. Format your drive(s) with a lower cluster size to save some space.
  3. Total commander is a good utility for this purpose. Maybe you need to develop your own business-oriented solution.
share|improve this answer
Regarding 1: I work with raw files produced in hundreds of thousands by mass spectrometers analyzing protein samples and with results of protein identification in these files. Maybe I should write to the major producers of mass spectrometers and to the developers of protein identification software that they should revise their solutions. –  Leo Jun 14 '11 at 2:51
Odds are good that these software developers are interested in their customers feedback, and if you are paying for support, then this type of request should absolutely be submitted to them. –  Kirk Jun 14 '11 at 3:29
@Leo: So small data is not suitable to be written on file system. A database is more suitable here. –  Xaqron Jun 14 '11 at 3:43
add comment

Your Answer


