title: Iterate over a very large number of files in a folder
body: <p>What is the fastest way to iterate over all files in a directory using NTFS and Windows 7, when the filecount in the directory is bigger than 2.500.000?
  All Files are flat under the top-level directory.</p>
  
  <p>Currently I use </p>
  
  <pre><code>for root, subFolders, files in os.walk(rootdir):
      for file in files:
          f = os.path.join(root,file)
          with open(f) as cf:
              [...]
  </code></pre>
  
  <p>but it is very very slow. The process has been running for about an hour and still has not processed a single file but still grows with about 2kB of Memory Usage per second.</p>