title: parallel directory walk python
body: <p>I need to read every file in the directory tree starting from a given root location. I would like to do this as fast as possible using parallelism. I have 48 cores at my disposal and 1 TB ram, so the thread resources are not an issue. I also need to log every file that was read.</p>
  
  <p>I looked at using joblib but am unable to combine joblib with os.walk. </p>
  
  <p>I can think of two ways:</p>
  
  <ul>
  <li>walk the tree and add all files to a queue or list and have a worker pool of threads dequeue files - best load balancing, maybe more time due to initial walk &amp; queue overhead</li>
  <li>spawn threads and statically assign portions of the tree to each thread - low load balancing, no initial walk, assign directories based on a hash of some sort.</li>
  </ul>
  
  <p>or is there a better way?</p>
  
  <p><strong>EDIT</strong> performance of storage is not a concern. assume there is an infinitely fast storage that can handle infinite number of parallel reads</p>
  
  <p><strong>EDIT</strong> removed multinode situation to keep the focus on parallel directory walk</p>