title: Python rapidly creating and removing directories will cause WindowsError [Error 5] intermittently
body: <p>I encountered this problem while using Scrapy's <code>FifoDiskQueue</code>. In windows, <code>FifoDiskQueue</code> will cause directories and files to be created by one file descriptor and consumed (and if no more message in the queue, removed) by another file descriptor.</p>
  
  <p>I will get error messages like the following, randomly:</p>
  
  <pre><code>2015-08-25 18:51:30 [scrapy] INFO: Error while handling downloader output
  Traceback (most recent call last):
    File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 588, in _runCallbacks
      current.result = callback(current.result, *args, **kw)
    File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 154, in _handle_downloader_output
      self.crawl(response, spider)
    File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 182, in crawl
      self.schedule(request, spider)
    File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 188, in schedule
      if not self.slot.scheduler.enqueue_request(request):
    File "C:\Python27\lib\site-packages\scrapy\core\scheduler.py", line 54, in enqueue_request
      dqok = self._dqpush(request)
    File "C:\Python27\lib\site-packages\scrapy\core\scheduler.py", line 83, in _dqpush
      self.dqs.push(reqd, -request.priority)
    File "C:\Python27\lib\site-packages\queuelib\pqueue.py", line 33, in push
      self.queues[priority] = self.qfactory(priority)
    File "C:\Python27\lib\site-packages\scrapy\core\scheduler.py", line 106, in _newdq
      return self.dqclass(join(self.dqdir, 'p%s' % priority))
    File "C:\Python27\lib\site-packages\queuelib\queue.py", line 43, in __init__
      os.makedirs(path)
    File "C:\Python27\lib\os.py", line 157, in makedirs
      mkdir(name, mode)
  WindowsError: [Error 5] : './sogou_job\\requests.queue\\p-50'
  </code></pre>
  
  <p>In Windows, <strong>Error 5</strong> means <strong>access is denied</strong>. A lot of explanations on the web quote the reason as lacking administrative rights, like <a href="http://answers.microsoft.com/en-us/windows/forum/windows_7-windows_programs/itunes-installation-error-windows-error-code-5/6ed1090b-de3e-43c3-9d58-5ca8652f1390" rel="noreferrer">this MSDN post</a>. But the reason is not related to access rights. When I run the <code>scrapy crawl</code> command in a <code>Administrator command prompt</code>, the problem still occurs.</p>
  
  <p>I then created a small test like this to try on windows and linux:</p>
  
  <pre><code>#!/usr/bin/python
  
  import os
  import shutil
  import time
  
  for i in range(1000):
      somedir = "testingdir"
      try:
          os.makedirs(somedir)
          with open(os.path.join(somedir, "testing.txt"), 'w') as out:
              out.write("Oh no")
          shutil.rmtree(somedir)
      except WindowsError as e:
          print 'round', i, e
          time.sleep(0.1)
          raise
  </code></pre>
  
  <p>When I run this, I will get:</p>
  
  <pre><code>round 13 [Error 5] : 'testingdir'
  Traceback (most recent call last):
    File "E:\FHT360\FHT360_Mobile\Source\keywordranks\test.py", line 10, in &lt;module&gt;
      os.makedirs(somedir)
    File "C:\Users\yj\Anaconda\lib\os.py", line 157, in makedirs
      mkdir(name, mode)
  WindowsError: [Error 5] : 'testingdir'
  </code></pre>
  
  <p>The <code>round</code> is different every time. So if I remove <code>raise</code> in the end, I will get something like this:</p>
  
  <pre><code>round 5 [Error 5] : 'testingdir'
  round 67 [Error 5] : 'testingdir'
  round 589 [Error 5] : 'testingdir'
  round 875 [Error 5] : 'testingdir'
  </code></pre>
  
  <p>It simply fails randomly, with a small probability, <strong>ONLY in Windows</strong>. I tried this test script in cygwin and linux, this error never happens there. I also tried the same code in another Windows machine and it occurs there.</p>
  
  <p>What are possible reasons for this?</p>
  
  <p>[Update]
  Screenshot of proof [管理员 means Administrator in Chinese]:
  <a href="https://i.stack.imgur.com/05GWq.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/05GWq.jpg" alt="enter image description here"></a></p>
  
  <p>Also proof that the test case still fails in an administrator command prompt:</p>
  
  <p><a href="https://i.stack.imgur.com/H5AEO.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/H5AEO.jpg" alt="enter image description here"></a></p>
  
  <p>@pss said that he couldn't reproduce the issue. I tried our Windows 7 Server. I installed a fresh new python 2.7.10 64-bit. I had to set a really large upper bound for round and only started to see errors appearing after round 19963:</p>
  
  <p><a href="https://i.stack.imgur.com/ickSd.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/ickSd.jpg" alt="enter image description here"></a></p>