title: Python os.system call slow ~1/2 second
body: <p>I'm writing a script to find all duplicate files in two different file trees. The script works fine except it's too slow to be practical on large numbers of files (>1000). Profiling my script with cProfile revealed that a single line in my code is responsible for almost all of the execution time.</p>
  
  <p>The line is a call to os.system():</p>
  
  <pre><code>cmpout = os.system("cmp -s -n 10MiB %s %s" % (callA, callB));
  </code></pre>
  
  <p>This call is inside a for loop that gets called about N times if I have N identical files. The average execution time is 0.53 seconds</p>
  
  <pre><code>  ncalls  tottime  percall  cumtime  percall filename:lineno(function)        
    563  301.540    0.536  301.540    0.536 {built-in method system}
  </code></pre>
  
  <p>This of course quickly adds up for more than a thousand files. I have tried speeding it up by replacing it with call from the subprocess module:</p>
  
  <pre><code>cmpout = call("cmp -s -n 10MiB %s %s" % (callA, callB), shell=True); 
  </code></pre>
  
  <p>But this has a near identical execution time. I have also tried reducing the byte limit on the cmp command itself, but this only saves a very small amount of time.</p>
  
  <p>Is there anyway I can speed this up?</p>
  
  <p>Full function I'm using: </p>
  
  <pre><code>def dirintersect(dirA, dirB):
      intersectionAB = []
      filesA = listfiles(dirA);
      filesB = listfiles(dirB);
      for (pathB, filenameB) in filesB:
          for (pathA, filenameA) in filesA:
              if filenameA == filenameB:
                  callA = shlex.quote(os.path.join(pathA, filenameA));
                  callB = shlex.quote(os.path.join(pathB, filenameB));
                  cmpout = os.system("cmp -s -n 10MiB %s %s" % (callA, callB));
                  #cmpout = call("cmp -s -n 10MiB %s %s" % (callA, callB), shell=True); 
                  if cmpout is 0:
                      intersectionAB.append((filenameB, pathB, pathA))                
      return intersectionAB
  </code></pre>
  
  <p><strong>Update</strong>: Thanks for all the feedback! I will try to address most of your comments and give some more information.
  @Iarsmans. You're absolutely right that my nested for loop scales with n² I had already figured out myself that I could do the same by using a dictionary or set and do set operations. But even the overhead of this 'bad' algorithm is insignificant to the time it takes to run os.system . The actual if clause triggers approximately once for each filename (that is I expect there to be only one duplicate for each filename). So os.system only gets run N times and not N² times, but even for this linear time it isn't fast enough.</p>
  
  <p>@Iarsman and @Alex Reynolds: The reason I didn't choose for a hashing solution like you suggest is because in the usage case I envision I compare a smaller directory tree with a larger one and hashing all of the files in the larger tree would take a very long time (as it could be all the files in an entire partition), while I would only require to do the actual comparison on a small fraction of the files.</p>
  
  <p>@abarnert: The reason I use shell=True in the call command is simply because I started with os.system and then read that it was better to use subprocess.call and this was the way to convert between the too. If there is a better way to run the cmp command, I'd like to know. The reason I qoute the arguments is because I had issues with spaces in filenames when I just passed the os.path.join result in the command.</p>
  
  <p>Thanks for your suggestion, I will change it to <code>if cmpout == 0</code></p>
  
  <p>@Gabe: I don't know how to time a bash command, but I believe it runs much faster than half a second when I just run the command.</p>
  
  <p>I said the byte limit didn't matter much, because when I changed it to only 10Kib it changed the total execution time of my test run to 290seconds instead of around 300s. The reason I kept the limit is to prevent it from comparing really large files (such as 1GiB video files). </p>
  
  <p><strong>Update 2:</strong>
  I have followed @abarnert 's suggestion and changed the call to:</p>
  
  <pre><code>cmpout = call(["cmp", '-s', '-n', '10MiB', callA, callB])
  </code></pre>
  
  <p>The execution time for my test scenario has now dropped to 270s from 300seconds. Not sufficient yet, but it's a start.</p>