title: How to structure code that distributes jobs to threads/nodes in Python?
body: <p>I have python code that takes a bunch of tasks and distributes them to either different threads or different nodes on a cluster. I always end up writing a main script <code>driver.py</code>, that takes two command line arguments: <code>--run-all</code> and <code>--run-task</code>. The first is just a wrapper that iterates through all tasks and then calls <code>driver.py --run-task</code> with each task passed as argument.  Example:</p>
  
  <pre><code>== driver.py ==
  # Determine the current script
  DRIVER = os.path.abspath(__file__)
  (opts, args) = parser.parse_args()
  if opts.run_all is not None:
    # Run all tasks
     for task in opts.run_all.split(","):
       # Call driver.py again with a specific task
       cmd = "python %s --run-task %s" %(DRIVER, task)
       # Execute on system
       distribute_cmd(cmd)
  elif opts.run_task is not None:
    # Run on an individual task
    # code here for processing a task...
  </code></pre>
  
  <p>The user would then call:</p>
  
  <pre><code>$ driver.py --run-all task1,task2,task3,task4
  </code></pre>
  
  <p>And each task would get distributed.</p>
  
  <p>The function <code>distribute_cmd</code> takes a shell executable command and sends in a system-specific way to either a node or a thread. The reason <code>driver.py</code> has to find its own name and call itself is because <code>distribute_cmd</code> needs an executable shell command; it cannot take a function name for example.</p>
  
  <p>This consideration led me to this design, of a driver script having two modes and having to call itself. This has two complications: (1) the script has to find out its own path via <code>__file__</code> and (2) when making this into a Python package, it's unclear where <code>driver.py</code> should go. It's meant to be an executable scripts, but if I put it in <code>setup.py</code>'s <code>scripts=</code>, then I will have to find out where the scripts live (see <a href="https://stackoverflow.com/questions/12975540/correct-way-to-find-scripts-directory-from-setup-py-in-python-distutils">correct way to find scripts directory from setup.py in Python distutils?</a>). This does not seem to be a good solution.</p>
  
  <p>What's an alternative design to this?  Keep in mind that the distribution of tasks has to result in an executable command that can be passed as a string to <code>distribute_cmd</code>.  thanks.</p>