title: Python Workflow Design Pattern
body: <p>I'm working on a piece of software design, and I'm stuck between not having any idea what I'm doing, and feeling like I'm reinventing the wheel.</p>
  
  <p>My situation is the following: I am designing a scientific utility with an interactive UI. User input should trigger visual feedback (duh), some of it directly, i.e. editing a domain geometry, and some of it as soon as possible, without blocking user interaction, say, solving some PDE over said domain.</p>
  
  <p>If I draw out a diagram of all operations I need to perform, I get this rather awesomely dense graph, exposing all kinds of opportunities for parallelism and caching/reuse of partial results. So what I want is primarily to exploit this parallelism in a transparent way (selected subtasks executing in seperate processes, results outmatically being 'joined' by downstream tasks waiting for all their inputs to be ready), plus only needing to recompute those input branches that actually have their input changed</p>
  
  <p>pyutilib.workflow seems to come closest to being what I'm looking for, except of course that it isn't (doesn't seem to do any subprocessing to begin with). That seems rather disappointing; while I'm not a software engineer, id say I'm not asking for anything crazy here.</p>
  
  <p>Another complicating factor is the tight user-interface integration I desire, which other scientific-workflow-solutions  seem not designed to handle. For instance, I would like to pass a drag-and-drop event through a transformation node for further processing. The transformation node has two inputs; an affine transform state input port, and a pointset class that knows what to do with it. If the affine transform input port is 'dirty' (waiting for its dependencies to update), the event should be held up until it becomes available. But when the event has passed the node, the eventinput port should be marked as handled, so it does not refire when the affine transform changes due to further user input. That's just an example of one of the many issues that come up that I don't see being adressed anywhere. Or what to do when a long-running forking-joining branch receives new input while it is in the middle of crunching a previous input.</p>
  
  <p>So my question: Do you happen to know of some good books/articles on workflow design patterns that I should read? Or am I trying to fit a square peg into a round hole, and you know of a completely different design pattern that I should know about? Or a python package that does what I want it to, regardless of the buzzwords it comes dressed up in?</p>
  
  <p>Ive rolled by own solution on top of enthought.traits, but I'm not perfectly happy with that either, as it feels like a rough and shoddy reinvention of the wheel. Except that I cant seem to find any wheels anywhere on the internet.</p>
  
  <p>NOTE: I'm not looking for webframeworks, graphical workflow designers, or any special-purpose tools. Just something conceptually like pyutilib.workflow, but including documentation and a featureset that I can work with.</p>
  
  <p>#
  #
  #
  EDIT: this is where I'm at after more reading and reflection on the issue:
  #
  #
  #</p>
  
  <p>The requirements one can tack onto a 'workflow architecture' are too diverse for there to be a single shoe that fits all. Do you want tight integration with disk storage, tight integration with web frameworks, asynchronicity, mix in custom finite state machine logic for task dispatch? They are all valid requirements, and they are largely incompatible, or make for senseless mixes. </p>
  
  <p>However, not all is lost. Looking for a generic workflow system to solve an arbitrary problem is like looking for a generic iterator to solve your custom iteration problem. Iterators are not primarily about reusability; you cant reuse your red-black-tree iterator to iterate over your tensor. Their strength lies in a clean separation of concerns, and definition of a uniform interface.</p>
  
  <p>What I'm looking for (and have started writing myself; its going to be pretty cool) will look like this: at its base is a general implementation-agnostic workflow-declaration mini-language, based on decorators and some meta-magic, to transform a statement like the below into a workflow declaration containing all required information:</p>
  
  <pre><code>@composite_task(inputs(x=Int), outputs(z=Float))
  class mycompositetask:
      @task(inputs(x=Int), outputs(y=Float))
      def mytask1(x):
          return outputs( y = x*2 )
      @task(inputs(x=Int, y=Float), outputs(z=Float))
      def mytask2(x, y):
          return outputs( z = x+y )
      mytask1.y = mytask2.y   #redundant, but for illustration; inputs/outputs matching in name and metadata autoconnect
  </code></pre>
  
  <p>What the decorators return is a task/compositetask/workflow <em>declaration</em> class. Instead of just type constraints, other metadata required for the workflow-type at hand is easily added to the syntax.</p>
  
  <p>Now this concise and pythonic declaration can be fed into a workflow instance factory that returns the actual workflow instance. This declaration language is fairly general and probably need not change much between different design requirements, but such a workflow instantiation factory is entirely up to your design requirements/imagination, aside from a common interface for delivering/retrieving input/output.</p>
  
  <p>In its simplest incarnation, wed have something like:</p>
  
  <pre><code>wf   = workflow_factory(mycompositetask)
  wf.z = lambda result: print result   #register callback on z-output socket
  wf.x = 1    #feed data into x input-socket
  </code></pre>
  
  <p>where wf is a trivial workflow instance, which does nothing but chain all contained function bodies together on the same thread, once all inputs are bound. A quite verbose way to chain two functions, but it illustrates the idea, and it already achieves the goal of separating the concern of keeping the definition of the flow of information in a central place rather than spread all throughout classes that would rather have nothing to do with it.</p>
  
  <p>That's more or less the functionality I've got implemented so far, but it means I can go on working on my project, and in due time ill add support for fancier workflow instance factories. For instance, I'm thinking of analyzing the graph of dependencies to identify forks and joins, and tracking the activity generated by each input supplied on the workflow-instance level, for elegant load balancing and cancellation of the effects of specific inputs that have lost their relevance but are still hogging resources.</p>
  
  <p>Either way, I think the project of separating workflow declaration, interface definition, and implementation of instantiation is a worthwhile effort. Once I have a few nontrivial types of workflow instances working well (I need at least two for the project I'm working on, I've realized*), I hope to find the time to publish this as a public project, because despite the diversity of design requirements in workflow systems, having this groundwork covered makes implementing your own specific requirements a lot simpler. And instead of a single bloated workflow framework, a swiss army knife of easily switched-out custom solutions could grow around such a core.</p>
  
  <p>*realizing that I need to split my code over two different workflow instance types rather than trying to bash all my design requirements into one solution, turned the square peg and round hole I had in my mind into two perfectly complementary holes and pegs.</p>