Announcement Announcement Module
No announcement yet.
Modeling two dependent parallel steps Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Modeling two dependent parallel steps

    I have a process that seems like it could be solved nicely with spring-batch but I haven't been able to find an example which shows how to model the following scenario:

    Step 1 - Takes 1 day to complete.
    Step 2 - Takes 2 days to complete.
    = Total time is 3 days if executed in series.

    Step 2 consumes the individual items written to the db by Step 1. Further I know that Step 2 will consume the output at a much slower rate than Step 1. If I were to run them in parallel the total process time would be closer to 2 days. The closest configuration I have seen to the ideal parallel solution is to use <split> along with <flow>:

    <split id="split1" next="step3" task-executor="taskExecutor">
          <step id="step1"/>
          <step id="step2"/>
    <step id="step3" />
    It is my understanding that this will cause step1 to execute in parallel with step2 but how would I prevent Step 2 from completing while Step 1 is still running (in the case that somehow step2 got ahead of step1 and temporarily didn't have input to read)?

    I considered merging these two steps into a single step and performing "Step 2" in the writer however it seems like I lose some of the power of spring-batch by adding retry/backoff/threadpooling in java code instead of configuring it via the spring context.

    I also considered separating the steps into two jobs where I kick off job1 before job2 but the original sync problem remains as well as having to maintain the dependency between the two jobs.

    Any thoughts on what I am missing?