title: Distributed state-machine's zookeeper ensemble fails while processing parallel regions with error KeeperErrorCode = BadVersion
body: <p><strong>Background :</strong></p>
  
  <p>Diagram :
  <a href="https://i.stack.imgur.com/Pd6nH.jpg" rel="nofollow noreferrer">Statemachine uml state diagram</a></p>
  
  <p>We have a normal state machine as depicted in diagram that monitors spring-BATCH micro-service(deployed on streams source/processor/sink design) ,for each batch that is started .</p>
  
  <p>We receive sequence of REST calls to  internally fire events per batch id on respective batch's machine object. i.e. per batch id the new state machine object is created .</p>
  
  <p>And each machine is having <code>n</code> number of parallel regions(representing spring batch's chunks ) also as shown in the diagram.</p>
  
  <p>REST calls made are using multi-threaded environment where 2 simultaneous calls of same batchId may come for different region Ids of BATCHPROCESSING state . </p>
  
  <p>Up till now we had a single node(single installation) running of this state machine micro-service but now we want to deploy it on multiple instances ; to receive REST calls .
  For this , the Distributed State Machine is what we want to introduce  . We have below configuration in place for Running Distributed State Machine . </p>
  
  <pre><code>@Configuration
  @EnableStateMachine
  public  class StateMachineUMLWayConfiguration extends 
  StateMachineConfigurerAdapter&lt;String, String&gt; {
  
  ..
  ..
  
  @Override
  public void configure(StateMachineModelConfigurer&lt;String,String&gt; model) 
  throws Exception {
      model
          .withModel()
              .factory(stateMachineModelFactory());
  }
  
  @Bean
  public StateMachineModelFactory&lt;String,String&gt; stateMachineModelFactory() {
  
      StorehubBatchUmlStateMachineModelFactory factory =null;
  
      try {
      factory = new StorehubBatchUmlStateMachineModelFactory
      (templateUMLInClasspath,stateMachineEnsemble());
      } catch (Exception e) {
      LOGGER.info("Config's State machine factory got exception 
      :"+factory);
      }
      LOGGER.info("Config's State machine factory method Called:"+factory);
  
  factory.setStateMachineComponentResolver(stateMachineComponentResolver());
      return factory;
  }
  
  
      @Override
      public void configure(StateMachineConfigurationConfigurer&lt;String, 
  String&gt; 
      config) throws Exception {
      config
          .withDistributed()
              .ensemble(stateMachineEnsemble());
  }
  
  @Bean
  public StateMachineEnsemble&lt;String, String&gt; stateMachineEnsemble() throws 
  Exception {
      return new ZookeeperStateMachineEnsemble&lt;String, String&gt;(curatorClient(), "/batchfoo1", true, 512);
  }
  
  @Bean
      public CuratorFramework curatorClient() throws Exception {
          CuratorFramework client = 
  CuratorFrameworkFactory.builder().defaultData(new byte[0])
                  .retryPolicy(new ExponentialBackoffRetry(1000, 3))
                  .connectString("localhost:2181").build();
          client.start();
          return client;
      }
  </code></pre>
  
  <p>StorehubBatchUmlStateMachineModelFactory's build method:</p>
  
  <pre><code>    @Override
      public StateMachineModel&lt;String, String&gt; build(String batchChunkId) {
  
      Model model = null;
      try {
          model = UmlUtils.getModel(getResourceUri(resolveResource(batchChunkId)).getPath());
      } catch (IOException e) {
          throw new IllegalArgumentException("Cannot build model from resource " + resource + " or location " + location, e);
      }
      UmlModelParser parser = new UmlModelParser(model, this);
      DataHolder dataHolder = parser.parseModel();
      ConfigurationData&lt;String, String&gt; configurationData = new ConfigurationData&lt;String, String&gt;( null, new SyncTaskExecutor(),
              new ConcurrentTaskScheduler() , false, stateMachineEnsemble,
              new ArrayList&lt;StateMachineListener&lt;String, String&gt;&gt;(), false,
              null, null,
              null, null, false,
              null , batchChunkId, null,
              null ) ;
      return new DefaultStateMachineModel&lt;String, String&gt;(configurationData, dataHolder.getStatesData(), dataHolder.getTransitionsData());
  }
  </code></pre>
  
  <p>Created new custom service interface level method in place of <code>DefaultStateMachineService.acquireStateMachine(machineId)</code></p>
  
  <pre><code>@Override
  public StateMachine&lt;String, String&gt; acquireDistributedStateMachine(String machineId, boolean start) {
  
      synchronized (distributedMachines) {
          DistributedStateMachine&lt;String,String&gt; distributedStateMachine = distributedMachines.get(machineId); 
          StateMachine&lt;String,String&gt; distMachineDelegateX = null;
          if (distributedStateMachine == null) { 
  
              StateMachine&lt;String, String&gt; machine = stateMachineFactory.getStateMachine(machineId);
              distributedStateMachine = (DistributedStateMachine&lt;String, String&gt;) machine;
  
          }
          distributedMachines.put(machineId, distributedStateMachine);
  
          return handleStart(distributedStateMachine, start);
      }
  }
  </code></pre>
  
  <p><strong>Problem :</strong></p>
  
  <p>Now problem is that , micro service deployed on single instance runs successfully even for events received by it are from multi threaded environment where one thread hits with the event REST call belonging to Region 1 and simultaneously other thread comes for region 2 of same batch . Machine  goes ahead in synch ,with successful parallel regions' processing , till its last state i.e BATCHCOMPLETED .
  Also we checked at zookeeper side that at last the BATCHCOMPLETED STATE was being recorded in node's current version.</p>
  
  <p>But , besides 1st instance , when we keep same micro service app-jar deployed on some other location to treat it as a 2nd instance of micro-service that is also now running to accept event REST calls(say by listening at another tomcat port 9002) ; it fails in middle somewhere randomly . This failure happens randomly after any one of the events among parallel regions is fired and when <code>ensemble.setState()</code> is being called internally on state change of that event .</p>
  
  <p>It gives following error:</p>
  
  <pre><code>    [36mo.s.s.support.AbstractStateMachine      [0;39m [2m:[0;39m Interceptors threw exception, skipping state change
  
  org.springframework.statemachine.StateMachineException: Error persisting data; nested exception is org.springframework.statemachine.StateMachineException: Error persisting data; nested exception is org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion
          at org.springframework.statemachine.zookeeper.ZookeeperStateMachineEnsemble.setState(ZookeeperStateMachineEnsemble.java:241) ~[spring-statemachine-zookeeper-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
          at org.springframework.statemachine.ensemble.DistributedStateMachine$LocalStateMachineInterceptor.preStateChange(DistributedStateMachine.java:209) ~[spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.StateMachineInterceptorList.preStateChange(StateMachineInterceptorList.java:101) ~[spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.AbstractStateMachine.callPreStateChangeInterceptors(AbstractStateMachine.java:859) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.AbstractStateMachine.switchToState(AbstractStateMachine.java:880) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.AbstractStateMachine.access$500(AbstractStateMachine.java:81) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.AbstractStateMachine$3.transit(AbstractStateMachine.java:335) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.DefaultStateMachineExecutor.handleTriggerTrans(DefaultStateMachineExecutor.java:286) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.DefaultStateMachineExecutor.handleTriggerTrans(DefaultStateMachineExecutor.java:211) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.DefaultStateMachineExecutor.processTriggerQueue(DefaultStateMachineExecutor.java:449) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.DefaultStateMachineExecutor.access$200(DefaultStateMachineExecutor.java:65) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.DefaultStateMachineExecutor$1.run(DefaultStateMachineExecutor.java:323) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
          at org.springframework.statemachine.support.DefaultStateMachineExecutor.scheduleEventQueueProcessing(DefaultStateMachineExecutor.java:352) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.DefaultStateMachineExecutor.execute(DefaultStateMachineExecutor.java:163) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.AbstractStateMachine.sendEventInternal(AbstractStateMachine.java:603) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.support.AbstractStateMachine.sendEvent(AbstractStateMachine.java:218) [spring-statemachine-core-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
          at org.springframework.statemachine.ensemble.DistributedStateMachine.sendEvent(DistributedStateMachine.java:108) 
  ..skipping Lines....
  Caused by: org.springframework.statemachine.StateMachineException: Error persisting data; nested exception is org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion
      at org.springframework.statemachine.zookeeper.ZookeeperStateMachinePersist.write(ZookeeperStateMachinePersist.java:113) ~[spring-statemachine-zookeeper-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
      at org.springframework.statemachine.zookeeper.ZookeeperStateMachinePersist.write(ZookeeperStateMachinePersist.java:50) ~[spring-statemachine-zookeeper-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
      at org.springframework.statemachine.zookeeper.ZookeeperStateMachineEnsemble.setState(ZookeeperStateMachineEnsemble.java:235) ~[spring-statemachine-zookeeper-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
      ... 73 common frames omitted
  Caused by: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion
  at org.apache.zookeeper.KeeperException.create(KeeperException.java:115) ~[zookeeper-3.4.8.jar!/:3.4.8--1]
  at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1006) ~[zookeeper-3.4.8.jar!/:3.4.8--1]
  at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910) ~[zookeeper-3.4.8.jar!/:3.4.8--1]
  at org.apache.curator.framework.imps.CuratorTransactionImpl.doOperation(CuratorTransactionImpl.java:159)
  </code></pre>
  
  <p><strong>Question :</strong><br>
  <strong>1.So is the configuration mentioned  above needs something more to be configured to avoid that exception mentioned above??</strong> 
  Because Both state-machine micro-service instances were tested with the case when they both were connecting to same instance i.e. same string <code>.connectString("localhost:2181").build()</code> or case when they were made to connect to different zookeeper instances(i.e. 'localhost:2181' , 'localhost:2182'). </p>
  
  <p>Same exception of BAD VERSION occurs during state machine ensemble's processing in both cases .</p>
  
  <p><strong>2.Also If Batches would run in parallel so their respective machines would need to be created to run in parallel at state-machine micro-service end .</strong> 
  So here , technically new State machine we need for new batchId , running simultaneously .
  But looking at the ZookeeperStateMachineEnsemble , One znode path seems to be associated with one ensemble  , whenever ensemble object is instantiated once in the main config class ("StateMachineUMLWayConfiguration") . </p>
  
  <p><strong>So is it expected to only use that singleton ensemble instance only? Can't multiple ensembles be created at run-time referencing different znode paths run in parallel to log their respective Distributed State Machine's states to their respective znode paths??</strong></p>
  
  <p><strong>a.</strong> Because batches running in parallel would need separate znode paths to be created . Thus due to our attempt of keeping separate znode path per batch , we need separate ensemble to be instantiated per batch's machine. But that seems to be getting into the lock condition while getting connection to znode through curator client.</p>
  
  <p><strong>b.</strong> REST call fired for event triggering does not complete , as the machine it acquired is stuck in ensemble to connect .</p>
  
  <p><strong>Thanks in advance .</strong></p>