title: TomEE chokes on too many @Asynchronous operations
body: <p>I am using Apache TomEE 1.5.2 JAX-RS, pretty much out of the box, with the predefined HSQLDB.</p>
  
  <p>The following is simplified code. I have a REST-style interface for receiving signals:</p>
  
  <pre><code>@Stateless
  @Path("signal")
  public class SignalEndpoint {
      @Inject
      private SignalStore store;
  
      @POST
      public void post() {
          store.createSignal();
      }
  }
  </code></pre>
  
  <p>Receiving a signal triggers a lot of stuff. The store will create an entity, then fire an asynchronous event.</p>
  
  <pre><code>public class SignalStore {
      @PersistenceContext
      private EntityManager em;
  
      @EJB
      private EventDispatcher dispatcher;
  
      @Inject
      private Event&lt;SignalEntity&gt; created;
  
      public void createSignal() {
          SignalEntity entity = new SignalEntity();
          em.persist(entity);
          dispatcher.fire(created, entity);
      }
  }
  </code></pre>
  
  <p>The dispatcher is very simple, and merely exists to make the event handling asynchronous.</p>
  
  <pre><code>@Stateless
  public class EventDispatcher {
      @Asynchronous
      public &lt;T&gt; void fire(Event&lt;T&gt; event, T parameter) {
          event.fire(parameter);
      }
  }
  </code></pre>
  
  <p>Receiving the event is something else, which derives data from the signal, stores it, and fires another asynchronous event:</p>
  
  <pre><code>@Stateless
  public class DerivedDataCreator {
      @PersistenceContext
      private EntityManager em;
  
      @EJB
      private EventDispatcher dispatcher;
  
      @Inject
      private Event&lt;DerivedDataEntity&gt; created;
  
      @Asynchronous
      public void onSignalEntityCreated(@Observes SignalEntity signalEntity) {
          DerivedDataEntity entity = new DerivedDataEntity(signalEntity);
          em.persist(entity);
          dispatcher.fire(created, entity);
      }
  }
  </code></pre>
  
  <p>Reacting to that is even a third layer of entity creation.</p>
  
  <p>To summarize, I have a REST call, which synchronously creates a <code>SignalEntity</code>, which asynchronously triggers the creation of a <code>DerivedDataEntity</code>, which asynchronously triggers the creation of a third type of entity. It all works perfectly, and the storage processes are beautifully decoupled.</p>
  
  <p>Except for when I programmatically trigger a lot (f.e. 1000) of signals in a for-loop. Depending on my <code>AsynchronousPool</code> size, after processing signals (quite fast) in the amount of about half of that size, the application completely freezes for up to some minutes. Then it resumes, to process about the same amount of signals, quite fast, before freezing again.</p>
  
  <p>I have been playing around with <code>AsynchronousPool</code> settings for the last half hour. Setting it to 2000, for instance, will easily make all my signals be processed at once, without any freezes. But the system isn't sane either, after that. Triggering another 1000 signals, resulted in them being created allright, but the entire creation of derived data never happened.</p>
  
  <p>Now I am completely at a loss as to what to do. I can of course get rid of all those asynchronous events and implement some sort of queue myself, but I always thought the point of an EE container was to relieve me of such tedium. Asynchronous EJB events should already bring their own queue mechanism. One which should not freeze as soon as the queue is too full.</p>
  
  <p>Any ideas?</p>
  
  <p><strong>UPDATE:</strong></p>
  
  <p>I have now tried it with 1.6.0-SNAPSHOT. It behaves a little bit differently: It still doesn't work, but I do get an exception:</p>
  
  <pre><code>Aug 01, 2013 3:12:31 PM org.apache.openejb.core.transaction.EjbTransactionUtil handleSystemException
  SEVERE: EjbTransactionUtil.handleSystemException: fail to allocate internal resource to execute the target task
  javax.ejb.EJBException: fail to allocate internal resource to execute the target task
      at org.apache.openejb.async.AsynchronousPool.invoke(AsynchronousPool.java:81)
      at org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:240)
      at org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:86)
      at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:303)
      at &lt;&lt;... my code ...&gt;&gt;
      ...
  Caused by: java.util.concurrent.RejectedExecutionException: Timeout waiting for executor slot: waited 30 seconds
      at org.apache.openejb.util.executor.OfferRejectedExecutionHandler.rejectedExecution(OfferRejectedExecutionHandler.java:55)
      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
      at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
      at org.apache.openejb.async.AsynchronousPool.invoke(AsynchronousPool.java:75)
      ... 38 more
  </code></pre>
  
  <p>It is as though TomEE would not do ANY queueing of operations. If no thread is free to process in the moment of the call, tough luck. Surely, this cannot be intended..?</p>
  
  <p><strong>UPDATE 2:</strong></p>
  
  <p>Okay, I seem to have stumbled upon a semi-solution: Setting the <code>AsynchronousPool.QueueSize</code> property to maxint solves the freeze. But questions remain: Why is the QueueSize so limited in the first place, and, more worryingly: Why would this block the entire application? If the queue is full, it blocks, but as soon as a task is taken from it, another should pop in, right? The queue appears to be blocked until it is completely empty again.</p>
  
  <p><strong>UPDATE 3:</strong></p>
  
  <p>For anyone who wants to have a go: <a href="http://github.com/JanDoerrenhaus/tomeefreezetestcase" rel="nofollow noreferrer">http://github.com/JanDoerrenhaus/tomeefreezetestcase</a></p>
  
  <p><strong>UPDATE 4:</strong></p>
  
  <p>As it turns out, increasing the queue size does NOT solve the problem, it merely delays it. The problem remains the same: Too many asynchronous operations at once, and TomEE chockes so bad, that it cannot even undeploy the application on termination anymore.</p>
  
  <p>So far, my diagnosis is that the task cleanup does not work properly. My tasks are all very small and fast (see the <a href="http://github.com/JanDoerrenhaus/tomeefreezetestcase" rel="nofollow noreferrer">test case on github</a>). I was already afraid that it would be OpenJPA or HSQLDB slowing down on too many concurrent calls, but I commented out all <code>em.persist</code> calls, and the problem remained the same. So if my tasks are quite small and fast, but still manage to block out TomEE so bad that it could not get any further task in after 30 seconds (<code>javax.ejb.EJBException: fail to allocate internal resource to execute the target task</code>), I would imagine that completed tasks linger, clogging up the pipe, so to speak.</p>
  
  <p>How could I resolve this issue?</p>