title: Threads blocking while accessing Mojarra JSF Flash in a clustered environment
body: <p>I'm running a large Java EE 7 application on a two-node JBoss cluster which was recently upgraded from JBoss EAP 6 to JBoss EAP 7.0.4. Intermittently, the application runs into a problem where access becomes very slow, up to the point that the application becomes difficult to access. After a couple of minutes, the issue resolves itself automatically, and operation returns to normal.</p>
  <p>Thread dump analysis shows that the incident is caused by the following behavior:</p>
  <ol>
  <li>A <strong>thread A</strong> tries to write to the session cache, but has to wait to acquire a lock to the cache entry. It is put in <strong>TIMED_WAITING</strong> state.</li>
  <li>A <strong>thread B</strong> tries to obtain the JSF <code>Flash</code> for the same session. In doing so, it tries to update the session, but because <strong>thread A</strong> is already locking the session, it is put in <strong>BLOCKED</strong> state.</li>
  <li><em>All</em> <strong>threads C</strong> that try to obtain the JSF <code>Flash</code> (for <em>any</em> session) are put in <strong>BLOCKED</strong> state, thus making the application unresponsive.</li>
  </ol>
  <p>Step 1 and 2 are caused by the fact that we're using a synchronous distributed Infinispan session cache, and don't really cause a problem as the locking is local to one user session. Step 3, however, is extremely problematic, because a blocked update to <em>one user</em>'s session suddenly impacts <em>all users</em>.</p>
  <p>The stack traces for all <strong>threads C</strong> are identical:</p>
  <blockquote>
  <p>stackTrace:<br />
  java.lang.Thread.State: BLOCKED (on object monitor)   at
  com.sun.faces.context.flash.ELFlash.getFlash(ELFlash.java:318)</p>
  <ul>
  <li>waiting to lock &lt;0x00007ef80a75e260&gt; (a <strong>io.undertow.servlet.spec.ServletContextImpl</strong>)</li>
  </ul>
  </blockquote>
  <p>Now this was very surprising to me. Why would all requests trying to obtain the JSF flash be blocked by the <a href="https://docs.oracle.com/javaee/7/api/javax/servlet/ServletContext.html" rel="nofollow noreferrer"><code>ServletContext</code></a>—which is expected to be <strong>global</strong> to the application? The cause can be found in the Mojarra <a href="https://github.com/javaserverfaces/mojarra/blob/2.2.12/jsf-ri/src/main/java/com/sun/faces/context/flash/ELFlash.java#L318" rel="nofollow noreferrer"><code>ELFlash</code></a> source code (line 318 contains the <code>synchronized</code> statement):</p>
  <pre><code>if (appMap.get(EnableDistributable.getQualifiedName()) != null) {
      synchronized (extContext.getContext()) {
          if (extContext.getSession(false) != null) {
              SessionHelper sessionHelper = SessionHelper.getInstance(extContext);
              if (sessionHelper == null) {
                  sessionHelper = new SessionHelper();
              }
              sessionHelper.update(extContext, flash);
          }
      }
  }
  </code></pre>
  <p>In a nutshell, what's happening here that at some point during the development of Mojarra 2.2.x, the developers <a href="https://github.com/ngriffin7a/mojarra-svn2git-2.2.x/commit/5b17d3bb2551654cf6c16620ab263cc23b5c9f7f" rel="nofollow noreferrer">decided</a> that if a JSF <code>Flash</code> is operated in a distributed environment, the <code>Flash</code> state needs to be stored in the session (if present) so that it can be replicated across the cluster.</p>
  <p>While I don't necessarily take issue with this, I have the following questions:</p>
  <ol>
  <li>What is the reasoning behind synchronizing on the global <code>ServletContext</code>? This seems overly broad and can potentially block a large number of requests.</li>
  <li>Is there any way of changing this behavior, specifically for the <code>Flash</code>? I've pored over the source code, but cannot seem to find a way other than removing <code>&lt;distributable/&gt;</code> from the <code>web.xml</code>, or potentially unsetting the <code>com.sun.faces.enableDistributable</code> context parameter. Both of these options have an impact that is too large for my use case.</li>
  </ol>
  <hr />
  <p><strong>Update</strong></p>
  <p>I've opened a GitHub issue (<a href="https://github.com/javaee/mojarra/issues/4376" rel="nofollow noreferrer">#4376</a>) about this behavior on the Mojarra issue tracker.</p>