title: Java: how to repair a hung thread?
body: <p><strong>Please note:</strong> I'm tagging this with JClouds because if you read the entire question and comments that ensue, I believe this to be either a bug with JClouds or a misuse of that library.</p>
  
  <p>I have an executable JAR that runs, works for a while, finishes the work without throwing any errors/exceptions, and then hangs forever when it should be exiting. I profiled it with VisualVM (paying attention to the running threads), and I also tossed in a log statement to print at the point (at the end of the <code>main()</code> method) where the app hangs.  Here is the last part of my main method:</p>
  
  <pre><code>Set&lt;Thread&gt; threadSet = Thread.getAllStackTraces().keySet();
  for(Thread t : threadSet) {
      String daemon = (t.isDaemon()? "Yes" : "No");
      System.out.println("The ${t.getName()} thread is currently running; is it a daemon? ${daemon}.");
  }
  </code></pre>
  
  <p>When my JAR executes this code, I see the following output:</p>
  
  <pre><code>The com.google.inject.internal.util.Finalizer thread is currently running; is it a daemon? Yes.
  The Signal Dispatcher thread is currently running; is it a daemon? Yes.
  The RMI Scheduler(0) thread is currently running; is it a daemon? Yes.
  The Attach Listener thread is currently running; is it a daemon? Yes.
  The user thread 3 thread is currently running; is it a daemon? No.
  The Finalizer thread is currently running; is it a daemon? Yes.
  The RMI TCP Accept-0 thread is currently running; is it a daemon? Yes.
  The main thread is currently running; is it a daemon? No.
  The RMI TCP Connection(1)-10.10.99.8 thread is currently running; is it a daemon? Yes.
  The Reference Handler thread is currently running; is it a daemon? Yes.
  The JMX server connection timeout 24 thread is currently running; is it a daemon? Yes.
  </code></pre>
  
  <p>I don't <em>think</em> I have to worry about daemons (correct me if I'm wrong), so filtering that to non-daemons:</p>
  
  <pre><code>The user thread 3 thread is currently running; is it a daemon? No.
  The main thread is currently running; is it a daemon? No.
  </code></pre>
  
  <p>Obviously, the main thread is still running because something is preventing it from exiting. Hmmm, <code>user thread 3</code> looks interesting. What does VisualVM tell us?</p>
  
  <p><img src="https://i.stack.imgur.com/HWiMm.png" alt="enter image description here"></p>
  
  <p>This is the thread view at the point that the app was hanging (what was happening while the console output above was printing). Hmmm <code>user thread 3</code> is looking even more suspicious!</p>
  
  <p>So before killing the app I took a thread dump. Here is the stacktrace for <code>user thread 3</code>:</p>
  
  <pre><code>"user thread 3" prio=6 tid=0x000000000dfd4000 nid=0x2360 waiting on condition [0x00000000114ff000]
      java.lang.Thread.State: WAITING (parking)
          at sun.misc.Unsafe.park(Native Method)
          - parking to wait for  &lt;0x0000000782cba410&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:744)
  
      Locked ownable synchronizers:
          - None
  </code></pre>
  
  <p>I've never had to analyze one of these before so it means gibberish to me (but perhaps not to a trained eye!).</p>
  
  <p>After killing the app, VisualVM's timeline stops ticking/incrementing every second, and I can scroll horizontally backwards in the timeline to where <code>user thread 3</code> was created and began it's life as a nagging thread:</p>
  
  <p><img src="https://i.stack.imgur.com/BdjC7.png" alt="enter image description here"></p>
  
  <p>However I cannot figure out how to tell where in the code <code>user thread 3</code> is being created. So I ask:</p>
  
  <ul>
  <li>How can I tell what is creating <code>user thread 3</code>, and where (especially since I <em>suspect</em> its a 3rd party OSS lib that is creating the thread) it is being created?</li>
  <li>How can I triage, diagnose and fix this thread hanging?</li>
  </ul>
  
  <h3>Update:</h3>
  
  <p>Here is my code that is firing <em>around the same time</em> that <code>user thread 3</code> seems to be getting created:</p>
  
  <pre><code>ExecutorService myExecutor = Executors.newCachedThreadPool();
  for(Node node : nodes) {
      BootstrapAndKickTask bootAndKickTask = new BootstrapAndKickTask(node, ctx);
      myExecutor.execute(bootAndKickTask);
  }
  
  myExecutor.shutdown();
  if(!myExecutor.awaitTermination(15, TimeUnit.MINUTES)) {
      TimeoutException toExc = new TimeoutException("Hung after the 15 minute timeout was reached.");
      log.error(toExc);
  
      throw toExc;
  }
  </code></pre>
  
  <p>Also here is my <a href="https://gist.github.com/anonymous/bed912d92a171b670061" rel="nofollow noreferrer">GitHub Gist</a> which contains the full thread dump.</p>