title: JVM OutOfMemory error "death spiral" (not memory leak)
body: <p>We have recently been migrating a number of applications from running under RedHat linux JDK1.6.0_03 to Solaris 10u8  JDK1.6.0_16 (much higher spec machines) and we have noticed what seems to be a rather pressing problem: under certain loads our JVMs get themselves into a "Death Spiral" and eventually go out of memory. Things to note:</p>
  
  <ul>
  <li>this is <strong>not a case of a memory leak</strong>. These are applications which have been running just fine (in one case for over 3 years) and the out-of-memory errors are not certain in any case. Sometimes the applications work, sometimes they don't</li>
  <li>this is <strong>not us moving to a 64-bit VM</strong> - we are still running 32 bit</li>
  <li>In one case, using the latest G1 garbage collector on 1.6.0_18 seems to have solved the problem. In another, moving back to 1.6.0_03 has worked</li>
  <li>Sometimes our apps are falling over with HotSpot <code>SIGSEGV</code> errors </li>
  <li>This is affecting applications written in Java as well as Scala</li>
  </ul>
  
  <p>The most important point is this: <em>the behaviour manifests itself in those applications which suddenly get a deluge of data (usually via TCP)</em>. It's as if the VM decides to keep adding more data (possibly progressing it to the TG) rather than running a GC on "newspace" until it realises that it has to do a full GC and then, <em>despite practically everything in the VM being garbage, it somehow decides not to collect it!</em> </p>
  
  <p>It sounds crazy but I just don't see what else it is. How else can you explain an app which one minute falls over with a max heap of 1Gb and the next works just fine (never going about 256M when the app is doing <em>exactly the same thing</em>)</p>
  
  <p>So my questions are:</p>
  
  <ol>
  <li>Has anyone else observed this kind of behaviour?</li>
  <li>has anyone any suggestions as to how I might debug the JVM itself (as opposed to my app)? How do I prove this is a VM issue?</li>
  <li>Are there any VM-specialist forums out there where I can ask the VM's authors (assuming they aren't on SO)? (We have no support contract)</li>
  <li>If this is a bug in the latest versions of the VM, how come no-one else has noticed it?</li>
  </ol>