title: RSS/VSS are growing unstopped until all memory & swap on machine ends
body: <p>We have a weblogic 9.2 server with Java1.5.0.16 on RHEL5.3 that we deploy on it a web service and an Alfresco content management system.</p>
  
  <p>We were running it fine for ~3 years on HP-UX i11.23 and a month ago we moved to Linux RH5.3 and from time to time (it happened 3 times) we noticed that the process is starting to use more and more memory until all the memory and swap on the machine ends.</p>
  
  <p>The process still works fine and all the log files looks normal (as if nothing happened) including GC log.</p>
  
  <pre><code>Glance for process ID 25450:
  
  B0000A Glance C.04.70.000 06:54:05 supra2 x86_64 Current Avg High
  CPU Util SU | 2% 2% 2%
  Disk Util D D | 97% 97% 97%
  Mem Util U U | 98% 98% 98%
  Swap Util U U | 60% 60% 60%
  Resources PID: 25450, java PPID: 25394 euid: 664 User:afspr04
  CPU Usage (util): 5.40 Total RSS : 40.6gb
  User CPU : 3.60 Text VSS : 56kb
  System CPU : 1.80 Data VSS : 66.1gb
  Priority : 15 Stack VSS : 2.0mb
  Nice Value : 0 Total VSS : 66.5gb
  Blocked On : SLEEP
  Major Faults : 235
  Minor Faults : 164
  Processor : 1
  Argv1: weblogic.Server
  Cmd : /opt/java1.5.0_16/bin/java -Dweblogic.Name=dmcmsserver -Doracle.net.tns_admin=/etc -server -javaagent:/opt/MercuryDiagn
  ostics/JavaAgent/DiagnosticsAgent/lib/probeagent.jar -Dprobe.id=supra2_afspr04_dmcms_ear_p4 -Dprobe.group=CMS_SERVER -D
  points.file.name=/opt/MercuryDiagnostics/JavaAgent/DiagnosticsAgent/etc/supra2_afspr04_dmcms_ear_p4 -Dcom.wily.introsco
  pe.agent.agentName=DMCMS -Xms7g -Xmx7g -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=1792m -XX:MaxNewSize=1792m -X
  X:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:+DisableExplicitGC -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Xnoclassg
  c -Xloggc:logs/gc.log -Doracle.net.tns_admin=/etc -Dweblogic.Stderr=/app/afspr04/dmcms_ear_p4/dmcmsdomain/logs/online.l
  og -Dweblogic.Stdout=/app/afspr04/dmcms_ear_p4/dmcmsdomain/logs/online.log -Damdocs.system.home=/app/afspr04/dmcms_ear_
  p4/properties/jesi -Damdocs.messageHandling.home=/app/afspr04/dmcms_ear_p4/properties/jesi -Djesi.config.loader=amdocs.
  ecommerce.esi.utils.config.InterfaceConfigXPathLoader -Damdocs.uams.config.resource=config/mvc/ldap ...
  </code></pre>
  
  <p>pmap shows the big allocation as anonymous
  pmap (sorted by the big once):</p>
  
  <pre><code>25450: /opt/java1.5.0_16/bin/java -Dweblogic.Name=dmcmsserver -Doracle.net.tns_admin=/etc -server -javaagent:/opt/MercuryDiagnostics/JavaAgent/DiagnosticsAgent/lib/probeagent.jar -Dprobe.id=supra2_afspr04_dmcms_ear_p4 -Dprobe.group=CMS_SERVER -Dpoints.file.name=/opt/MercuryDiagnostics/JavaAgent/DiagnosticsAgent/etc/supra2_afspr04_dmcms_ear_p4 -Dcom.wily.introscope.agent.agentName=DMCMS -Xms7g -Xmx7g -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=1792m -XX:MaxNewSize=1792m -XX:SurvivorRatio=4 -XX:TargetSurvivo
  00002ab0f8000000    10518548    rwx--   [anon]
  00002ab798009000    8388612 rwx--   [anon]
  000000005fcce000    8038976 rwx--   [anon]
  00002aac7aab0000    7602176 rwx--   [anon]
  00002aaf74000000    5259284 rwx--   [anon]
  00002ab688000000    4194308 rwx--   [anon]
  00002aae4b930000    1684124 rwx--   [anon]
  00002aab80000000    1314836 rwx--   [anon]
  00002aab20000000    655376  rwx--   [anon]
  00002aac28000000    532488  rwx--   [anon]
  00002aac50000000    524292  rwx--   [anon]
  00002aaaec000000    327696  rwx--   [anon]
  00002aaad8000000    131088  rwx--   [anon]
  00002ab658000000    131060  rwx--   [anon]
  00002ab0dc000000    131044  rwx--   [anon]
  00002aaacc2f5000    114708  rwx--   [anon]
  ...
  total 69733292K 
  </code></pre>
  
  <p>Have anyone encountered something similar?</p>
  
  <p>Thanks,
  Oz</p>