
Issue Details (XML | Word | Printable)

Key: QB-2628
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Critical Critical
Assignee: Robin Shen
Reporter: Phong Trinh
Votes: 0
Watchers: 0


Out of Memory in QB Server

Created: 20/Jan/16 05:46 PM   Updated: 30/Apr/16 01:01 AM
Component/s: None
Affects Version/s: 6.1.2
Fix Version/s: None

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown
Environment: Linux

 Description  « Hide
 We just upgraded our QuickBuild server to the latest version of QuickBuild and Java8. Then we are experiencing the memory issue which caused the server down several times since the upgrade. Look like the Java consumes a lot of memory. The physical RAM on the machine is 4GB which is as same as before the upgrade. Is QB compatible with Java8? The server log is as the follow, please advise:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 3145728 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
# Out of Memory Error (os_linux.cpp:2627), pid=21777, tid=140167248578304
# JRE version: OpenJDK Runtime Environment (8.0_65-b17) (build 1.8.0_65-b17)
# Java VM: OpenJDK 64-Bit Server VM (25.65-b01 mixed mode linux-amd64 compressed oops)

Current thread (0x00007f7b54081800): VMThread [stack: 0x00007f7b3aff8000,0x00007f7b3b0f9000] [id=21781]

Stack: [0x00007f7b3aff8000,0x00007f7b3b0f9000], sp=0x00007f7b3b0f7460, free space=1021k
V [libjvm.so+0xa05a3d]
V [libjvm.so+0x4c2c6a]V
V [libjvm.so+0x86441e]
V [libjvm.so+0x8dac2b]
V [libjvm.so+0x8db900]
V [libjvm.so+0x8dbc5b]
V [libjvm.so+0x8d81cc]
V [libjvm.so+0x8d89a8]
V [libjvm.so+0x88d673]
V [libjvm.so+0xa07693]
V [libjvm.so+0xa0bbf7]
V [libjvm.so+0xa0a5e8]
V [libjvm.so+0xa0aab9]
V [libjvm.so+0xa0af11]
V [libjvm.so+0x8670e2]

VM_Operation (0x00007f7acfc359a0): ParallelGCFailedAllocation, mode: safepoint, requested by thread 0x00007f7af4033800

Phong Trinh

 All   Comments   Work Log   Change History      Sort Order:
Robin Shen [21/Jan/16 12:17 AM]
It is not strange that different JVM versions has different memory consumptions. How many memory you've assigned to QuickBuild via wrapper.java.maxmemory option in "conf/wrapper.conf"?

Phong Trinh [21/Jan/16 03:03 AM]
Thank you for promptly response. I am unable to login the server to obtain the current value of maxmemory in a moment. What ideal value for it? Thanks.

Robin Shen [21/Jan/16 03:27 AM]
QB server typically needs more than 2G mem for production use (and may need more if you have thousands of configurations), and if your machine only has 4G physical memory, it might not be sufficient considering other applications in the OS.

Phong Trinh [21/Jan/16 03:42 AM]
Thank you again for promptly response. This machine has 4GB of RAM and dedicates to QB server and local agent. I don't run builds on local agent, it is for maintenance tasks only. I have around a thousand configurations.

Robin Shen [21/Jan/16 05:08 AM]
Please check "conf/wrapper.conf" of both QB server and QB local agent to see what is the value of "wrapper.java.maxmemory" is.

Phong Trinh [28/Jan/16 03:07 PM]
The value of "wrapper.java.maxmemory" is 2048m. I found that QB itself had memory leak, and the issue is fixed 6.1.3.

Robin Shen [29/Jan/16 11:57 AM]
Yes that is true, please upgrade to 6.1.3 to see if it solves the issue at your side.

Phong Trinh [29/Apr/16 02:47 PM]
The issue is fixed after the upgrade. Thanks.