{"action":"created","issue":{"url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/3282","repository_url":"https://api.github.com/repos/ow2-proactive/scheduling","labels_url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/3282/labels{/name}","comments_url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/3282/comments","events_url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/3282/events","html_url":"https://github.com/ow2-proactive/scheduling/issues/3282","id":347078965,"node_id":"MDU6SXNzdWUzNDcwNzg5NjU=","number":3282,"title":"OOM under no load (dump added)","user":{"login":"febael","id":335810,"node_id":"MDQ6VXNlcjMzNTgxMA==","avatar_url":"https://avatars3.githubusercontent.com/u/335810?v=4","gravatar_id":"","url":"https://api.github.com/users/febael","html_url":"https://github.com/febael","followers_url":"https://api.github.com/users/febael/followers","following_url":"https://api.github.com/users/febael/following{/other_user}","gists_url":"https://api.github.com/users/febael/gists{/gist_id}","starred_url":"https://api.github.com/users/febael/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/febael/subscriptions","organizations_url":"https://api.github.com/users/febael/orgs","repos_url":"https://api.github.com/users/febael/repos","events_url":"https://api.github.com/users/febael/events{/privacy}","received_events_url":"https://api.github.com/users/febael/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-08-02T16:09:15Z","updated_at":"2018-08-03T12:31:12Z","closed_at":null,"author_association":"NONE","body":"Bonjour Mesdames et Messieurs,\r\n\r\nWe get OOM errors at server. [dump file](https://we.tl/cVcReep4WA)\r\n\r\n- We get such errors unexpectedly at random times. It tends to occur less frequent after our switch from version 7.x to 8. Still, We had it several times at periods of several days under no load.\r\n- You don't need to do anything special, just wait with idle 2 local nodes connected to server. We run a few jobs though, but it doesn't look to be related to that.\r\n- We run version 8.0.0 on ubuntu and in a docker container. \r\n- You can check the dump file, What is interesting is, a 1.1 Gb byteArray receive on network and OOM occurring while trying to create a new byteArray to store it. Delving into the array gives an array starting with \"/ HTTP/1.0 \\13\\10\\13\\10\" and more than billion byte value 0's. Call stack of the thread at time of OOM is : \r\n\r\n \r\n\"New I/O worker #2\" prio=5 tid=16 RUNNABLE\r\n\tat java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)\r\n\tat org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)\r\n\tat org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:34)\r\n\tat org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)\r\n\tat org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:68)\r\n\tat org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChannelBufferFactory.java:48)\r\n\tat org.objectweb.proactive.extensions.pnp.PNPServerFrameDecoder.getBuffer(PNPServerFrameDecoder.java:194)\r\n\tat org.objectweb.proactive.extensions.pnp.PNPServerFrameDecoder.decode(PNPServerFrameDecoder.java:138)\r\n\tat org.objectweb.proactive.extensions.pnp.PNPServerFrameDecoder.callDecode(PNPServerFrameDecoder.java:104)\r\n\t   Local Variable: java.net.InetSocketAddress#157\r\n\tat org.objectweb.proactive.extensions.pnp.PNPServerFrameDecoder.handleUpstream(PNPServerFrameDecoder.java:86)\r\n\t   Local Variable: org.objectweb.proactive.extensions.pnp.PNPServerFrameDecoder#3\r\n\t   Local Variable: org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext#20\r\n\t   Local Variable: org.jboss.netty.buffer.BigEndianHeapChannelBuffer#6\r\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\r\n\t   Local Variable: org.jboss.netty.channel.DefaultChannelPipeline#6\r\n\t   Local Variable: org.jboss.netty.channel.UpstreamMessageEvent#1\r\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\r\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)\r\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)\r\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\r\n\t   Local Variable: org.jboss.netty.channel.socket.nio.NioAcceptedSocketChannel#3\r\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\r\n\t   Local Variable: sun.nio.ch.SelectionKeyImpl#40\r\n\t   Local Variable: java.util.HashMap$KeyIterator#1\r\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\r\n\t   Local Variable: sun.nio.ch.EPollSelectorImpl#4\r\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\r\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\r\n\t   Local Variable: org.jboss.netty.channel.socket.nio.NioWorker#3\r\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\r\n\t   Local Variable: java.lang.String#8438\r\n\t   Local Variable: java.lang.String#8439\r\n\t   Local Variable: org.jboss.netty.util.ThreadRenamingRunnable#5\r\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\t   Local Variable: org.jboss.netty.util.internal.DeadLockProofWorker$1#4\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\t   Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#5\r\n\tat java.lang.Thread.run(Thread.java:748)\r\n"},"comment":{"url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/comments/410239073","html_url":"https://github.com/ow2-proactive/scheduling/issues/3282#issuecomment-410239073","issue_url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/3282","id":410239073,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMDIzOTA3Mw==","user":{"login":"febael","id":335810,"node_id":"MDQ6VXNlcjMzNTgxMA==","avatar_url":"https://avatars3.githubusercontent.com/u/335810?v=4","gravatar_id":"","url":"https://api.github.com/users/febael","html_url":"https://github.com/febael","followers_url":"https://api.github.com/users/febael/followers","following_url":"https://api.github.com/users/febael/following{/other_user}","gists_url":"https://api.github.com/users/febael/gists{/gist_id}","starred_url":"https://api.github.com/users/febael/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/febael/subscriptions","organizations_url":"https://api.github.com/users/febael/orgs","repos_url":"https://api.github.com/users/febael/repos","events_url":"https://api.github.com/users/febael/events{/privacy}","received_events_url":"https://api.github.com/users/febael/received_events","type":"User","site_admin":false},"created_at":"2018-08-03T12:31:12Z","updated_at":"2018-08-03T12:31:12Z","author_association":"NONE","body":"Thank you @fviale for your analysis. I used the jvisualvm tool (should be shipped with every jdk installation), and I could get all thread dumps and objects in the heap with that. \r\n\r\n* How did you manage to decode the network message ? Especially if this message huge size triggered the out-of-memory error, how did you manage to analyse it retroactively ?  \r\n  * The thread causing the OOM shows allocation of a byteArray in a constructor. In the biggest objects list, a byteArray is the far winner with 1.1 GB. You an easily see the contents of this byteArray with jvisualvm.\r\n* While this OOM appeared, you were strictly doing nothing at the scheduler level ? There was no task running ? No file transfer in progress ?\r\n  * I can say with 99.9% certainty that nothing was being done at the time of error.\r\n* Do you see any unusual error in the scheduler logs (scheduler_home/logs/Scheduler.log) ?\r\n  * Nothing unusual in Scheduler.log or Node-local-LocalNodes-0.log\r\n* Could you send us the full thread dump of the JVM at the moment of the OOM ?\r\n  * I think jvisualvm can help for this purpose\r\n\r\nDo you have any idea on how to monitor the case for more clues? "}}
{"actor":{"display_login":"febael"}}