{"action":"created","issue":{"url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/2932","repository_url":"https://api.github.com/repos/Graylog2/graylog2-server","labels_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/2932/labels{/name}","comments_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/2932/comments","events_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/2932/events","html_url":"https://github.com/Graylog2/graylog2-server/issues/2932","id":181618074,"number":2932,"title":"Graylog server memory leak issue","user":{"login":"4BRobby","id":22675949,"avatar_url":"https://avatars.githubusercontent.com/u/22675949?v=3","gravatar_id":"","url":"https://api.github.com/users/4BRobby","html_url":"https://github.com/4BRobby","followers_url":"https://api.github.com/users/4BRobby/followers","following_url":"https://api.github.com/users/4BRobby/following{/other_user}","gists_url":"https://api.github.com/users/4BRobby/gists{/gist_id}","starred_url":"https://api.github.com/users/4BRobby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/4BRobby/subscriptions","organizations_url":"https://api.github.com/users/4BRobby/orgs","repos_url":"https://api.github.com/users/4BRobby/repos","events_url":"https://api.github.com/users/4BRobby/events{/privacy}","received_events_url":"https://api.github.com/users/4BRobby/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Graylog2/graylog2-server/labels/needs-input","name":"needs-input","color":"fbca04"}],"state":"open","locked":false,"assignee":{"login":"kroepke","id":52014,"avatar_url":"https://avatars.githubusercontent.com/u/52014?v=3","gravatar_id":"","url":"https://api.github.com/users/kroepke","html_url":"https://github.com/kroepke","followers_url":"https://api.github.com/users/kroepke/followers","following_url":"https://api.github.com/users/kroepke/following{/other_user}","gists_url":"https://api.github.com/users/kroepke/gists{/gist_id}","starred_url":"https://api.github.com/users/kroepke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kroepke/subscriptions","organizations_url":"https://api.github.com/users/kroepke/orgs","repos_url":"https://api.github.com/users/kroepke/repos","events_url":"https://api.github.com/users/kroepke/events{/privacy}","received_events_url":"https://api.github.com/users/kroepke/received_events","type":"User","site_admin":false},"assignees":[{"login":"kroepke","id":52014,"avatar_url":"https://avatars.githubusercontent.com/u/52014?v=3","gravatar_id":"","url":"https://api.github.com/users/kroepke","html_url":"https://github.com/kroepke","followers_url":"https://api.github.com/users/kroepke/followers","following_url":"https://api.github.com/users/kroepke/following{/other_user}","gists_url":"https://api.github.com/users/kroepke/gists{/gist_id}","starred_url":"https://api.github.com/users/kroepke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kroepke/subscriptions","organizations_url":"https://api.github.com/users/kroepke/orgs","repos_url":"https://api.github.com/users/kroepke/repos","events_url":"https://api.github.com/users/kroepke/events{/privacy}","received_events_url":"https://api.github.com/users/kroepke/received_events","type":"User","site_admin":false}],"milestone":null,"comments":9,"created_at":"2016-10-07T09:04:28Z","updated_at":"2016-10-17T16:08:42Z","closed_at":null,"body":"<!--- Provide a general summary of the issue in the Title above -->\r\n\r\n## Expected Behavior\r\n<!--- If you're describing a bug, tell us what should happen -->\r\n<!--- If you're suggesting a change/improvement, tell us how it should work -->\r\nGraylog should clean up memory space it is using.\r\n\r\n## Current Behavior\r\n<!--- If describing a bug, tell us what happens instead of the expected behavior -->\r\n<!--- If suggesting a change/improvement, explain the difference from current behavior -->\r\nWhen starting up the graylog server java process is using ~ 5%-15% of cpu time while processing web frontend requests and the few messages we have coming in.\r\nHeap Memory usage in the frontend says \"The JVM is using  480.9MB of  972.8MB heap space and will not attempt to use more than  1.4GB\"\r\n\r\nAfter several night jobs firing some more messages the graylog process seems to have trouble doing a proper garbage collection (see attached screen shot). \r\nHeap Memory usage in the frontend says \"The JVM is using  1.4GB of  1.4GB heap space and will not attempt to use more than  1.4GB\".\r\nAnd even the garbage collection will not free more, so it is stuck with 1.4 GB. The screenshot was made shortly before it will crash with several exceptions including an out of memory exception while the process is running CPU ~ 105%.\r\n\r\nThe image was shoot after ~2 days running time.\r\n\r\nThe thing is that graylog should literally die of boredom because NO messages are coming in and only a few since the night. Buffer and journal utilization shows that the qeue is not used at all (see attached screenshot).\r\n\r\nTo me this looks like a memory leak as used heap space is growing and growing and all garbage collection events free less memory than the one before. So in the end it crashes.\r\n\r\nI had the same set up running at a different company with A LOT more messages coming in but it was upgraded to early Graylog 2.0 from Graylog2 and there was never the 2.1 running there. No problems at all.\r\n\r\nWith this machine there was no tweaking done except raising the ulimits core nofile nproc which did not improve anything. So basically out of the box configuration of the OVA image (2.1.0).\r\n\r\nError:\r\n```\r\n2016-10-09_08:05:41.75311 2016-10-09 10:05:40,492 WARN : org.graylog2.shared.buffers.InputBufferImpl - Unable to process event <invalid>, sequence 66707\r\n2016-10-09_08:05:41.75314 java.lang.OutOfMemoryError: Java heap space\r\n2016-10-09_08:05:45.19320 2016-10-09 10:05:45,192 ERROR: org.graylog2.shared.buffers.JournallingMessageHandler - Unable to convert RawMessageEvent to Journal.Entry - skipping event\r\n2016-10-09_08:05:45.19323 java.lang.NullPointerException\r\n2016-10-09_08:05:45.19323       at org.graylog2.shared.buffers.JournallingMessageHandler$Converter.apply(JournallingMessageHandler.java:138) [graylog.jar:?]\r\n2016-10-09_08:05:45.19324       at org.graylog2.shared.buffers.JournallingMessageHandler$Converter.apply(JournallingMessageHandler.java:120) [graylog.jar:?]\r\n2016-10-09_08:05:45.19324       at com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:640) [graylog.jar:?]\r\n2016-10-09_08:05:45.19324       at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) [graylog.jar:?]\r\n2016-10-09_08:05:45.19324       at java.util.AbstractCollection.toArray(AbstractCollection.java:141) [?:1.8.0_101]\r\n2016-10-09_08:05:45.19325       at java.util.ArrayList.<init>(ArrayList.java:177) [?:1.8.0_101]\r\n2016-10-09_08:05:45.19325       at com.google.common.collect.Lists.newArrayList(Lists.java:146) [graylog.jar:?]\r\n2016-10-09_08:05:45.19325       at org.graylog2.shared.buffers.JournallingMessageHandler.onEvent(JournallingMessageHandler.java:83) [graylog.jar:?]\r\n2016-10-09_08:05:45.19325       at org.graylog2.shared.buffers.JournallingMessageHandler.onEvent(JournallingMessageHandler.java:45) [graylog.jar:?]\r\n2016-10-09_08:05:45.19326       at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129) [graylog.jar:?]\r\n2016-10-09_08:05:45.19327       at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]\r\n2016-10-09_08:05:45.19327       at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]\r\n\r\n```\r\n\r\n\r\n## Possible Solution\r\n<!--- Not obligatory, but suggest a fix/reason for the bug, -->\r\n<!--- or ideas how to implement the addition or change -->\r\n\r\n## Steps to Reproduce (for bugs)\r\n<!--- Provide a link to a live example, or an unambiguous set of steps to -->\r\n<!--- reproduce this bug. Include code to reproduce, if relevant -->\r\nAfter ~ 2 days and ~300.000 messages the heap memory is fully used. Graylog becomes unresponsive.\r\n\r\n## Context\r\n<!--- How has this issue affected you? What are you trying to accomplish? -->\r\n<!--- Providing context helps us come up with a solution that is most useful in the real world -->\r\n\r\n## Your Environment\r\n<!--- Include as many relevant details about the environment you experienced the bug in -->\r\nThe OVA from graylog deployed via Xenserver, the node has 6GB RAM and 2 processors assigned\r\n* Graylog Version: 2.1.1+01d50e5, codename Smuttynose JVM Oracle Corporation 1.8.0_101 on Linux 4.2.0-42-generic\r\n* Elasticsearch Version: 2.3.1\r\n* MongoDB Version: db version v3.2.5\r\n* Operating System: Ubuntu\r\n* Browser version:\r\n\r\n* Set up 1 GELF UDP input globally, appliance-gelf-udp and appliance-syslog-udp are running locally.\r\n* JAVA Options\r\n```\r\n1460 ?        Ssl   16:56 /opt/graylog/embedded/jre/bin/java -Xms3g -Xmx3g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -server -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=3333 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/graylog/elasticsearch -cp /opt/graylog/elasticsearch/lib/elasticsearch-2.3.1.jar:/opt/graylog/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start\r\n 1511 ?        Ssl  479:09 /opt/graylog/embedded/jre/bin/java -Xms1g -Xmx1500m -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -jar -Dlog4j.configurationFile=file:///opt/graylog/conf/log4j2.xml -Djava.library.path=/opt/graylog/server/lib/sigar/ -Dgraylog2.installation_source=ova /opt/graylog/server/graylog.jar server -f /opt/graylog/conf/graylog.conf\r\n```\r\n[graylogdump.txt](https://github.com/Graylog2/graylog2-server/files/515661/graylogdump.txt)\r\n![graylogstatus](https://cloud.githubusercontent.com/assets/22675949/19184105/d5ad7ba2-8c7b-11e6-8bea-c6ffee231dd2.png)"},"comment":{"url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/comments/254253270","html_url":"https://github.com/Graylog2/graylog2-server/issues/2932#issuecomment-254253270","issue_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/2932","id":254253270,"user":{"login":"kroepke","id":52014,"avatar_url":"https://avatars.githubusercontent.com/u/52014?v=3","gravatar_id":"","url":"https://api.github.com/users/kroepke","html_url":"https://github.com/kroepke","followers_url":"https://api.github.com/users/kroepke/followers","following_url":"https://api.github.com/users/kroepke/following{/other_user}","gists_url":"https://api.github.com/users/kroepke/gists{/gist_id}","starred_url":"https://api.github.com/users/kroepke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kroepke/subscriptions","organizations_url":"https://api.github.com/users/kroepke/orgs","repos_url":"https://api.github.com/users/kroepke/repos","events_url":"https://api.github.com/users/kroepke/events{/privacy}","received_events_url":"https://api.github.com/users/kroepke/received_events","type":"User","site_admin":false},"created_at":"2016-10-17T16:08:41Z","updated_at":"2016-10-17T16:08:41Z","body":"Great, OOM is always a source of concern, so I'd be more than happy to find out that it is just a tuning thing :)"}}
{"actor":{"display_login":"kroepke"}}