{"action":"created","issue":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-beats/issues/286","repository_url":"https://api.github.com/repos/logstash-plugins/logstash-input-beats","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-input-beats/issues/286/labels{/name}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-input-beats/issues/286/comments","events_url":"https://api.github.com/repos/logstash-plugins/logstash-input-beats/issues/286/events","html_url":"https://github.com/logstash-plugins/logstash-input-beats/issues/286","id":291248131,"number":286,"title":"Very High Memory use with Beats 6.1.x","user":{"login":"original-brownbear","id":6490959,"avatar_url":"https://avatars0.githubusercontent.com/u/6490959?v=4","gravatar_id":"","url":"https://api.github.com/users/original-brownbear","html_url":"https://github.com/original-brownbear","followers_url":"https://api.github.com/users/original-brownbear/followers","following_url":"https://api.github.com/users/original-brownbear/following{/other_user}","gists_url":"https://api.github.com/users/original-brownbear/gists{/gist_id}","starred_url":"https://api.github.com/users/original-brownbear/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/original-brownbear/subscriptions","organizations_url":"https://api.github.com/users/original-brownbear/orgs","repos_url":"https://api.github.com/users/original-brownbear/repos","events_url":"https://api.github.com/users/original-brownbear/events{/privacy}","received_events_url":"https://api.github.com/users/original-brownbear/received_events","type":"User","site_admin":false},"labels":[{"id":362685596,"url":"https://api.github.com/repos/logstash-plugins/logstash-input-beats/labels/crashes","name":"crashes","color":"e11d21","default":false},{"id":362685643,"url":"https://api.github.com/repos/logstash-plugins/logstash-input-beats/labels/resiliency","name":"resiliency","color":"bfd4f2","default":false}],"state":"open","locked":false,"assignee":{"login":"robbavey","id":4061337,"avatar_url":"https://avatars2.githubusercontent.com/u/4061337?v=4","gravatar_id":"","url":"https://api.github.com/users/robbavey","html_url":"https://github.com/robbavey","followers_url":"https://api.github.com/users/robbavey/followers","following_url":"https://api.github.com/users/robbavey/following{/other_user}","gists_url":"https://api.github.com/users/robbavey/gists{/gist_id}","starred_url":"https://api.github.com/users/robbavey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robbavey/subscriptions","organizations_url":"https://api.github.com/users/robbavey/orgs","repos_url":"https://api.github.com/users/robbavey/repos","events_url":"https://api.github.com/users/robbavey/events{/privacy}","received_events_url":"https://api.github.com/users/robbavey/received_events","type":"User","site_admin":false},"assignees":[{"login":"robbavey","id":4061337,"avatar_url":"https://avatars2.githubusercontent.com/u/4061337?v=4","gravatar_id":"","url":"https://api.github.com/users/robbavey","html_url":"https://github.com/robbavey","followers_url":"https://api.github.com/users/robbavey/followers","following_url":"https://api.github.com/users/robbavey/following{/other_user}","gists_url":"https://api.github.com/users/robbavey/gists{/gist_id}","starred_url":"https://api.github.com/users/robbavey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robbavey/subscriptions","organizations_url":"https://api.github.com/users/robbavey/orgs","repos_url":"https://api.github.com/users/robbavey/repos","events_url":"https://api.github.com/users/robbavey/events{/privacy}","received_events_url":"https://api.github.com/users/robbavey/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2018-01-24T15:23:30Z","updated_at":"2018-03-13T23:04:18Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Running Beats 6.1.x with default settings, namely `pipelining: 5` causes the beats input to consume a very large amount of memory.\r\n\r\nThis situation needs to be improved since the combination of `pipelining: x` with `x > 0` is prohibitively expensive when one Logstash instance is written to by multiple Beats clients and backpressure sets in.\r\n\r\nBelow graph shows a configuration of just Filebeat -> (beats-input -> stdout)\r\n\r\n<img width=\"768\" alt=\"screen shot 2018-01-24 at 08 16 08\" src=\"https://user-images.githubusercontent.com/6490959/35339523-1cf9efb6-0121-11e8-8c01-55f24d411940.png\">\r\n\r\nThe graph starts off showing just a single Filebeat writing to the LS instance.\r\nAt the `5:10` mark a second Filebeat is started way increasing the memory used by the input plugin.\r\n\r\nThe bulk of the memory use is made up by `org.logstash.beats.Message` instances referenced as shown below:\r\n\r\n<img width=\"2277\" alt=\"screen shot 2018-01-24 at 07 25 52\" src=\"https://user-images.githubusercontent.com/6490959/35339629-5ef611ec-0121-11e8-9512-71b27789ca7b.png\">\r\n\r\nPossible issues I identified here are:\r\n\r\n* We're running `Runtime.getRuntime().availableProcessors() * 4` worker threads in `Server`, that's way too much since we appear to be handling decoding (and its necessary buffering) in each of these.\r\n* `org.logstash.beats.Message` get deserialized before use and hold redundant data as a result of `org.logstash.beats.Message#extractIdentityStream`\r\n   * I would likely help **a lot** making the decoding here lazy and having each `org.logstash.beats.Message` simply reference a `ByteBuf` from an off-heap pool (bonus point for reusing that `ByteBuf` or even the whole `Message` instance)\r\n* We could also optimize decoding of the actual message, constantly allocating `byte[]` for every read in `org.logstash.beats.BeatsParser#decode` puts quite some additional pressure on the GC\r\n\r\n```java\r\n                byte[] bytes = new byte[requiredBytes];\r\n                in.readBytes(bytes);\r\n                Message message = new Message(sequence, (Map) MAPPER.readValue(bytes, Object.class));\r\n```\r\n\r\n... I'll try to add a few more specific things I found to this issue later. The most important thing as far as I can tell from the profiling is to make `Message` cheaper by ideally just having it be a pointer to a chunk of memory (`ByteBuf` seems easiest, but slices of a `MappedByteBuffer` might be even better since it would allow paging out if necessary, I'd start with `ByteBuf` :)) that gets deserialized when used."},"comment":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-beats/issues/comments/372848403","html_url":"https://github.com/logstash-plugins/logstash-input-beats/issues/286#issuecomment-372848403","issue_url":"https://api.github.com/repos/logstash-plugins/logstash-input-beats/issues/286","id":372848403,"user":{"login":"jakelandis","id":976291,"avatar_url":"https://avatars2.githubusercontent.com/u/976291?v=4","gravatar_id":"","url":"https://api.github.com/users/jakelandis","html_url":"https://github.com/jakelandis","followers_url":"https://api.github.com/users/jakelandis/followers","following_url":"https://api.github.com/users/jakelandis/following{/other_user}","gists_url":"https://api.github.com/users/jakelandis/gists{/gist_id}","starred_url":"https://api.github.com/users/jakelandis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakelandis/subscriptions","organizations_url":"https://api.github.com/users/jakelandis/orgs","repos_url":"https://api.github.com/users/jakelandis/repos","events_url":"https://api.github.com/users/jakelandis/events{/privacy}","received_events_url":"https://api.github.com/users/jakelandis/received_events","type":"User","site_admin":false},"created_at":"2018-03-13T23:04:18Z","updated_at":"2018-03-13T23:04:18Z","author_association":"CONTRIBUTOR","body":"@kovyrin -   One of the root cause [issues](https://github.com/logstash-plugins/logstash-input-beats/issues/310) has been fixed and released. This may help some, but is not a complete solution. \r\n\r\n```\r\nbin/logstash-plugin update logstash-input-beats\r\n```\r\n6.x - `5.0.11`\r\n5.x - `3.1.30`\r\n\r\nWe have identified where the excessive memory is coming from and working on a strategy to place an upper limit on how much data we can accept.  In the mean time, if you are on the latest version of the plugins you maybe able to increase the max off heap memory used by setting the `-XX:MaxDirectMemorySize=<size>` to a size larger then `-Xmx` value in the jvm.options. This of course assumes the host has enough memory as to not start swapping. (LS doesn't mlock like ES)\r\n\r\n"}}
{"actor":{"display_login":"jakelandis"}}