{"action":"created","issue":{"url":"https://api.github.com/repos/arangodb/arangodb/issues/5579","repository_url":"https://api.github.com/repos/arangodb/arangodb","labels_url":"https://api.github.com/repos/arangodb/arangodb/issues/5579/labels{/name}","comments_url":"https://api.github.com/repos/arangodb/arangodb/issues/5579/comments","events_url":"https://api.github.com/repos/arangodb/arangodb/issues/5579/events","html_url":"https://github.com/arangodb/arangodb/issues/5579","id":331171092,"node_id":"MDU6SXNzdWUzMzExNzEwOTI=","number":5579,"title":"Possible memory leak / regression in 3.3","user":{"login":"choppedpork","id":11997044,"node_id":"MDQ6VXNlcjExOTk3MDQ0","avatar_url":"https://avatars1.githubusercontent.com/u/11997044?v=4","gravatar_id":"","url":"https://api.github.com/users/choppedpork","html_url":"https://github.com/choppedpork","followers_url":"https://api.github.com/users/choppedpork/followers","following_url":"https://api.github.com/users/choppedpork/following{/other_user}","gists_url":"https://api.github.com/users/choppedpork/gists{/gist_id}","starred_url":"https://api.github.com/users/choppedpork/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/choppedpork/subscriptions","organizations_url":"https://api.github.com/users/choppedpork/orgs","repos_url":"https://api.github.com/users/choppedpork/repos","events_url":"https://api.github.com/users/choppedpork/events{/privacy}","received_events_url":"https://api.github.com/users/choppedpork/received_events","type":"User","site_admin":false},"labels":[{"id":711322364,"node_id":"MDU6TGFiZWw3MTEzMjIzNjQ=","url":"https://api.github.com/repos/arangodb/arangodb/labels/3%20OOM","name":"3 OOM","color":"f9d0c4","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2018-06-11T12:37:20Z","updated_at":"2018-09-12T14:01:16Z","closed_at":null,"author_association":"NONE","body":"## my environment running ArangoDB\r\nI'm using ArangoDB version:\r\n- [x] 3.3.8 (subsequently upgraded to 3.3.10)\r\n\r\nMode:\r\n- [x] Cluster\r\n\r\nStorage-Engine:\r\n- [x] rocksdb\r\n\r\nOn this operating system:\r\n- [x] Linux \r\n  - [x] other: self-built Docker container using the latest Ubuntu package\r\n\r\n### this is an installation-related issue:\r\nHello,\r\n\r\nSince upgrading to 3.3 we've noticed a rather drastic increase in memory usage of server nodes over time. I've had a look at some of the existing open tickets and this looks quite similar to #5414 - I've decided to open a separate issue and let you decide though.\r\n\r\nWe're seeing this problem only in server instances - both the agency and coordinator nodes have very stable memory usage. Our standard deployment model for Arango is 3x agency, 3x coordinator and 3x server. We're using the following config options (I've skipped ones that seem irrelevant to me like directory locations, ip addresses etc - let me know if you'd like the full list):\r\n```\r\n--server.authentication=false\r\n--cluster.my-role PRIMARY\r\n--log.level info\r\n--javascript.v8-contexts 16\r\n--javascript.v8-max-heap 3072\r\n--server.storage-engine rocksdb\r\n```\r\n\r\nHere's a list of sysctls we're setting:\r\n```\r\n- { name: \"vm.max_map_count\", value: \"262144\" }\r\n- { name: \"vm.overcommit_memory\", value: \"0\" } # arangodb recommend setting this to 2 but this causes a lot of issues bringing other containers up\r\n- { name: \"vm.zone_reclaim_mode\", value: \"0\" }\r\n- { name: \"vm.swappiness\", value: \"1\" }\r\n- { name: \"net.core.somaxconn\", value: \"65535\" }\r\n```\r\n\r\n`net.core.somaxconn` is also set (same value) on the Docker container. \r\n\r\nWe're setting the transparent_hugepage `defrag` and `enabled` properties to `never`.\r\n\r\nWe've upgraded from 3.2.9 to 3.3.0 and have since used 3.3.3, 3.3.4 and on 3.3.8 now.\r\nHere's a memory usage (RSS) for the server nodes in one of our environments (which got upgraded to 3.3.x around 25th April - note that this environment gets shutdown in the evening every day hence the large gaps in the graph):\r\n\r\n![screen shot 2018-06-11 at 13 06 54](https://user-images.githubusercontent.com/11997044/41230684-a708feb8-6d78-11e8-8c52-410beaf88094.png)\r\n\r\nThis is the above graph zoomed in to the last 5 working day period:\r\n\r\n![screen shot 2018-06-11 at 13 06 22](https://user-images.githubusercontent.com/11997044/41230685-a72d8b5c-6d78-11e8-8811-f868ac11a7c0.png)\r\n\r\nThis is another environment, the upgrade to 3.3.x happened on 3rd of April. The change in the memory usage pattern on the 27th April has been caused by applying a docker memory limit of 2g.\r\n\r\n![screen shot 2018-06-11 at 13 05 16](https://user-images.githubusercontent.com/11997044/41230686-a7531430-6d78-11e8-84ce-a1d9af97a344.png)\r\n\r\nThe above environments have extremely light usage of Arango.\r\n\r\nHere's one of which gets used a bit more:\r\n![screen shot 2018-06-11 at 13 22 43](https://user-images.githubusercontent.com/11997044/41231428-f6db39ae-6d7a-11e8-8673-a818cb106cae.png)\r\n\r\nAs a reference point of sorts, here's what it looks like when we run a load test against our application:\r\n![screen shot 2018-06-11 at 13 23 39](https://user-images.githubusercontent.com/11997044/41231360-c47c39ea-6d7a-11e8-89e7-cce0bcbd178d.png)\r\nThe server nodes eventually tail off at 6.4GB and memory usage remains perfectly stable afterwards.\r\n\r\nAll of the above graphs were taken using the same settings I've mentioned earlier.\r\n\r\nLet me know what other information would be useful to provide - I guess disabling statistics and or Foxx queues would be something you might want us to try? If so - shall we try disabling both off or try them one by one (if so - what order would you prefer)?\r\n\r\nThanks,\r\nSimon\r\n\r\nedit: as part of the investigation I have upgraded from 3.3.8 to 3.3.10\r\n"},"comment":{"url":"https://api.github.com/repos/arangodb/arangodb/issues/comments/420659337","html_url":"https://github.com/arangodb/arangodb/issues/5579#issuecomment-420659337","issue_url":"https://api.github.com/repos/arangodb/arangodb/issues/5579","id":420659337,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDY1OTMzNw==","user":{"login":"martinkunevtoptal","id":26064352,"node_id":"MDQ6VXNlcjI2MDY0MzUy","avatar_url":"https://avatars2.githubusercontent.com/u/26064352?v=4","gravatar_id":"","url":"https://api.github.com/users/martinkunevtoptal","html_url":"https://github.com/martinkunevtoptal","followers_url":"https://api.github.com/users/martinkunevtoptal/followers","following_url":"https://api.github.com/users/martinkunevtoptal/following{/other_user}","gists_url":"https://api.github.com/users/martinkunevtoptal/gists{/gist_id}","starred_url":"https://api.github.com/users/martinkunevtoptal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinkunevtoptal/subscriptions","organizations_url":"https://api.github.com/users/martinkunevtoptal/orgs","repos_url":"https://api.github.com/users/martinkunevtoptal/repos","events_url":"https://api.github.com/users/martinkunevtoptal/events{/privacy}","received_events_url":"https://api.github.com/users/martinkunevtoptal/received_events","type":"User","site_admin":false},"created_at":"2018-09-12T14:01:16Z","updated_at":"2018-09-12T14:01:16Z","author_association":"NONE","body":"I can confirm the memory leak is still present in v3.4.0-rc1. It seems as of this point the cause is still unknown."}}
{"actor":{"display_login":"martinkunevtoptal"}}