{"action":"created","issue":{"url":"https://api.github.com/repos/kairosdb/kairosdb/issues/458","repository_url":"https://api.github.com/repos/kairosdb/kairosdb","labels_url":"https://api.github.com/repos/kairosdb/kairosdb/issues/458/labels{/name}","comments_url":"https://api.github.com/repos/kairosdb/kairosdb/issues/458/comments","events_url":"https://api.github.com/repos/kairosdb/kairosdb/issues/458/events","html_url":"https://github.com/kairosdb/kairosdb/issues/458","id":315911268,"number":458,"title":"Memory leak in queueMemoryQueueProcessor","user":{"login":"rdzimmer","id":17729406,"avatar_url":"https://avatars3.githubusercontent.com/u/17729406?v=4","gravatar_id":"","url":"https://api.github.com/users/rdzimmer","html_url":"https://github.com/rdzimmer","followers_url":"https://api.github.com/users/rdzimmer/followers","following_url":"https://api.github.com/users/rdzimmer/following{/other_user}","gists_url":"https://api.github.com/users/rdzimmer/gists{/gist_id}","starred_url":"https://api.github.com/users/rdzimmer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rdzimmer/subscriptions","organizations_url":"https://api.github.com/users/rdzimmer/orgs","repos_url":"https://api.github.com/users/rdzimmer/repos","events_url":"https://api.github.com/users/rdzimmer/events{/privacy}","received_events_url":"https://api.github.com/users/rdzimmer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-04-19T14:47:55Z","updated_at":"2018-04-20T12:51:49Z","closed_at":null,"author_association":"NONE","body":"I'm running kairosdb-1.2.0-1.  I've been running well with it since it was released.  I recently switched to a different set of metrics coming from new data collectors.  Unfortunately, every ~2 hours my KairosDB is heap dumping.  Based on my load I don't expect it to be and think it could be a leak?  With the old metrics set I was doing ~1M metrics per minute.  With these new tests I was doing <200K/minute.  I don't see any errors in the logs (other than the heap dumps).  \r\n\r\nThe heap dump shows:\r\n![kairosdb_heap_dump](https://user-images.githubusercontent.com/17729406/38998397-799b2a08-43bd-11e8-868f-a06b833bd8f3.PNG)\r\n\r\nHere are the write_batch_size.sum for data_points (blue), row_keys (red), string_index (green).  I have KAIROSDB_ROW_KEY_CACHE_SIZE 1,500,000 and KAIROSDB_STRING_CACHE_SIZE 500,000 with a 2GB Xmx.  In my previous testing, ~2M cache items took up ~1.5GB.  And based on the heap dump and key/index write I don't believe that was the problem.\r\n![batch sizes](https://user-images.githubusercontent.com/17729406/38998420-87558922-43bd-11e8-8caa-088ffe00f421.PNG)\r\n\r\nHere is the jvm.free_memory.  Around 09:00 is when I stopped things and at ~10:00 switched to the old data collectors (which so far have not caused memory problems).  \r\n![image](https://user-images.githubusercontent.com/17729406/39000650-86f81a4e-43c2-11e8-8af7-74a048a57294.png)\r\n\r\nI've checked kairosdb.queue.memory_queue.size and it's normally tiny, at most 1500 occasionally (8000 temporarily after the crash and restart).  I'm using the org.kairosdb.core.queue.MemoryQueueProcessor, not the new file system.  Any KairosDB logging that might help?  Perhaps there's a new metric that's getting stuck in the queue?  "},"comment":{"url":"https://api.github.com/repos/kairosdb/kairosdb/issues/comments/383085931","html_url":"https://github.com/kairosdb/kairosdb/issues/458#issuecomment-383085931","issue_url":"https://api.github.com/repos/kairosdb/kairosdb/issues/458","id":383085931,"user":{"login":"rdzimmer","id":17729406,"avatar_url":"https://avatars3.githubusercontent.com/u/17729406?v=4","gravatar_id":"","url":"https://api.github.com/users/rdzimmer","html_url":"https://github.com/rdzimmer","followers_url":"https://api.github.com/users/rdzimmer/followers","following_url":"https://api.github.com/users/rdzimmer/following{/other_user}","gists_url":"https://api.github.com/users/rdzimmer/gists{/gist_id}","starred_url":"https://api.github.com/users/rdzimmer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rdzimmer/subscriptions","organizations_url":"https://api.github.com/users/rdzimmer/orgs","repos_url":"https://api.github.com/users/rdzimmer/repos","events_url":"https://api.github.com/users/rdzimmer/events{/privacy}","received_events_url":"https://api.github.com/users/rdzimmer/received_events","type":"User","site_admin":false},"created_at":"2018-04-20T12:51:49Z","updated_at":"2018-04-20T12:51:49Z","author_association":"NONE","body":"@brianhks From the heapdump, do you recognize that hashmap?  Any ideas from that where the memory storage is building up?  I know it's always hit or miss if those things show enough info for developers to know exactly.  It's okay if not, just looking for hints as I work with the data collectors dev team on what their new metrics/tags look like.  "}}
{"actor":{"display_login":"rdzimmer"}}