{"action":"created","issue":{"url":"https://api.github.com/repos/raintank/raintank-metric/issues/153","repository_url":"https://api.github.com/repos/raintank/raintank-metric","labels_url":"https://api.github.com/repos/raintank/raintank-metric/issues/153/labels{/name}","comments_url":"https://api.github.com/repos/raintank/raintank-metric/issues/153/comments","events_url":"https://api.github.com/repos/raintank/raintank-metric/issues/153/events","html_url":"https://github.com/raintank/raintank-metric/pull/153","id":135075458,"number":153,"title":"only persist chunks asynchronously","user":{"login":"Dieterbe","id":20774,"avatar_url":"https://avatars.githubusercontent.com/u/20774?v=3","gravatar_id":"","url":"https://api.github.com/users/Dieterbe","html_url":"https://github.com/Dieterbe","followers_url":"https://api.github.com/users/Dieterbe/followers","following_url":"https://api.github.com/users/Dieterbe/following{/other_user}","gists_url":"https://api.github.com/users/Dieterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/Dieterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dieterbe/subscriptions","organizations_url":"https://api.github.com/users/Dieterbe/orgs","repos_url":"https://api.github.com/users/Dieterbe/repos","events_url":"https://api.github.com/users/Dieterbe/events{/privacy}","received_events_url":"https://api.github.com/users/Dieterbe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-02-20T13:46:26Z","updated_at":"2016-02-21T06:58:47Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/raintank/raintank-metric/pulls/153","html_url":"https://github.com/raintank/raintank-metric/pull/153","diff_url":"https://github.com/raintank/raintank-metric/pull/153.diff","patch_url":"https://github.com/raintank/raintank-metric/pull/153.patch"},"body":"i suspect that calling persist() when we step into a new chunkspan\r\nis an operation that:\r\n1) is called for a whole lot of metrics at nearly the same time\r\n2) creates a lot of lock contention by simultaneously trying to add\r\n   to the store/write queues\r\n\r\nI don't have hard facts to prove this yet.\r\nBut this approach removes this assumed burden and instead relies\r\nsolely on the GC tasks to persist chunks.\r\nAt least the GC does this metric by metric.\r\nYou may want to increase your GC frequency to ~ chunkSpan\r\n\r\nwe loose the NSQ backpressure on full-queue, perhaps it could be\r\nre-added.  With this approach it becomes more a matter of verifying\r\nGC can keep up, which will need a bit more instrumentation.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/raintank/raintank-metric/issues/comments/186762140","html_url":"https://github.com/raintank/raintank-metric/pull/153#issuecomment-186762140","issue_url":"https://api.github.com/repos/raintank/raintank-metric/issues/153","id":186762140,"user":{"login":"woodsaj","id":8415072,"avatar_url":"https://avatars.githubusercontent.com/u/8415072?v=3","gravatar_id":"","url":"https://api.github.com/users/woodsaj","html_url":"https://github.com/woodsaj","followers_url":"https://api.github.com/users/woodsaj/followers","following_url":"https://api.github.com/users/woodsaj/following{/other_user}","gists_url":"https://api.github.com/users/woodsaj/gists{/gist_id}","starred_url":"https://api.github.com/users/woodsaj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/woodsaj/subscriptions","organizations_url":"https://api.github.com/users/woodsaj/orgs","repos_url":"https://api.github.com/users/woodsaj/repos","events_url":"https://api.github.com/users/woodsaj/events{/privacy}","received_events_url":"https://api.github.com/users/woodsaj/received_events","type":"User","site_admin":false},"created_at":"2016-02-21T06:58:47Z","updated_at":"2016-02-21T06:58:47Z","body":"I dont think the current problem is just the call to persist().  non-primary metrc-tank nodes, which dont see a slowdown, also call persist(), they just exit early after calling chunk.finish().\r\n\r\nSo the problem is either with how we scan the older chunks to check if they also need persisting.\r\nhttps://github.com/raintank/raintank-metric/blob/master/metric_tank/aggmetric.go#L357-L378\r\n\r\nor the call to add the chunk to the writeQueue.\r\nhttps://github.com/raintank/raintank-metric/blob/master/metric_tank/aggmetric.go#L393\r\n\r\nI would like to get some metrics to confirm/deny if either of these are the slowdown.\r\n\r\n"}}