{"action":"created","issue":{"url":"https://api.github.com/repos/influxdata/kapacitor/issues/360","repository_url":"https://api.github.com/repos/influxdata/kapacitor","labels_url":"https://api.github.com/repos/influxdata/kapacitor/issues/360/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/kapacitor/issues/360/comments","events_url":"https://api.github.com/repos/influxdata/kapacitor/issues/360/events","html_url":"https://github.com/influxdata/kapacitor/pull/360","id":142083759,"number":360,"title":"Fork by measurement","user":{"login":"yosiat","id":306253,"avatar_url":"https://avatars.githubusercontent.com/u/306253?v=3","gravatar_id":"","url":"https://api.github.com/users/yosiat","html_url":"https://github.com/yosiat","followers_url":"https://api.github.com/users/yosiat/followers","following_url":"https://api.github.com/users/yosiat/following{/other_user}","gists_url":"https://api.github.com/users/yosiat/gists{/gist_id}","starred_url":"https://api.github.com/users/yosiat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yosiat/subscriptions","organizations_url":"https://api.github.com/users/yosiat/orgs","repos_url":"https://api.github.com/users/yosiat/repos","events_url":"https://api.github.com/users/yosiat/events{/privacy}","received_events_url":"https://api.github.com/users/yosiat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2016-03-19T17:27:37Z","updated_at":"2016-03-22T21:20:20Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/influxdata/kapacitor/pulls/360","html_url":"https://github.com/influxdata/kapacitor/pull/360","diff_url":"https://github.com/influxdata/kapacitor/pull/360.diff","patch_url":"https://github.com/influxdata/kapacitor/pull/360.patch"},"body":"Hi,\r\n\r\nThis pull request greatly improves performance on the write benchmarks.\r\nI did this performance improvements, in 5 steps:\r\n\r\n_All of benchmark ran on my Macbook Pro (13-inch, Late 2011) with Intel Core i5 (2.4Ghz), 8gb memory and 120gb SSD_\r\n\r\n### Filtering by measurement\r\nTo the fork struct I added measurements map from string to bool and compared it in the forkPoint.\r\nAnd got the next improvement:\r\n```\r\nbenchmark                                            old ns/op      new ns/op      delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     8633314476     57042          -100.00%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        7915678886     8229547562     +3.97%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      37434          22472          -39.97%\r\nBenchmark_Write_MeasurementNameMatches_100-4         38474          41502          +7.87%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       22950          23601          +2.84%\r\nBenchmark_Write_MeasurementNameMatches_10-4          23109          24814          +7.38%\r\n\r\nbenchmark                                            old allocs     new allocs     delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     57450          50             -99.91%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        57426          57424          -0.00%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      49             49             +0.00%\r\nBenchmark_Write_MeasurementNameMatches_100-4         49             49             +0.00%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       49             49             +0.00%\r\nBenchmark_Write_MeasurementNameMatches_10-4          49             49             +0.00%\r\n\r\nbenchmark                                            old bytes     new bytes     delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     4264608       3950          -99.91%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        4261568       4261440       -0.00%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      3889          3837          -1.34%\r\nBenchmark_Write_MeasurementNameMatches_100-4         3889          3889          +0.00%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       3838          3838          +0.00%\r\nBenchmark_Write_MeasurementNameMatches_10-4          3838          3839          +0.03%\r\n```\r\n_This performance numbers are compared to the baseline - benchmarks run on the master_\r\n\r\n### Changing equality order\r\nI tried to change check of:\r\n```go\r\nif fork.dbrps[dbrp] && fork.measurements[p.Name] {\r\n   // ...\r\n}\r\n```\r\nTo first check the measurement and then dbrp, and got the next results:\r\n```\r\nbenchmark                                            old ns/op      new ns/op      delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     57042          29203          -48.80%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        8229547562     8787711023     +6.78%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      22472          36940          +64.38%\r\nBenchmark_Write_MeasurementNameMatches_100-4         41502          55299          +33.24%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       23601          36820          +56.01%\r\nBenchmark_Write_MeasurementNameMatches_10-4          24814          44957          +81.18%\r\n\r\nbenchmark                                            old allocs     new allocs     delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     50             49             -2.00%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        57424          57438          +0.02%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      49             49             +0.00%\r\nBenchmark_Write_MeasurementNameMatches_100-4         49             50             +2.04%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       49             49             +0.00%\r\nBenchmark_Write_MeasurementNameMatches_10-4          49             49             +0.00%\r\n\r\nbenchmark                                            old bytes     new bytes     delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     3950          3837          -2.86%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        4261440       4262336       +0.02%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      3837          3888          +1.33%\r\nBenchmark_Write_MeasurementNameMatches_100-4         3889          3953          +1.65%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       3838          3838          +0.00%\r\nBenchmark_Write_MeasurementNameMatches_10-4          3839          3838          -0.03%\r\n```\r\n_This is compared between the first step and the second_\r\nAs you can see the performance got better for Benchmark_Write_MeasurementNameNotMatches_1000-4 but worse for the benchmarks (+33 to +64)\r\n\r\n### Change the fork structure - map from dbrp&measurement to edges\r\nI am skipping the forth step which is to take \"dbrp\" struct assignment in forkPoint out of the loop, and going to the biggest perf improvement.\r\n\r\nInstead of checking all forks if they match criteria I pivoted it to map from the criteria (db,rp,measurement) to edges.\r\n\r\n\r\n\r\nAnd we get this huge improvement:\r\n```\r\nbenchmark                                            old ns/op      new ns/op      delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     29203          20774          -28.86%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        8787711023     5675405636     -35.42%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      36940          21771          -41.06%\r\nBenchmark_Write_MeasurementNameMatches_100-4         55299          36193          -34.55%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       36820          23315          -36.68%\r\nBenchmark_Write_MeasurementNameMatches_10-4          44957          24562          -45.37%\r\n\r\nbenchmark                                            old allocs     new allocs     delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     49             48             -2.04%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        57438          57436          -0.00%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      49             48             -2.04%\r\nBenchmark_Write_MeasurementNameMatches_100-4         50             49             -2.00%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       49             49             +0.00%\r\nBenchmark_Write_MeasurementNameMatches_10-4          49             49             +0.00%\r\n\r\nbenchmark                                            old bytes     new bytes     delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     3837          3800          -0.96%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        4262336       4262208       -0.00%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      3888          3800          -2.26%\r\nBenchmark_Write_MeasurementNameMatches_100-4         3953          3888          -1.64%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       3838          3838          +0.00%\r\nBenchmark_Write_MeasurementNameMatches_10-4          3838          3839          +0.03%\r\n```\r\n_The baseline is 'Changing equality order'_\r\n\r\nAnother sign of performance improvement, while running \"Benchmark_Write_MeasurementNameNotMatches_1000-4\" on the master my 4 cores are 99% steady after this improvement only 2 cores are 59% ~ and the other 2 cores are  9%\r\n\r\n## Final Results\r\nAnd the overall benchmark results, where the baseline is the master benchmark results and the new perf is the current status of this branch:\r\n```\r\nbenchmark                                            old ns/op      new ns/op      delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     8633314476     23139          -100.00%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        7915678886     6381307112     -19.38%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      37434          23787          -36.46%\r\nBenchmark_Write_MeasurementNameMatches_100-4         38474          34923          -9.23%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       22950          24076          +4.91%\r\nBenchmark_Write_MeasurementNameMatches_10-4          23109          25433          +10.06%\r\n\r\nbenchmark                                            old allocs     new allocs     delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     57450          48             -99.92%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        57426          57442          +0.03%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      49             48             -2.04%\r\nBenchmark_Write_MeasurementNameMatches_100-4         49             49             +0.00%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       49             49             +0.00%\r\nBenchmark_Write_MeasurementNameMatches_10-4          49             49             +0.00%\r\n\r\nbenchmark                                            old bytes     new bytes     delta\r\nBenchmark_Write_MeasurementNameNotMatches_1000-4     4264608       3799          -99.91%\r\nBenchmark_Write_MeasurementNameMatches_1000-4        4261568       4262592       +0.02%\r\nBenchmark_Write_MeasurementNameNotMatches_100-4      3889          3800          -2.29%\r\nBenchmark_Write_MeasurementNameMatches_100-4         3889          3889          +0.00%\r\nBenchmark_Write_MeasurementNameNotMatches_10-4       3838          3838          +0.00%\r\nBenchmark_Write_MeasurementNameMatches_10-4          3838          3838          +0.00%\r\n```\r\n\r\n## Drawbacks\r\nThis benchmark come with one drawback, the creation and deletion of a task will be slower (I have no benchmarks, but we are doing more - we no longer have o(1)  complexity) and the deletion is harder to read thanks to \"Change the fork structure - map from dbrp&measurement to edges\".\r\n\r\nI am open to suggestions on how to improve the delFork method for better readability.\r\n"},"comment":{"url":"https://api.github.com/repos/influxdata/kapacitor/issues/comments/200035117","html_url":"https://github.com/influxdata/kapacitor/pull/360#issuecomment-200035117","issue_url":"https://api.github.com/repos/influxdata/kapacitor/issues/360","id":200035117,"user":{"login":"nathanielc","id":3771906,"avatar_url":"https://avatars.githubusercontent.com/u/3771906?v=3","gravatar_id":"","url":"https://api.github.com/users/nathanielc","html_url":"https://github.com/nathanielc","followers_url":"https://api.github.com/users/nathanielc/followers","following_url":"https://api.github.com/users/nathanielc/following{/other_user}","gists_url":"https://api.github.com/users/nathanielc/gists{/gist_id}","starred_url":"https://api.github.com/users/nathanielc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathanielc/subscriptions","organizations_url":"https://api.github.com/users/nathanielc/orgs","repos_url":"https://api.github.com/users/nathanielc/repos","events_url":"https://api.github.com/users/nathanielc/events{/privacy}","received_events_url":"https://api.github.com/users/nathanielc/received_events","type":"User","site_admin":false},"created_at":"2016-03-22T21:20:20Z","updated_at":"2016-03-22T21:20:20Z","body":"Thanks for digging in.\r\n> do you want if there is a way to compare two cpu profiles in go?\r\n\r\n Not directly but I was using flame graphs and they seemed to work alright https://github.com/uber/go-torch\r\n\r\n>  I think there is high allocations because we are sending 4,800~points which gets emitted 4800\\*1000 (points\\*number of tasks)\r\n\r\nIf that is the case shouldn't we see that the number of allocs for the 10 and 100 cases be similar to 4800\\*10 and 4800\\*100 respectively, because we don't see that.\r\n\r\n"}}