{"push_id":986414304,"size":4,"distinct_size":4,"ref":"refs/heads/async-persist","head":"0ce2703bdde0d93c73f304a906327898b618316e","before":"23427bffac00f519a58195cdc2ab97a3622c8d3f","commits":[{"sha":"dbd22533a93e2be76a2a954424d7dff27c895255","author":{"email":"b617834f89be0c2a85d223f602f0a5edf9446141@raintank.io","name":"Dieter Plaetinck"},"message":"performance fix: don't re-save chunk if it has already been saved","distinct":true,"url":"https://api.github.com/repos/raintank/raintank-metric/commits/dbd22533a93e2be76a2a954424d7dff27c895255"},{"sha":"dc5c22a50c235ee1c442d73d20eab2b1c5966d4a","author":{"email":"b617834f89be0c2a85d223f602f0a5edf9446141@raintank.io","name":"Dieter Plaetinck"},"message":"instrument aggMetrics being GC'd\n\nit will be nice to see how many per second we can do,\nas well as how long the total run takes","distinct":true,"url":"https://api.github.com/repos/raintank/raintank-metric/commits/dc5c22a50c235ee1c442d73d20eab2b1c5966d4a"},{"sha":"ee9667f64e7c3b17665daf6ab76ce92910551bfb","author":{"email":"b617834f89be0c2a85d223f602f0a5edf9446141@raintank.io","name":"Dieter Plaetinck"},"message":"only persist chunks asynchronously\n\ni suspect that calling persist() when we step into a new chunkspan\nis an operation that:\n1) is called for a whole lot of metrics at nearly the same time\n2) creates a lot of lock contention by simultaneously trying to add\n   to the store/write queues\n\nI don't have hard facts to prove this yet.\nBut this approach removes this assumed burden and instead relies\nsolely on the GC tasks to persist chunks.\nAt least the GC does this metric by metric.\nYou may want to increase your GC frequency to ~ chunkSpan\n\nwe loose the NSQ backpressure on full-queue, perhaps it could be\nre-added.  With this approach it becomes more a matter of verifying\nGC can keep up, which will need a bit more instrumentation.\n\nto do this we need some refactoring for GC & persister:\n\n* it's now GC's job to not only persist the latest chunk when it\n  hasn't been written to in a while, it also has to make sure that\n  any previous chunks are being persisted.\n* since we now run persist solely as part of a maintenance task, and no longer\n  at the exact point where we know we need to persist a metric,\n  it is now being invoked with different semantics: the specified pos\n  refers to a chunk we may want to save, subject to the same constraints\n  as the others. it is not guaranteed unlike before.\n* bonus: don't do needless work when we're not primary, in particular\n  we only need to finish a chunk when it's being persisted","distinct":true,"url":"https://api.github.com/repos/raintank/raintank-metric/commits/ee9667f64e7c3b17665daf6ab76ce92910551bfb"},{"sha":"0ce2703bdde0d93c73f304a906327898b618316e","author":{"email":"b617834f89be0c2a85d223f602f0a5edf9446141@raintank.io","name":"Dieter Plaetinck"},"message":"should now run GC more often","distinct":true,"url":"https://api.github.com/repos/raintank/raintank-metric/commits/0ce2703bdde0d93c73f304a906327898b618316e"}]}