{"push_id":2566274181,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"3fabbc576203c7fd63808a259adafc5c3cea1838","before":"d610d2a3f57ca551f72cb4e5dfed78f27be62eec","commits":[{"sha":"3fabbc576203c7fd63808a259adafc5c3cea1838","author":{"name":"Jose Torres","email":"d5cf49c7c48e2e3ac635d3df6cf89d9f9e2d1e5f@gmail.com"},"message":"[SPARK-24040][SS] Support single partition aggregates in continuous processing.\n\n## What changes were proposed in this pull request?\n\nSupport aggregates with exactly 1 partition in continuous processing.\n\nA few small tweaks are needed to make this work:\n\n* Replace currentEpoch tracking with an ThreadLocal. This means that current epoch is scoped to a task rather than a node, but I think that's sustainable even once we add shuffle.\n* Add a new testing-only flag to disable the UnsupportedOperationChecker whitelist of allowed continuous processing nodes. I think this is preferable to writing a pile of custom logic to enforce that there is in fact only 1 partition; we plan to support multi-partition aggregates before the next Spark release, so we'd just have to tear that logic back out.\n* Restart continuous processing queries from the first available uncommitted epoch, rather than one that's guaranteed to be unused. This is required for stateful operators to overwrite partial state from the previous attempt at the epoch, and there was no specific motivation for the original strategy. In another PR before stabilizing the StreamWriter API, we'll need to narrow down and document more precise semantic guarantees for the epoch IDs.\n* We need a single-partition ContinuousMemoryStream. The way MemoryStream is constructed means it can't be a text option like it is for rate source, unfortunately.\n\n## How was this patch tested?\n\nnew unit tests\n\nAuthor: Jose Torres <torres.joseph.f+github@gmail.com>\n\nCloses #21239 from jose-torres/withAggr.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/3fabbc576203c7fd63808a259adafc5c3cea1838"}]}
{"actor":{"display_login":"dongjoon-hyun"}}