{"push_id":1244525190,"size":6,"distinct_size":2,"ref":"refs/heads/executor-reconnect","head":"565025ef73aa8b411b5849db0ccfd1ba2d40aec1","before":"b280f5954d2196005f6f69ae813e1c2b4bf4121e","commits":[{"sha":"0acc02e6b7e8e636d3c303a726602c1fb8d8839f","author":{"name":"Matthew Rocklin","email":"684543d2204ce7ddf1ffcf21500e08ea3b6a0d99@gmail.com"},"message":"Refactor Scheduler (#398)\n\nThis is a large refactor of the distributed scheduler.\r\n\r\n### Problem\r\n\r\n* Recently we've found deadlocks in the scheduler.  These have become increasingly time consuming to identify and resolve.\r\n* No one feels comfortable operating on the scheduler code except for myself\r\n\r\nHistorically as we've added more and more features we convolute the scheduler code more and more.  This has led to a difficult and time consuming situation where resilience and performance are hard to guarantee, and hard to fix once an issue has been identified.\r\n\r\nThe previous architecture was a set of scheduler methods that called each other in a difficult-to-reason-about way that were triggered by a few events.\r\n\r\n### Solution\r\n\r\nThis PR adds a bit of top-down design to the management of scheduler state.  This has a few positive effects:\r\n\r\n1.  The resulting code is cleaner and easier to follow\r\n2.  The design of the scheduler is easier to talk about.  What were previously conventions and ways of thinking about what's happening have become actual function calls\r\n3.  Much of the state changes happen in a central place.  This enables consistent logging, and runtime checking, which greatly accelerate the debugging and development process.\r\n\r\n### Transitions and Stimuli\r\n\r\nTo learn more about the new design you should read the doc page that comes along with this PR\r\n\r\n* Docs: https://github.com/mrocklin/distributed/blob/scheduler-refactor/docs/source/scheduler.rst\r\n* Code: https://github.com/mrocklin/distributed/blob/scheduler-refactor/distributed/scheduler.py","distinct":false,"url":"https://api.github.com/repos/mrocklin/distributed/commits/0acc02e6b7e8e636d3c303a726602c1fb8d8839f"},{"sha":"8b1edff5c9c37e2811190b448bf0cbf99e0a58b9","author":{"name":"Matthew Rocklin","email":"684543d2204ce7ddf1ffcf21500e08ea3b6a0d99@gmail.com"},"message":"Avoid large literal in code\n\nsetup.py install seems to actually evaluate the constant b'0' * 2**31\nWe avoid this by wrapping in a needless function call.","distinct":false,"url":"https://api.github.com/repos/mrocklin/distributed/commits/8b1edff5c9c37e2811190b448bf0cbf99e0a58b9"},{"sha":"19bc45dc28e8c144e17bbd4aa3ee4c7d8635e9e8","author":{"name":"Matthew Rocklin","email":"684543d2204ce7ddf1ffcf21500e08ea3b6a0d99@gmail.com"},"message":"remove known_dtype from dataframe tests","distinct":false,"url":"https://api.github.com/repos/mrocklin/distributed/commits/19bc45dc28e8c144e17bbd4aa3ee4c7d8635e9e8"},{"sha":"012c2527c668732e23b2049969968d8f2cfd6786","author":{"name":"Matthew Rocklin","email":"684543d2204ce7ddf1ffcf21500e08ea3b6a0d99@gmail.com"},"message":"Update collections tests for dask.dataframe refactor","distinct":false,"url":"https://api.github.com/repos/mrocklin/distributed/commits/012c2527c668732e23b2049969968d8f2cfd6786"},{"sha":"06dfcba623f36f8f3a025274568ff6ceea561bec","author":{"name":"Matthew Rocklin","email":"684543d2204ce7ddf1ffcf21500e08ea3b6a0d99@gmail.com"},"message":"add executor reconnect test","distinct":true,"url":"https://api.github.com/repos/mrocklin/distributed/commits/06dfcba623f36f8f3a025274568ff6ceea561bec"},{"sha":"565025ef73aa8b411b5849db0ccfd1ba2d40aec1","author":{"name":"Matthew Rocklin","email":"684543d2204ce7ddf1ffcf21500e08ea3b6a0d99@gmail.com"},"message":"Reconnect Executor after scheduler fails","distinct":true,"url":"https://api.github.com/repos/mrocklin/distributed/commits/565025ef73aa8b411b5849db0ccfd1ba2d40aec1"}]}
{"actor":{"display_login":"mrocklin"}}