{"push_id":3002607352,"size":5,"distinct_size":0,"ref":"refs/heads/tonytung-mp-refactor","head":"436087e0b5ec1d32e9695432d1628ad31521df85","before":"75a4e4617151c85b0f26386e05e9d433d7a8cbfc","commits":[{"sha":"215e98f494dd870bd851154b9af79ee5381e6ae1","author":{"name":"Tony Tung","email":"230d196ff2416c9232661c39ad7b0996916b1710@chanzuckerberg.com"},"message":"Wrapper class for the xarray in ImageStack.\n\nRight now, we only wrap the xarray.  Next we will add the shmem construct that will allow us to pass data to and from subprocesses without copying.\n\nTest plan: make -j fast run_notebooks","distinct":false,"url":"https://api.github.com/repos/spacetx/starfish/commits/215e98f494dd870bd851154b9af79ee5381e6ae1"},{"sha":"3c1ad9978c959c14cbb95de778759c4dc7685396","author":{"name":"Tony Tung","email":"230d196ff2416c9232661c39ad7b0996916b1710@chanzuckerberg.com"},"message":"fake commit","distinct":false,"url":"https://api.github.com/repos/spacetx/starfish/commits/3c1ad9978c959c14cbb95de778759c4dc7685396"},{"sha":"2a6a9eeaa78764feb43f3f40bf8230071853f134","author":{"name":"Tony Tung","email":"230d196ff2416c9232661c39ad7b0996916b1710@chanzuckerberg.com"},"message":"merge","distinct":false,"url":"https://api.github.com/repos/spacetx/starfish/commits/2a6a9eeaa78764feb43f3f40bf8230071853f134"},{"sha":"eab20b61ae12b0ad4b6cfa575b8c114aa793d5ce","author":{"name":"Tony Tung","email":"230d196ff2416c9232661c39ad7b0996916b1710@chanzuckerberg.com"},"message":"Make MPDataArray actually shared-memory\n\nhttps://github.com/spacetx/starfish/pull/737 created the wrapper class for xr.DataArray, but fundamentally, it was still a copy-on-write construct that did not allow multiple writers across processes to operate on it.  This swaps out the underpinnings with the constructs introduced and tested in https://github.com/spacetx/starfish/pull/738, such that processes that we fork out in `ImageStack` can cooperatively work on multiple parts of the tensor concurrently.\n\nWhenever we construct a `MPDataArray`, we now construct a `multiprocessing.Array`, and layer a `numpy` object on top of that memory.  Then we construct a `xr.DataArray` around that `numpy` object.  Fun, right?\n\nWe freely copy xr.DataArray around when we want to run `ImageStack.apply` not in-place, so we have a `__deepcopy__` method in `MPDataArray` to copy correctly (i.e., create a new `multiprocessing.Array`, and layer everything on top of that).\n\nWe also intercept `xr.DataArray``s copy method, such that if someone attempts to deepcopy the xarray directly, we warn them that it has consequences.\n\nTest plan: Added a few tests:\n1. a test that exercises `xr_deepcopy`\n2. a test that exercises `copy.deepcopy(..)` on an ImageStack.\n3. a test that exercises `copy.deepcopy(..)` on an ImageStack's xarray (which will work but will trigger a warning).","distinct":false,"url":"https://api.github.com/repos/spacetx/starfish/commits/eab20b61ae12b0ad4b6cfa575b8c114aa793d5ce"},{"sha":"436087e0b5ec1d32e9695432d1628ad31521df85","author":{"name":"Tony Tung","email":"230d196ff2416c9232661c39ad7b0996916b1710@chanzuckerberg.com"},"message":"Pass the entire tensor to child processes as well as the slice indices\n\nCurrently, we are slicing the tensor in the parent process, and passing the slices to each of the child processes.  To share memory more efficiently, we want to pass the entire tensor (and the backing shared memory construct) to the child processes, and instead just pass the slice indices to the child process.  The child process will slice out the part of the tensor it is interested in, and then pass it to the callable (filter, spot calling code, etc.).\n\n`_multiprocessing_workflow` is the entry point for the child processes.  It retrieves the tensor using the constructs provided by https://github.com/spacetx/starfish/pull/738, and slices the data.\n\nWe also add a finalizer callable, which is something defined by the caller, has access to the entire tensor, and the slice indices.  This is to be used to merge the results into the original tensor for the `apply` case.  This will be sent out in the next PR. :)\n\nRemove `_iter_tiles` as it is no longer needed.\n\nDepends on #736, #737, #738, #739, #740","distinct":false,"url":"https://api.github.com/repos/spacetx/starfish/commits/436087e0b5ec1d32e9695432d1628ad31521df85"}]}
{"actor":{"display_login":"ttung"}}