{"action":"created","issue":{"url":"https://api.github.com/repos/LiberTEM/LiberTEM/issues/205","repository_url":"https://api.github.com/repos/LiberTEM/LiberTEM","labels_url":"https://api.github.com/repos/LiberTEM/LiberTEM/issues/205/labels{/name}","comments_url":"https://api.github.com/repos/LiberTEM/LiberTEM/issues/205/comments","events_url":"https://api.github.com/repos/LiberTEM/LiberTEM/issues/205/events","html_url":"https://github.com/LiberTEM/LiberTEM/issues/205","id":390344525,"node_id":"MDU6SXNzdWUzOTAzNDQ1MjU=","number":205,"title":"Do all filesystem operations via the executor","user":{"login":"sk1p","id":5778,"node_id":"MDQ6VXNlcjU3Nzg=","avatar_url":"https://avatars2.githubusercontent.com/u/5778?v=4","gravatar_id":"","url":"https://api.github.com/users/sk1p","html_url":"https://github.com/sk1p","followers_url":"https://api.github.com/users/sk1p/followers","following_url":"https://api.github.com/users/sk1p/following{/other_user}","gists_url":"https://api.github.com/users/sk1p/gists{/gist_id}","starred_url":"https://api.github.com/users/sk1p/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sk1p/subscriptions","organizations_url":"https://api.github.com/users/sk1p/orgs","repos_url":"https://api.github.com/users/sk1p/repos","events_url":"https://api.github.com/users/sk1p/events{/privacy}","received_events_url":"https://api.github.com/users/sk1p/received_events","type":"User","site_admin":false},"labels":[{"id":1133991061,"node_id":"MDU6TGFiZWwxMTMzOTkxMDYx","url":"https://api.github.com/repos/LiberTEM/LiberTEM/labels/AAA-Priority","name":"AAA-Priority","color":"ed5a7e","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-12-12T18:05:18Z","updated_at":"2018-12-13T20:06:38Z","closed_at":"2018-12-13T17:50:42Z","author_association":"COLLABORATOR","body":"We want to do all filesystem operations via the executor. This actually has two reasons:\r\n\r\n1) They may take some time, for example when listing a large directory, or listing a NFS share. Running them via the executor means we get to do this in the background.\r\n2) The worker nodes may not have the same filesystem as the API server or `dask.distributed` `Client`\r\n\r\nIn the first iteration, this means we make sure `Context.load(...)` pushes the dataset load operation into the cluster, including the `check_valid()` call. The web API basically needs the same changes.\r\n\r\nThe next step would be to update the web API to also do directory listing / existence checks etc. via functions pushed into the cluster. \r\n\r\nThis is highly related to #154 which also may need to do more work on dataset initialization. Also related is #186 as listing a directory and opening files should take priority over another job that may take a while to finish.\r\n\r\n- [x] add ability to run simpler functions via the executor\r\n- [x] update `Context` and `DataSetDetailHandler` to  do `load` and `check_valid` via executor\r\n- [x] test connecting via `Context` to a remote dask scheduler\r\n- [x] update `DataSetDetectHandler` \r\n- [x] update `LocalFSBrowseHandler`\r\n- [x] test starting the web client with a connection to a remote dask scheduler"},"comment":{"url":"https://api.github.com/repos/LiberTEM/LiberTEM/issues/comments/447103047","html_url":"https://github.com/LiberTEM/LiberTEM/issues/205#issuecomment-447103047","issue_url":"https://api.github.com/repos/LiberTEM/LiberTEM/issues/205","id":447103047,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NzEwMzA0Nw==","user":{"login":"uellue","id":25689052,"node_id":"MDQ6VXNlcjI1Njg5MDUy","avatar_url":"https://avatars2.githubusercontent.com/u/25689052?v=4","gravatar_id":"","url":"https://api.github.com/users/uellue","html_url":"https://github.com/uellue","followers_url":"https://api.github.com/users/uellue/followers","following_url":"https://api.github.com/users/uellue/following{/other_user}","gists_url":"https://api.github.com/users/uellue/gists{/gist_id}","starred_url":"https://api.github.com/users/uellue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uellue/subscriptions","organizations_url":"https://api.github.com/users/uellue/orgs","repos_url":"https://api.github.com/users/uellue/repos","events_url":"https://api.github.com/users/uellue/events{/privacy}","received_events_url":"https://api.github.com/users/uellue/received_events","type":"User","site_admin":false},"created_at":"2018-12-13T20:06:38Z","updated_at":"2018-12-13T20:06:38Z","author_association":"MEMBER","body":"Tested with the async Digital Micrograph prototype with processing on moellenstedt. It ROCKS! 👍 👍 👍 "}}
{"actor":{"display_login":"uellue"}}