{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/3926","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/3926/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/3926/comments","events_url":"https://api.github.com/repos/dask/dask/issues/3926/events","html_url":"https://github.com/dask/dask/pull/3926","id":356027719,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEyNDIwMTcx","number":3926,"title":"RFC: Example of enabling glob with HTTP file-system","user":{"login":"martindurant","id":6042212,"node_id":"MDQ6VXNlcjYwNDIyMTI=","avatar_url":"https://avatars1.githubusercontent.com/u/6042212?v=4","gravatar_id":"","url":"https://api.github.com/users/martindurant","html_url":"https://github.com/martindurant","followers_url":"https://api.github.com/users/martindurant/followers","following_url":"https://api.github.com/users/martindurant/following{/other_user}","gists_url":"https://api.github.com/users/martindurant/gists{/gist_id}","starred_url":"https://api.github.com/users/martindurant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martindurant/subscriptions","organizations_url":"https://api.github.com/users/martindurant/orgs","repos_url":"https://api.github.com/users/martindurant/repos","events_url":"https://api.github.com/users/martindurant/events{/privacy}","received_events_url":"https://api.github.com/users/martindurant/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-08-31T15:31:17Z","updated_at":"2018-09-10T16:01:19Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/dask/dask/pulls/3926","html_url":"https://github.com/dask/dask/pull/3926","diff_url":"https://github.com/dask/dask/pull/3926.diff","patch_url":"https://github.com/dask/dask/pull/3926.patch"},"body":"(not meant to be merged, just for discussion)\r\n\r\nI have implemented a simplistic way to do `ls` for HTTP, which has been often requested. It laods the target page, and looks for HREFs that look like they are children. I'm sure it has many cases that would break it, but the point is to demonstrate reuse of code in [fsspec](https://github.com/martindurant/filesystem_spec/blob/master/fsspec/spec.py), so we get `walk` and `glob` for free by defining `ls`. \r\n\r\nThis kind of thing begs the question: do file-systems like this belong in dask? What about the rest of the bytes functionality, some of which is very dask specific, some of which is not?"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/419966489","html_url":"https://github.com/dask/dask/pull/3926#issuecomment-419966489","issue_url":"https://api.github.com/repos/dask/dask/issues/3926","id":419966489,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTk2NjQ4OQ==","user":{"login":"martindurant","id":6042212,"node_id":"MDQ6VXNlcjYwNDIyMTI=","avatar_url":"https://avatars1.githubusercontent.com/u/6042212?v=4","gravatar_id":"","url":"https://api.github.com/users/martindurant","html_url":"https://github.com/martindurant","followers_url":"https://api.github.com/users/martindurant/followers","following_url":"https://api.github.com/users/martindurant/following{/other_user}","gists_url":"https://api.github.com/users/martindurant/gists{/gist_id}","starred_url":"https://api.github.com/users/martindurant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martindurant/subscriptions","organizations_url":"https://api.github.com/users/martindurant/orgs","repos_url":"https://api.github.com/users/martindurant/repos","events_url":"https://api.github.com/users/martindurant/events{/privacy}","received_events_url":"https://api.github.com/users/martindurant/received_events","type":"User","site_admin":false},"created_at":"2018-09-10T16:01:19Z","updated_at":"2018-09-10T16:01:19Z","author_association":"MEMBER","body":"Yes, the idea was to get a conversation going on a couple of interrelated things, which is why this makes for a poor PR:\r\n- the usage of ls/walk/glob in HTTP, is it a decent implementation, should it be updates in dask using the existing code in fsspec. This question (from Intake) started things.\r\n- an example of depending on fsspec, which is unreleased and largely unvetted; could act as a motivator for pushing that forward\r\n- general question on whether should move some/all FS implementation code over to fsspec, since it doesn't really belong in dask. That idea could more generally apply to much of dask.bytes; e.g., `open_files` is *very useful* everywhere, as Intake has found, but some stuff in there really is dasky."}}
{"actor":{"display_login":"martindurant"}}