{"action":"created","issue":{"url":"https://api.github.com/repos/jupyterlab/jupyterlab/issues/5548","repository_url":"https://api.github.com/repos/jupyterlab/jupyterlab","labels_url":"https://api.github.com/repos/jupyterlab/jupyterlab/issues/5548/labels{/name}","comments_url":"https://api.github.com/repos/jupyterlab/jupyterlab/issues/5548/comments","events_url":"https://api.github.com/repos/jupyterlab/jupyterlab/issues/5548/events","html_url":"https://github.com/jupyterlab/jupyterlab/issues/5548","id":374706102,"node_id":"MDU6SXNzdWUzNzQ3MDYxMDI=","number":5548,"title":"Proposal: JupyterLab Data Bus","user":{"login":"ellisonbg","id":27600,"node_id":"MDQ6VXNlcjI3NjAw","avatar_url":"https://avatars3.githubusercontent.com/u/27600?v=4","gravatar_id":"","url":"https://api.github.com/users/ellisonbg","html_url":"https://github.com/ellisonbg","followers_url":"https://api.github.com/users/ellisonbg/followers","following_url":"https://api.github.com/users/ellisonbg/following{/other_user}","gists_url":"https://api.github.com/users/ellisonbg/gists{/gist_id}","starred_url":"https://api.github.com/users/ellisonbg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ellisonbg/subscriptions","organizations_url":"https://api.github.com/users/ellisonbg/orgs","repos_url":"https://api.github.com/users/ellisonbg/repos","events_url":"https://api.github.com/users/ellisonbg/events{/privacy}","received_events_url":"https://api.github.com/users/ellisonbg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-10-28T01:13:18Z","updated_at":"2018-11-08T09:47:39Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"This is a proposal for the creation of a data bus for JupyterLab. I have talked to a few folks about these ideas, but wanted to put them down in concrete form to open up the discussion and work.\r\n\r\n## Why?\r\n\r\nDifferent JupyterLab extensions are being created to work with datasets of different types in different ways:\r\n\r\n* Notebooks that work with Dataframes, which are visualized and tables, and plots.\r\n* Extensions that allow a user to create data visualizations without code (https://github.com/altair-viz/jupyterlab_voyager and a similar one for plotly).\r\n* The datagrid extension, which visualizes csv files as a table.\r\n\r\nRight now, these different extensions have to create a lot of custom glue to move data between each other. This is painful for developers and artificially limits the ways that data can be used. Once a dataset has been used in JupyterLab in any way, all other extensions that can work with that dataset should immediately become aware of it.\r\n\r\nAdditionally, datasets don't exist in a vaccum. They are surrounded by rich metadata. Such metadata includes associated notebooks, code snippets, links to publications and people that have used the dataset, etc. Furthermore, users collaborating with others will need collaboration facilities, including the ability to create new metadata through comments and annotations on the dataset. Right now, there is no place to store this metadata or associate it with a dataset in JupyterLab.\r\n\r\nFinally, there are a wide range of datasets that cannot be fully downloaded to JupyterLab. This includes large files (CSV, hdf5, etc.), SQL databases, streaming datasets, video, and other API endpoints for working with data (such as s3).\r\n\r\n## What?\r\n\r\nWe propose the creation of a Data Bus that enables datasets to become a first class entity in JupyterLab. This Data Bus will have the following characteristics:\r\n\r\n* It will be based on an abstract notion of a dataset that includes a MIME type, a schema for metadata, some sort of URI that points to the actual data, and typescript interfaces for working with the data.\r\n* While it ins't strictly required, the usage cases we have in mind initially are for immutable data. However, the metadata should be treated as mutable, to allow user generated metadata to be added.\r\n* The Data Bus will be a set of APIs in the frontend that allow other extensions to register as **providers** or **consumers** of datasets of a given MIME type.\r\n* We will want to consider the scoping of the Data Bus in its design. What state should be per user, per notebook server, per workspace, etc.? This will be important to enable productive collaboration. The different scoping will require storing related state in appropriate places (frontend, real-time store, file system, etc.).\r\n* It will be up to those other extensions to determine the semantics around when a datasets is registered with the Data Bus. For example, the file browser could register all datasets in the current working directory, or it could wait until a user opens a dataset to register it.\r\n* Data consumers will be able to query datasets of a given MIME type (or all) as well as watch a signal that fires anytime a new datasets of that type is provided.\r\n* In initial API brainstorming, the consumer/provider APIs ends up looking a lot like a traditional PUB/SUB system. We may want to introduce the idea of topics as is common in PUB/SUB systems, but maybe the MIME types are sufficient.\r\n* When a data provider registers a dataset with the Data Bus, consumers will be notified of the dataset and be able to get a handle of the abstract dataset. Again this handle will include the mimetype, metadata, URI, and interfaces for accessing the data.\r\n* To enable collaborative commenting and annotation on the dataset metadata (imaging having a discussion with someone about a column of a SQL DB table), the metadata will likely need to be stored in the real-time datastore. We have begun to do separate work on a general commenting and annotation system, but the general idea is that comments/annotations will be another table in a real-time datastore schema.\r\n* Additional server side work will likely be required to make remote datasets of different types available to JupyterLab. @danielballan and @ian-r-rose have started to do work on this that integrates with the Jupyter server content API. It is intended for the Data Bus to be independent of the server side work, but integrate cleanly with it. The dataset notion should be sufficiently abstract that the details of where the data comes from is a separate concern.\r\n* We may want to introduce the notion of adaptors between different MIME types. The reason for this is that there may be multiple different MIME types that expose compatible data. An example would be that csv files, SQL DB tables, etc. all may expose a tabular dataset. Some extensions (vega-lite) expect tabular data to be in memory JSON, whereas others (datagrid) can handle dynamically loaded data. An adaptor patterns will make it easier to deal with the realities of different extensions that consume the data.\r\n* User interfaces for browsing datasets that the Data Bus is aware of would be an example of a data consumer.\r\n* User interfaces for rendering a dataset or allowing a user to work with metadata would be done as data consumers.\r\n* In some ways, this is similar to how we do MIME renderers for files and output, but a more general approach that separates registration/providing from the rendering step, and also includes metadata.\r\n\r\n## How\r\n\r\n@ian-r-rose has started some explorations here:\r\n\r\nhttps://github.com/ian-r-rose/jupyterlab-remote-data\r\n\r\nWe also have funding from the Schmidt Foundation @ Cal Poly and are getting that setup to fund other core JupyterLab folks to help out.\r\n\r\n@saulshanabrook @sccolbert @jasongrout @afshin @blink1073 @ian-r-rose\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/jupyterlab/jupyterlab/issues/comments/436935097","html_url":"https://github.com/jupyterlab/jupyterlab/issues/5548#issuecomment-436935097","issue_url":"https://api.github.com/repos/jupyterlab/jupyterlab/issues/5548","id":436935097,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNjkzNTA5Nw==","user":{"login":"10Dev","id":17344588,"node_id":"MDQ6VXNlcjE3MzQ0NTg4","avatar_url":"https://avatars2.githubusercontent.com/u/17344588?v=4","gravatar_id":"","url":"https://api.github.com/users/10Dev","html_url":"https://github.com/10Dev","followers_url":"https://api.github.com/users/10Dev/followers","following_url":"https://api.github.com/users/10Dev/following{/other_user}","gists_url":"https://api.github.com/users/10Dev/gists{/gist_id}","starred_url":"https://api.github.com/users/10Dev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/10Dev/subscriptions","organizations_url":"https://api.github.com/users/10Dev/orgs","repos_url":"https://api.github.com/users/10Dev/repos","events_url":"https://api.github.com/users/10Dev/events{/privacy}","received_events_url":"https://api.github.com/users/10Dev/received_events","type":"User","site_admin":false},"created_at":"2018-11-08T09:47:39Z","updated_at":"2018-11-08T09:47:39Z","author_association":"NONE","body":"FWIW, a reminder to keep #2815 in mind and how this would efficiently work inter-cell."}}
{"actor":{"display_login":"10Dev"}}