{"action":"created","issue":{"url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/509","repository_url":"https://api.github.com/repos/jupyter/enterprise_gateway","labels_url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/509/labels{/name}","comments_url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/509/comments","events_url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/509/events","html_url":"https://github.com/jupyter/enterprise_gateway/issues/509","id":385752240,"node_id":"MDU6SXNzdWUzODU3NTIyNDA=","number":509,"title":"Decoupling notebooks from computation? ","user":{"login":"ziedbouf","id":5924038,"node_id":"MDQ6VXNlcjU5MjQwMzg=","avatar_url":"https://avatars0.githubusercontent.com/u/5924038?v=4","gravatar_id":"","url":"https://api.github.com/users/ziedbouf","html_url":"https://github.com/ziedbouf","followers_url":"https://api.github.com/users/ziedbouf/followers","following_url":"https://api.github.com/users/ziedbouf/following{/other_user}","gists_url":"https://api.github.com/users/ziedbouf/gists{/gist_id}","starred_url":"https://api.github.com/users/ziedbouf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ziedbouf/subscriptions","organizations_url":"https://api.github.com/users/ziedbouf/orgs","repos_url":"https://api.github.com/users/ziedbouf/repos","events_url":"https://api.github.com/users/ziedbouf/events{/privacy}","received_events_url":"https://api.github.com/users/ziedbouf/received_events","type":"User","site_admin":false},"labels":[{"id":1005852039,"node_id":"MDU6TGFiZWwxMDA1ODUyMDM5","url":"https://api.github.com/repos/jupyter/enterprise_gateway/labels/Discussion","name":"Discussion","color":"027d7f","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-11-29T14:06:14Z","updated_at":"2018-11-30T19:36:58Z","closed_at":null,"author_association":"NONE","body":"This is a not an issue per say but more on ideation around the jupyter stack. for the past few weeks  I used Databricks and i like the idea of how they decouple notebooks from the compute infrastructure.\r\n\r\nAlso, i read this [article](https://www.slideshare.net/databricks/bighead-airbnbs-endtoend-machine-learning-platform-with-krishna-puttaswamy-and-andrew-hoh) published by Airbnb and the challenges they founds to scale their analytics infrastructure. It seems they had to deal with the same issue of decoupling notebooks from computation in order to streamline the data analytics operations.\r\n\r\nJupyter stacks in general helps to solve the same problem but with the cost of integrating those components together seems to be the stopping barrier \r\n\r\nI would like to know if there are any initiatives in the same direction?  in case we wish similair stack, how to combine all of those jupyter projects (jupyter hub + jupyter entreprise gatweay and jupyter Lab) together to get similar infrastructure?"},"comment":{"url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/comments/443315575","html_url":"https://github.com/jupyter/enterprise_gateway/issues/509#issuecomment-443315575","issue_url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/509","id":443315575,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MzMxNTU3NQ==","user":{"login":"kevin-bates","id":22599560,"node_id":"MDQ6VXNlcjIyNTk5NTYw","avatar_url":"https://avatars0.githubusercontent.com/u/22599560?v=4","gravatar_id":"","url":"https://api.github.com/users/kevin-bates","html_url":"https://github.com/kevin-bates","followers_url":"https://api.github.com/users/kevin-bates/followers","following_url":"https://api.github.com/users/kevin-bates/following{/other_user}","gists_url":"https://api.github.com/users/kevin-bates/gists{/gist_id}","starred_url":"https://api.github.com/users/kevin-bates/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kevin-bates/subscriptions","organizations_url":"https://api.github.com/users/kevin-bates/orgs","repos_url":"https://api.github.com/users/kevin-bates/repos","events_url":"https://api.github.com/users/kevin-bates/events{/privacy}","received_events_url":"https://api.github.com/users/kevin-bates/received_events","type":"User","site_admin":false},"created_at":"2018-11-30T19:36:58Z","updated_at":"2018-11-30T19:36:58Z","author_association":"COLLABORATOR","body":"@lresende can speak more about scheduling notebooks and ETL/model execution.\r\n\r\nYour second bullet sounds like you want the notebook running on the newly spun up cluster.  However, the title of this topic implies a separation of the notebook from the compute resources (i.e., the notebook's kernel).  I will assume you really want to talk about the latter.\r\n\r\nEG should not get into the business of physically spinning up clusters itself.  However, its pluggable proxy framework does not preclude anyone writing a process proxy that manually spins up a cluster.  Also keep in mind that various frameworks, like k8s, may \"spin up a cluster\" just by virtue of the parameters used in your existing kernel-pod.yaml file.  In fact, their spark implementation will invoke a number of executor pods based on your parameters, so in that sense, its spinning up a set of pods.\r\n\r\nIf the jupyter stack was going to contain code to explicitly spin up clusters on behalf of requests, I would think that kind of thing would be done via JupyterHub - and that may even be possible today.\r\n\r\nYou should also note that there's a pending [Notebook PR](https://github.com/jupyter/notebook/pull/4170) that could disrupt some of this.  However, the goal of that PR is to introduce _kernel providers_ that essentially perform their own management operations.  We will be working to ensure the pluggable proxy framework continues to work in that redesign by making the underlying process launcher/management abstract as well.  I want to point you to that PR because you might have some interest into where that's heading."}}
{"actor":{"display_login":"kevin-bates"}}