{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/serving/issues/422","repository_url":"https://api.github.com/repos/tensorflow/serving","labels_url":"https://api.github.com/repos/tensorflow/serving/issues/422/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/serving/issues/422/comments","events_url":"https://api.github.com/repos/tensorflow/serving/issues/422/events","html_url":"https://github.com/tensorflow/serving/issues/422","id":224480088,"node_id":"MDU6SXNzdWUyMjQ0ODAwODg=","number":422,"title":"Best practice: loading new models during runtime (not versions)","user":{"login":"markusnagel","id":9336024,"node_id":"MDQ6VXNlcjkzMzYwMjQ=","avatar_url":"https://avatars0.githubusercontent.com/u/9336024?v=4","gravatar_id":"","url":"https://api.github.com/users/markusnagel","html_url":"https://github.com/markusnagel","followers_url":"https://api.github.com/users/markusnagel/followers","following_url":"https://api.github.com/users/markusnagel/following{/other_user}","gists_url":"https://api.github.com/users/markusnagel/gists{/gist_id}","starred_url":"https://api.github.com/users/markusnagel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markusnagel/subscriptions","organizations_url":"https://api.github.com/users/markusnagel/orgs","repos_url":"https://api.github.com/users/markusnagel/repos","events_url":"https://api.github.com/users/markusnagel/events{/privacy}","received_events_url":"https://api.github.com/users/markusnagel/received_events","type":"User","site_admin":false},"labels":[{"id":327079593,"node_id":"MDU6TGFiZWwzMjcwNzk1OTM=","url":"https://api.github.com/repos/tensorflow/serving/labels/contributions%20welcome","name":"contributions welcome","color":"009800","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2017-04-26T14:05:55Z","updated_at":"2018-09-14T20:58:37Z","closed_at":null,"author_association":"NONE","body":"I'm curious to see/discuss what are the best options to deploy new models during runtime. I learned from the tutorials and examples how new versions of a model are loaded as well as how to load multiple models (at the start of the server). Unfortunately I could not find the functionality of loading new models during runtime. (I'm aware that parts where discussed in this issue #380.) I guess my first question is if there exists already any type of dynamic loading of new models during runtime? (I think technically all components should be there, however I could not find a 'trigger' which starts the process of loading the model.)\r\n\r\nIn general I can think of several ways to serve multiple models during runtime:\r\n- Use a separate server for each model. A small manager module could start a new server or even docker container for each model. (Question: How to best forward the prediction request to the right server?)\r\n- Similar to watching multiple versions on the file system, we could watch the parent directory if a new folder for a new model.\r\n- If we use the 'custom_model_config' we periodically check it for changes and see if a new model is in there (or one is missing).\r\n- Have a gRPC call which sends the required information for the new model.\r\n\r\nIt would be great if some of the developers could post their opinion on how such a case should be handled or people share their opinion on best practices related to deploying new models.\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/serving/issues/comments/421482627","html_url":"https://github.com/tensorflow/serving/issues/422#issuecomment-421482627","issue_url":"https://api.github.com/repos/tensorflow/serving/issues/422","id":421482627,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMTQ4MjYyNw==","user":{"login":"yuzheng21","id":3047403,"node_id":"MDQ6VXNlcjMwNDc0MDM=","avatar_url":"https://avatars0.githubusercontent.com/u/3047403?v=4","gravatar_id":"","url":"https://api.github.com/users/yuzheng21","html_url":"https://github.com/yuzheng21","followers_url":"https://api.github.com/users/yuzheng21/followers","following_url":"https://api.github.com/users/yuzheng21/following{/other_user}","gists_url":"https://api.github.com/users/yuzheng21/gists{/gist_id}","starred_url":"https://api.github.com/users/yuzheng21/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yuzheng21/subscriptions","organizations_url":"https://api.github.com/users/yuzheng21/orgs","repos_url":"https://api.github.com/users/yuzheng21/repos","events_url":"https://api.github.com/users/yuzheng21/events{/privacy}","received_events_url":"https://api.github.com/users/yuzheng21/received_events","type":"User","site_admin":false},"created_at":"2018-09-14T20:58:37Z","updated_at":"2018-09-14T20:58:37Z","author_association":"NONE","body":"Hi @chrisolston, is the second item (delta api) in this comment implemented? https://github.com/tensorflow/serving/issues/422#issuecomment-340527225"}}
{"actor":{"display_login":"yuzheng21"}}