{"action":"created","issue":{"url":"https://api.github.com/repos/lablup/backend.ai-manager/issues/73","repository_url":"https://api.github.com/repos/lablup/backend.ai-manager","labels_url":"https://api.github.com/repos/lablup/backend.ai-manager/issues/73/labels{/name}","comments_url":"https://api.github.com/repos/lablup/backend.ai-manager/issues/73/comments","events_url":"https://api.github.com/repos/lablup/backend.ai-manager/issues/73/events","html_url":"https://github.com/lablup/backend.ai-manager/pull/73","id":313624390,"node_id":"MDExOlB1bGxSZXF1ZXN0MTgxMTI0OTc2","number":73,"title":"Scaling group and scaling drivers","user":{"login":"achimnol","id":555156,"node_id":"MDQ6VXNlcjU1NTE1Ng==","avatar_url":"https://avatars3.githubusercontent.com/u/555156?v=4","gravatar_id":"","url":"https://api.github.com/users/achimnol","html_url":"https://github.com/achimnol","followers_url":"https://api.github.com/users/achimnol/followers","following_url":"https://api.github.com/users/achimnol/following{/other_user}","gists_url":"https://api.github.com/users/achimnol/gists{/gist_id}","starred_url":"https://api.github.com/users/achimnol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/achimnol/subscriptions","organizations_url":"https://api.github.com/users/achimnol/orgs","repos_url":"https://api.github.com/users/achimnol/repos","events_url":"https://api.github.com/users/achimnol/events{/privacy}","received_events_url":"https://api.github.com/users/achimnol/received_events","type":"User","site_admin":false},"labels":[{"id":793489362,"node_id":"MDU6TGFiZWw3OTM0ODkzNjI=","url":"https://api.github.com/repos/lablup/backend.ai-manager/labels/API","name":"API","color":"fbca04","default":false},{"id":794603799,"node_id":"MDU6TGFiZWw3OTQ2MDM3OTk=","url":"https://api.github.com/repos/lablup/backend.ai-manager/labels/DB%20changes","name":"DB changes","color":"fef2c0","default":false},{"id":223727376,"node_id":"MDU6TGFiZWwyMjM3MjczNzY=","url":"https://api.github.com/repos/lablup/backend.ai-manager/labels/enhancement","name":"enhancement","color":"1d76db","default":true},{"id":899213897,"node_id":"MDU6TGFiZWw4OTkyMTM4OTc=","url":"https://api.github.com/repos/lablup/backend.ai-manager/labels/in-progress","name":"in-progress","color":"bced9c","default":false}],"state":"open","locked":false,"assignee":{"login":"achimnol","id":555156,"node_id":"MDQ6VXNlcjU1NTE1Ng==","avatar_url":"https://avatars3.githubusercontent.com/u/555156?v=4","gravatar_id":"","url":"https://api.github.com/users/achimnol","html_url":"https://github.com/achimnol","followers_url":"https://api.github.com/users/achimnol/followers","following_url":"https://api.github.com/users/achimnol/following{/other_user}","gists_url":"https://api.github.com/users/achimnol/gists{/gist_id}","starred_url":"https://api.github.com/users/achimnol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/achimnol/subscriptions","organizations_url":"https://api.github.com/users/achimnol/orgs","repos_url":"https://api.github.com/users/achimnol/repos","events_url":"https://api.github.com/users/achimnol/events{/privacy}","received_events_url":"https://api.github.com/users/achimnol/received_events","type":"User","site_admin":false},"assignees":[{"login":"achimnol","id":555156,"node_id":"MDQ6VXNlcjU1NTE1Ng==","avatar_url":"https://avatars3.githubusercontent.com/u/555156?v=4","gravatar_id":"","url":"https://api.github.com/users/achimnol","html_url":"https://github.com/achimnol","followers_url":"https://api.github.com/users/achimnol/followers","following_url":"https://api.github.com/users/achimnol/following{/other_user}","gists_url":"https://api.github.com/users/achimnol/gists{/gist_id}","starred_url":"https://api.github.com/users/achimnol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/achimnol/subscriptions","organizations_url":"https://api.github.com/users/achimnol/orgs","repos_url":"https://api.github.com/users/achimnol/repos","events_url":"https://api.github.com/users/achimnol/events{/privacy}","received_events_url":"https://api.github.com/users/achimnol/received_events","type":"User","site_admin":false},{"login":"Zeniuus","id":24506624,"node_id":"MDQ6VXNlcjI0NTA2NjI0","avatar_url":"https://avatars3.githubusercontent.com/u/24506624?v=4","gravatar_id":"","url":"https://api.github.com/users/Zeniuus","html_url":"https://github.com/Zeniuus","followers_url":"https://api.github.com/users/Zeniuus/followers","following_url":"https://api.github.com/users/Zeniuus/following{/other_user}","gists_url":"https://api.github.com/users/Zeniuus/gists{/gist_id}","starred_url":"https://api.github.com/users/Zeniuus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zeniuus/subscriptions","organizations_url":"https://api.github.com/users/Zeniuus/orgs","repos_url":"https://api.github.com/users/Zeniuus/repos","events_url":"https://api.github.com/users/Zeniuus/events{/privacy}","received_events_url":"https://api.github.com/users/Zeniuus/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/lablup/backend.ai-manager/milestones/12","html_url":"https://github.com/lablup/backend.ai-manager/milestone/12","labels_url":"https://api.github.com/repos/lablup/backend.ai-manager/milestones/12/labels","id":3491921,"node_id":"MDk6TWlsZXN0b25lMzQ5MTkyMQ==","number":12,"title":"18.12","description":"","creator":{"login":"achimnol","id":555156,"node_id":"MDQ6VXNlcjU1NTE1Ng==","avatar_url":"https://avatars3.githubusercontent.com/u/555156?v=4","gravatar_id":"","url":"https://api.github.com/users/achimnol","html_url":"https://github.com/achimnol","followers_url":"https://api.github.com/users/achimnol/followers","following_url":"https://api.github.com/users/achimnol/following{/other_user}","gists_url":"https://api.github.com/users/achimnol/gists{/gist_id}","starred_url":"https://api.github.com/users/achimnol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/achimnol/subscriptions","organizations_url":"https://api.github.com/users/achimnol/orgs","repos_url":"https://api.github.com/users/achimnol/repos","events_url":"https://api.github.com/users/achimnol/events{/privacy}","received_events_url":"https://api.github.com/users/achimnol/received_events","type":"User","site_admin":false},"open_issues":29,"closed_issues":4,"state":"open","created_at":"2018-07-12T15:38:52Z","updated_at":"2018-11-06T09:34:18Z","due_on":"2018-12-28T08:00:00Z","closed_at":null},"comments":6,"created_at":"2018-04-12T08:25:02Z","updated_at":"2018-11-08T05:00:37Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/lablup/backend.ai-manager/pulls/73","html_url":"https://github.com/lablup/backend.ai-manager/pull/73","diff_url":"https://github.com/lablup/backend.ai-manager/pull/73.diff","patch_url":"https://github.com/lablup/backend.ai-manager/pull/73.patch"},"body":"Let's implement policy-based scalers and scaling groups.\r\n\r\nConcept:\r\n* On clouds\r\n  - Agents are assigned instance metadata describing their scaling group.\r\n  - The scaling driver of each group runs its own auto-scaling policy and algorithm. It may be a proxy of vendor-specific auto-scaling APIs.\r\n  - The scaling driver may be a proxy of other container orchestration framework such as Kubernetes.\r\n* On premise\r\n  - The default policy is \"fixed\".\r\n  - Let's the users be able to provide their own plugins for external batch job schedulers such as:\r\n    - https://github.com/IBMSpectrumComputing/lsf-python-api\r\n    - http://www.adaptivecomputing.com/products/torque/\r\n    - http://www.arc.ox.ac.uk/content/torque-job-scheduler\r\n    - etc.\r\n\r\nPolicy options:\r\n* image auto-update\r\n  - on/off\r\n  - update check interval\r\n* agent auto-update\r\n  - on/off\r\n  - idle wait time after last kernel termination on agent\r\n* container termination policy\r\n  - auto-terminate after specified idle time / persistent\r\n* external agents\r\n  - on/off\r\n  - public key (external agents should provide the auth message encrypted using its own private key)\r\n  - limit (max number of allowed external agents)\r\n\r\nBatch job functions:\r\n* Get the job list and their status\r\n* Submit a job (input + output + program + resource spec)\r\n  - `JobSpecAdaptor`\r\n* Get the cluster node list\r\n* Get/set the scheduler configurations\r\n* (Optional) Cancel / interrupt a job\r\n\r\nRelated:\r\n* lablup/backend.ai-agent#5 — container termination policy (idle / persistent / etc.)\r\n* lablup/backend.ai-agent#39 — on/off option for self-autoupgrade\r\n* https://github.com/lablup/backend.ai-agent/projects/2 — on/off option and auth config to accept external agents"},"comment":{"url":"https://api.github.com/repos/lablup/backend.ai-manager/issues/comments/436873959","html_url":"https://github.com/lablup/backend.ai-manager/pull/73#issuecomment-436873959","issue_url":"https://api.github.com/repos/lablup/backend.ai-manager/issues/73","id":436873959,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNjg3Mzk1OQ==","user":{"login":"Zeniuus","id":24506624,"node_id":"MDQ6VXNlcjI0NTA2NjI0","avatar_url":"https://avatars3.githubusercontent.com/u/24506624?v=4","gravatar_id":"","url":"https://api.github.com/users/Zeniuus","html_url":"https://github.com/Zeniuus","followers_url":"https://api.github.com/users/Zeniuus/followers","following_url":"https://api.github.com/users/Zeniuus/following{/other_user}","gists_url":"https://api.github.com/users/Zeniuus/gists{/gist_id}","starred_url":"https://api.github.com/users/Zeniuus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zeniuus/subscriptions","organizations_url":"https://api.github.com/users/Zeniuus/orgs","repos_url":"https://api.github.com/users/Zeniuus/repos","events_url":"https://api.github.com/users/Zeniuus/events{/privacy}","received_events_url":"https://api.github.com/users/Zeniuus/received_events","type":"User","site_admin":false},"created_at":"2018-11-08T05:00:37Z","updated_at":"2018-11-08T05:00:37Z","author_association":"CONTRIBUTOR","body":"I have a doubt about total separation of `AbstractScalingDriver` and `AbstractJobScheduler`.\r\n\r\nConsider the following case. `AbstractJobScheduler` implements a policy that assigns kernels with no GPU usage to only t2.2xlarge instances, never to p2.xlarge instances, to minimize the usage of expensive p2.xlarge instance, whereas `AbstractScalingDriver` determines to scale out when the total sum of resources of any type of agents is insufficient to deal with all pending `SessionCreationRequest`s. And currently, there is an only one p2.xlarge instance with no session.\r\n\r\nIn this case, if session creation request with no GPU usage comes in, then `AbstractScalingDriver` would not scale out because there are enough CPU resource shares in the p2.xlarge instance, and `AbstractJobScheduler` would not schedule the request since there is no t2.2xlarge instance. So every CPU-only kernels will be pending infinitely.\r\n\r\nBut adding a method like `calc_required_instances` to `AbstractJobScheduler` does not seem to be right since detail logic of how to scale out/in is the responsibility of `AbstractScalingDriver`.\r\n\r\nDo you have any idea for this issue?"}}
{"actor":{"display_login":"Zeniuus"}}