{"action":"created","issue":{"url":"https://api.github.com/repos/ray-project/ray/issues/3033","repository_url":"https://api.github.com/repos/ray-project/ray","labels_url":"https://api.github.com/repos/ray-project/ray/issues/3033/labels{/name}","comments_url":"https://api.github.com/repos/ray-project/ray/issues/3033/comments","events_url":"https://api.github.com/repos/ray-project/ray/issues/3033/events","html_url":"https://github.com/ray-project/ray/pull/3033","id":367510897,"node_id":"MDExOlB1bGxSZXF1ZXN0MjIwOTIwOTM0","number":3033,"title":"[sgd] Merge sharded param server based SGD implementation","user":{"login":"ericl","id":14922,"node_id":"MDQ6VXNlcjE0OTIy","avatar_url":"https://avatars3.githubusercontent.com/u/14922?v=4","gravatar_id":"","url":"https://api.github.com/users/ericl","html_url":"https://github.com/ericl","followers_url":"https://api.github.com/users/ericl/followers","following_url":"https://api.github.com/users/ericl/following{/other_user}","gists_url":"https://api.github.com/users/ericl/gists{/gist_id}","starred_url":"https://api.github.com/users/ericl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericl/subscriptions","organizations_url":"https://api.github.com/users/ericl/orgs","repos_url":"https://api.github.com/users/ericl/repos","events_url":"https://api.github.com/users/ericl/events{/privacy}","received_events_url":"https://api.github.com/users/ericl/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":27,"created_at":"2018-10-07T01:21:44Z","updated_at":"2018-10-27T18:51:04Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/ray-project/ray/pulls/3033","html_url":"https://github.com/ray-project/ray/pull/3033","diff_url":"https://github.com/ray-project/ray/pull/3033.diff","patch_url":"https://github.com/ray-project/ray/pull/3033.patch"},"body":"## What do these changes do?\r\n\r\nThis includes most of the TF code used for the OSDI experiment. Perf sanity check on p3.16xl instances: Overall scaling looks OK. However, single-node perf is noticeably worse than before (200 images/s/gpu is expected on single node).  Not sure why.\r\n\r\nCaveats for numbers below:\r\n- Single node perf seems lower than expected. There might be some issue with the benchmark model.\r\n- I didn't enable huge pages.\r\n\r\n```\r\n$ RAY_USE_XRAY=1 ./test_sgd.py --gpu --batch-size=64 --num-workers=N \\\r\n  --devices-per-worker=M --strategy=<simple|ps> \\\r\n  --warmup --object-store-memory=10000000000\r\n\r\nImages per second total\r\ngpus total              | simple | ps\r\n========================================\r\n1                       | 218\r\n2 (1 worker)            | 136    | 331\r\n4 (1 worker)            | 267    | 606\r\n4 (2 workers)           | 176    | 623    <--- OSDI perf was 815\r\n8 (1 worker)            | 516    | 889\r\n8 (2 workers)           | 349    | 1031\r\n16 (2 nodes, 2 workers) | 600    | 1661\r\n16 (2 nodes, 4 workers) | 468    | 1712   <--- OSDI perf was 1817\r\n```\r\n\r\n## Related issue number\r\n\r\nhttps://github.com/ray-project/ray/issues/3019"},"comment":{"url":"https://api.github.com/repos/ray-project/ray/issues/comments/433645875","html_url":"https://github.com/ray-project/ray/pull/3033#issuecomment-433645875","issue_url":"https://api.github.com/repos/ray-project/ray/issues/3033","id":433645875,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMzY0NTg3NQ==","user":{"login":"richardliaw","id":4529381,"node_id":"MDQ6VXNlcjQ1MjkzODE=","avatar_url":"https://avatars0.githubusercontent.com/u/4529381?v=4","gravatar_id":"","url":"https://api.github.com/users/richardliaw","html_url":"https://github.com/richardliaw","followers_url":"https://api.github.com/users/richardliaw/followers","following_url":"https://api.github.com/users/richardliaw/following{/other_user}","gists_url":"https://api.github.com/users/richardliaw/gists{/gist_id}","starred_url":"https://api.github.com/users/richardliaw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/richardliaw/subscriptions","organizations_url":"https://api.github.com/users/richardliaw/orgs","repos_url":"https://api.github.com/users/richardliaw/repos","events_url":"https://api.github.com/users/richardliaw/events{/privacy}","received_events_url":"https://api.github.com/users/richardliaw/received_events","type":"User","site_admin":false},"created_at":"2018-10-27T18:51:04Z","updated_at":"2018-10-27T18:51:04Z","author_association":"CONTRIBUTOR","body":"(maybe should sit down with @stephanie-wang to walk over code before merge)"}}
{"actor":{"display_login":"richardliaw"}}