{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/29785","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/29785/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/29785/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/29785/events","html_url":"https://github.com/pytorch/pytorch/pull/29785","id":522569734,"node_id":"MDExOlB1bGxSZXF1ZXN0MzQwNzQ1NzYz","number":29785,"title":"[pytorch] Improve process_group_agent() serialization speed","user":{"login":"jjlilley","id":55768316,"node_id":"MDQ6VXNlcjU1NzY4MzE2","avatar_url":"https://avatars0.githubusercontent.com/u/55768316?v=4","gravatar_id":"","url":"https://api.github.com/users/jjlilley","html_url":"https://github.com/jjlilley","followers_url":"https://api.github.com/users/jjlilley/followers","following_url":"https://api.github.com/users/jjlilley/following{/other_user}","gists_url":"https://api.github.com/users/jjlilley/gists{/gist_id}","starred_url":"https://api.github.com/users/jjlilley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jjlilley/subscriptions","organizations_url":"https://api.github.com/users/jjlilley/orgs","repos_url":"https://api.github.com/users/jjlilley/repos","events_url":"https://api.github.com/users/jjlilley/events{/privacy}","received_events_url":"https://api.github.com/users/jjlilley/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2019-11-14T01:33:32Z","updated_at":"2019-12-01T21:36:10Z","closed_at":"2019-11-28T17:58:10Z","author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/pytorch/pytorch/pulls/29785","html_url":"https://github.com/pytorch/pytorch/pull/29785","diff_url":"https://github.com/pytorch/pytorch/pull/29785.diff","patch_url":"https://github.com/pytorch/pytorch/pull/29785.patch"},"body":"Stack from [ghstack](https://github.com/ezyang/ghstack):\n* **#29785 [pytorch] Improve process_group_agent() serialization speed**\n\r\nTLDR: This change improves process_group's serialization speed:\r\n\r\n```\r\n  Serialize_Tensor64:     12.38us ->   1.99us  (~-84%)\r\n  Deserialize_Tensor64:   33.89us ->   5.62us  (~-84%)\r\n  Serialize_Tensor1M:    525.74us -> 285.43us  (~-45%)\r\n  Deserialize_Tensor1M:  892.61us -> 273.68us  (~-70%)\r\n```\r\n\r\nAfter speaking with the jit team, we had consensus that torch::save()/load()\r\nare somewhat high-overhead for RPC serialization, mostly intended for\r\npersistent disk data.\r\n\r\n(Particularly, for large tensors, 35% of the time is spent in CRC checking, even\r\nwith the fb-side changes to subsitute 40x faster SSE-accelerated crc checking;\r\nAlso, for small tensors, the zip container overhead is considerable, as is the\r\noverhead of lexing/parsing an embedded text python program for each RPC).\r\n\r\nThe jit team encouraged us to use jit::pickler, with the WriteableTensorData\r\nway of outputting result tensors (not the default side-tensor table, or\r\nwith pickling the actual tensors). This ends up just pickling some tensor\r\nmetadata, and giving us some tensor blobs that we can mindlessly\r\nblit over the wire (they copy to cpu memory if needed).\r\n\r\nThere is yet no standardized container format for the pickled data\r\n(there is jit::pickle_save() checked in, but but it's experimental,\r\nno load function is yet provided), but they encouraged us to just use\r\nsomething sensible for this, and possibly revisit later. For now, I made\r\nthe directory headers slightly http-inspired.\r\n\r\nNote that serialization is just one component of the pipeline, but that\r\nsaid, we also see reasonable reductions in end-to-end echo times (noisier):\r\n\r\n```\r\n   ProcessGroupAgent_Echo(Tensor_Small)   855.25us -> 492.65us  (~-42%)\r\n   ProcessGroupAgent_Echo(Tensor_1M)       10.82ms -> 6.94ms    (~-35%)\r\n   ProcessGroupAgent_Echo(Small_NoTensor) 688.82us -> 301.72us  (~-56%)\r\n   ProcessGroupAgent_Echo(1MB_NoTensor)     4.65ms -> 3.71ms    (~-20%)\r\n```\r\n\r\nI moved the \"wire serialization\" logic to a separate file to assist with\r\nunittesting.\r\n\r\nDifferential Revision: [D18493938](https://our.internmc.facebook.com/intern/diff/D18493938/)"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/560162414","html_url":"https://github.com/pytorch/pytorch/pull/29785#issuecomment-560162414","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/29785","id":560162414,"node_id":"MDEyOklzc3VlQ29tbWVudDU2MDE2MjQxNA==","user":{"login":"jjlilley","id":55768316,"node_id":"MDQ6VXNlcjU1NzY4MzE2","avatar_url":"https://avatars0.githubusercontent.com/u/55768316?v=4","gravatar_id":"","url":"https://api.github.com/users/jjlilley","html_url":"https://github.com/jjlilley","followers_url":"https://api.github.com/users/jjlilley/followers","following_url":"https://api.github.com/users/jjlilley/following{/other_user}","gists_url":"https://api.github.com/users/jjlilley/gists{/gist_id}","starred_url":"https://api.github.com/users/jjlilley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jjlilley/subscriptions","organizations_url":"https://api.github.com/users/jjlilley/orgs","repos_url":"https://api.github.com/users/jjlilley/repos","events_url":"https://api.github.com/users/jjlilley/events{/privacy}","received_events_url":"https://api.github.com/users/jjlilley/received_events","type":"User","site_admin":false},"created_at":"2019-12-01T21:36:10Z","updated_at":"2019-12-01T21:36:10Z","author_association":"CONTRIBUTOR","body":"@suo That's sensible to do, since I don't see how this is related.\r\n I just wrote up a rollback for the (active) process_group_agent.{cpp,h} part at:\r\n    https://github.com/pytorch/pytorch/pull/30593"}}
{"actor":{"display_login":"jjlilley"}}