{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/9486","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/9486/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/9486/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/9486/events","html_url":"https://github.com/pytorch/pytorch/issues/9486","id":341790745,"node_id":"MDU6SXNzdWUzNDE3OTA3NDU=","number":9486,"title":"Benchmarking of Cudnn convolutions is bugged when Tensor Cores are used.","user":{"login":"rotabulo","id":28045295,"node_id":"MDQ6VXNlcjI4MDQ1Mjk1","avatar_url":"https://avatars3.githubusercontent.com/u/28045295?v=4","gravatar_id":"","url":"https://api.github.com/users/rotabulo","html_url":"https://github.com/rotabulo","followers_url":"https://api.github.com/users/rotabulo/followers","following_url":"https://api.github.com/users/rotabulo/following{/other_user}","gists_url":"https://api.github.com/users/rotabulo/gists{/gist_id}","starred_url":"https://api.github.com/users/rotabulo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rotabulo/subscriptions","organizations_url":"https://api.github.com/users/rotabulo/orgs","repos_url":"https://api.github.com/users/rotabulo/repos","events_url":"https://api.github.com/users/rotabulo/events{/privacy}","received_events_url":"https://api.github.com/users/rotabulo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-07-17T07:02:52Z","updated_at":"2018-08-17T20:25:44Z","closed_at":null,"author_association":"NONE","body":"## Issue description\r\n\r\nThe CUDNN function `cudnnFindConvolutionForwardAlgorithmEx` (used in https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn/Conv.cpp#L481) searches an optimal algorithm satisfying the provided workspace constraint by varying the algorithm type but also by changing the math type. The current code fixes the math type to `CUDNN_TENSOR_OP_MATH` if fp16 is used (see https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/cudnn/Descriptors.h#L199) irrespective of the math type of the optimal algorithm returned by `cudnnFindConvolutionForwardAlgorithmEx`. The worst consequence of this oversight is that the workspace that gets allocated under the wrong math type assumption violates the actual memory constraint estimated with `getMaxWorkspaceSize`, resulting in a `CUDA Out Of Memory`.\r\n\r\nA possible solution to this issue could be to cache the `cudnnConvolutionFwdAlgoPerf_t` structures rather than `cudnnConvolutionFwdAlgo_t`. In this way the information about the correct `math type` is available and can be properly set before computing the actual workspace and before running the convolution.\r\n\r\nThe issue does not affect only the forward pass, but also eventually `BwdData` and `BwdFilter`."},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/413979119","html_url":"https://github.com/pytorch/pytorch/issues/9486#issuecomment-413979119","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/9486","id":413979119,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMzk3OTExOQ==","user":{"login":"ezyang","id":13564,"node_id":"MDQ6VXNlcjEzNTY0","avatar_url":"https://avatars0.githubusercontent.com/u/13564?v=4","gravatar_id":"","url":"https://api.github.com/users/ezyang","html_url":"https://github.com/ezyang","followers_url":"https://api.github.com/users/ezyang/followers","following_url":"https://api.github.com/users/ezyang/following{/other_user}","gists_url":"https://api.github.com/users/ezyang/gists{/gist_id}","starred_url":"https://api.github.com/users/ezyang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ezyang/subscriptions","organizations_url":"https://api.github.com/users/ezyang/orgs","repos_url":"https://api.github.com/users/ezyang/repos","events_url":"https://api.github.com/users/ezyang/events{/privacy}","received_events_url":"https://api.github.com/users/ezyang/received_events","type":"User","site_admin":false},"created_at":"2018-08-17T20:25:44Z","updated_at":"2018-08-17T20:25:44Z","author_association":"CONTRIBUTOR","body":"Thanks @rotabulo. Do you have a repro, by any chance?"}}
{"actor":{"display_login":"ezyang"}}