{"action":"created","issue":{"url":"https://api.github.com/repos/facebookresearch/TensorComprehensions/issues/207","repository_url":"https://api.github.com/repos/facebookresearch/TensorComprehensions","labels_url":"https://api.github.com/repos/facebookresearch/TensorComprehensions/issues/207/labels{/name}","comments_url":"https://api.github.com/repos/facebookresearch/TensorComprehensions/issues/207/comments","events_url":"https://api.github.com/repos/facebookresearch/TensorComprehensions/issues/207/events","html_url":"https://github.com/facebookresearch/TensorComprehensions/issues/207","id":308281314,"number":207,"title":"intermittent error happens in autotuning","user":{"login":"seongwook-ham","id":21188737,"avatar_url":"https://avatars0.githubusercontent.com/u/21188737?v=4","gravatar_id":"","url":"https://api.github.com/users/seongwook-ham","html_url":"https://github.com/seongwook-ham","followers_url":"https://api.github.com/users/seongwook-ham/followers","following_url":"https://api.github.com/users/seongwook-ham/following{/other_user}","gists_url":"https://api.github.com/users/seongwook-ham/gists{/gist_id}","starred_url":"https://api.github.com/users/seongwook-ham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seongwook-ham/subscriptions","organizations_url":"https://api.github.com/users/seongwook-ham/orgs","repos_url":"https://api.github.com/users/seongwook-ham/repos","events_url":"https://api.github.com/users/seongwook-ham/events{/privacy}","received_events_url":"https://api.github.com/users/seongwook-ham/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-24T17:16:36Z","updated_at":"2018-03-25T00:26:54Z","closed_at":null,"author_association":"NONE","body":"import tensor_comprehensions as tc\r\n\r\n\r\nimport math as ma\r\nimport torch\r\nimport torch.nn as nn\r\nimport torch.nn.functional as F\r\nimport random\r\nimport time\r\nimport os\r\n\r\nfrom torch.autograd import Function\r\n\r\nLANG = \"\"\"\r\ndef function(float(N,C,H,W) I, float(C) S, int32(L,A) R, int32(C) R2) -> (output){\r\noutput(n,l,h,w) +=! (R(l,a)==C? float(0):I(n,R(l,a),h,w) * S(R(l,a)))\r\n}\r\ndef function_grad(float(N,C,H,W) I, float(C) S, int32(L,A) R, int32(C) R2, float(N,L,H,W) output_grad) -> (I_grad, S_grad, R_grad, R2_grad){\r\nI_grad(n,c,h,w) = S(c) * output_grad(n,R2(c),h,w)  where c in 0:C\r\nS_grad(kk) +=! I(n,kk,h,w) * output_grad(n,R2(kk),h,w)  where kk in 0:C\r\nR_grad(l,a) = 0 where l in 0:L, a in 0:A\r\nR2_grad(k) = 0 where k in 0:C\r\n}\r\n\"\"\"\r\n\r\n\r\n\r\nbatch=32\r\nwidth=height=32\r\ngenerations=35\r\n\r\nL=92\r\nA=9\r\n\r\nC = 656\r\n\r\n\r\n\r\nfor i in range(20):\r\n    mat1 = torch.ones(batch, C, width, height).cuda()\r\n    mat2 = torch.ones(C).cuda()\r\n    \r\n    \r\n    mat3 = torch.ones(L,A).int().random_(0,C+100).clamp_(0, C)\r\n    \r\n    mat3=mat3.cuda()\r\n    \r\n    \r\n    \r\n    \r\n    mat4 = torch.ones(C).int().random_(0,L).cuda()\r\n    \r\n    \r\n    mat1 = torch.autograd.Variable(mat1, requires_grad =True)\r\n    mat2= torch.autograd.Variable(mat2, requires_grad =True)\r\n    mat3= torch.autograd.Variable(mat3)\r\n    mat4= torch.autograd.Variable(mat4)\r\n    \r\n    autograd = tc.define(LANG, training=True, name=\"function\", backward=\"function_grad\")\r\n    \r\n    autograd.autotune(mat1, mat2, mat3, mat4, options=[tc.Options(\"mlp\"), tc.Options(\"mlp\")], \r\n    cache=\"temp.tc\", pop_size=100, generations=generations, gpus=\"0,1,2,3\", threads=15)\r\n\r\n    del mat1,mat2,mat3,mat4\r\n        \r\n    torch.cuda.empty_cache()\r\n\r\n(indent is not properly expressed in github. see \r\n[code_for_error.txt](https://github.com/facebookresearch/TensorComprehensions/files/1844641/code_for_error.txt)\r\n)\r\n\r\n\r\n\r\nwith code above, when i use autotuner for funcion, inconsisient error happens. but when i call function it acts normally.\r\nfor example autotuner  act normally in forward generation 1-35 and backward generation 1-19, but in backward generation 20, error occurs.\r\nhere is error message\r\n\r\n[error.txt](https://github.com/facebookresearch/TensorComprehensions/files/1844640/error.txt)\r\n\r\n\r\nwhy this intermittent error occurs? and how could i solve it?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/facebookresearch/TensorComprehensions/issues/comments/375935346","html_url":"https://github.com/facebookresearch/TensorComprehensions/issues/207#issuecomment-375935346","issue_url":"https://api.github.com/repos/facebookresearch/TensorComprehensions/issues/207","id":375935346,"user":{"login":"prigoyal","id":13488275,"avatar_url":"https://avatars0.githubusercontent.com/u/13488275?v=4","gravatar_id":"","url":"https://api.github.com/users/prigoyal","html_url":"https://github.com/prigoyal","followers_url":"https://api.github.com/users/prigoyal/followers","following_url":"https://api.github.com/users/prigoyal/following{/other_user}","gists_url":"https://api.github.com/users/prigoyal/gists{/gist_id}","starred_url":"https://api.github.com/users/prigoyal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prigoyal/subscriptions","organizations_url":"https://api.github.com/users/prigoyal/orgs","repos_url":"https://api.github.com/users/prigoyal/repos","events_url":"https://api.github.com/users/prigoyal/events{/privacy}","received_events_url":"https://api.github.com/users/prigoyal/received_events","type":"User","site_admin":false},"created_at":"2018-03-25T00:26:54Z","updated_at":"2018-03-25T00:26:54Z","author_association":"CONTRIBUTOR","body":"Hi @seongwook-ham , have you tried re-running the autotuning? If not, can you re-try and report if you still get the error? thanks"}}
{"actor":{"display_login":"prigoyal"}}