{"action":"created","issue":{"url":"https://api.github.com/repos/Theano/Theano/issues/3910","labels_url":"https://api.github.com/repos/Theano/Theano/issues/3910/labels{/name}","comments_url":"https://api.github.com/repos/Theano/Theano/issues/3910/comments","events_url":"https://api.github.com/repos/Theano/Theano/issues/3910/events","html_url":"https://github.com/Theano/Theano/issues/3910","id":128018871,"number":3910,"title":"Convolutions don't work with NanGuardMode","user":{"login":"matthias-k","id":4093707,"avatar_url":"https://avatars.githubusercontent.com/u/4093707?v=3","gravatar_id":"","url":"https://api.github.com/users/matthias-k","html_url":"https://github.com/matthias-k","followers_url":"https://api.github.com/users/matthias-k/followers","following_url":"https://api.github.com/users/matthias-k/following{/other_user}","gists_url":"https://api.github.com/users/matthias-k/gists{/gist_id}","starred_url":"https://api.github.com/users/matthias-k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthias-k/subscriptions","organizations_url":"https://api.github.com/users/matthias-k/orgs","repos_url":"https://api.github.com/users/matthias-k/repos","events_url":"https://api.github.com/users/matthias-k/events{/privacy}","received_events_url":"https://api.github.com/users/matthias-k/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-01-21T21:02:55Z","updated_at":"2016-01-21T21:27:39Z","closed_at":null,"body":"While trying to debug some NaNs in my function, I came across the problem that I can't use NanGuardMode on a function involving convolutions:\r\n\r\n```python\r\nimport numpy as np\r\nimport theano\r\nimport theano.tensor as T\r\nfrom theano.compile.nanguardmode import NanGuardMode\r\n\r\ninput = T.tensor4('input')\r\nW = theano.shared(np.random.randn(10,3,10,10))\r\nconv = T.nnet.conv2d(input, W)\r\nloss = conv.sum()\r\nf1 = theano.function([input], loss)\r\nf2 = theano.function([input], loss,\r\n                    mode=NanGuardMode())\r\n```\r\nWhile f1 runs fine, f2 raises a `NotImplementedError`:\r\n\r\n```python\r\nNotImplementedError                       Traceback (most recent call last)\r\n/usr/local/lib/python3.4/dist-packages/theano/gof/link.py in f()\r\n   1011                 try:\r\n-> 1012                     wrapper(i, node, *thunks)\r\n   1013                 except Exception:\r\n\r\n/usr/local/lib/python3.4/dist-packages/theano/compile/nanguardmode.py in nan_check(i, node, fn)\r\n    322                     do_check_on(x[0], node, fn, True)\r\n--> 323             fn()\r\n    324             outputs = fn.outputs\r\n\r\n/usr/local/lib/python3.4/dist-packages/theano/gof/op.py in rval(p, i, o, n)\r\n    882             def rval(p=p, i=node_input_storage, o=node_output_storage, n=node):\r\n--> 883                 r = p(n, [x[0] for x in i], o)\r\n    884                 for o in node.outputs:\r\n\r\n/usr/local/lib/python3.4/dist-packages/theano/tensor/nnet/abstract_conv.py in perform(self, node, inp, out_)\r\n    306         raise NotImplementedError(\r\n--> 307             'AbstractConv2d theano optimization failed. '\r\n    308             'Did you exclude both \"conv_dnn\" and \"conv_gemm\" from '\r\n\r\nNotImplementedError: AbstractConv2d theano optimization failed. Did you exclude both \"conv_dnn\" and \"conv_gemm\" from the optimizer? Is cudnn available and does the GPU support it?\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nNotImplementedError                       Traceback (most recent call last)\r\n<ipython-input-20-0f706dd28e6e> in <module>()\r\n----> 1 f2(input_data)\r\n\r\n/usr/local/lib/python3.4/dist-packages/theano/compile/function_module.py in __call__(self, *args, **kwargs)\r\n    857         t0_fn = time.time()\r\n    858         try:\r\n--> 859             outputs = self.fn()\r\n    860         except Exception:\r\n    861             if hasattr(self.fn, 'position_of_error'):\r\n\r\n/usr/local/lib/python3.4/dist-packages/theano/gof/link.py in f()\r\n   1012                     wrapper(i, node, *thunks)\r\n   1013                 except Exception:\r\n-> 1014                     raise_with_op(node, *thunks)\r\n   1015         f.thunk_groups = thunk_groups\r\n   1016 \r\n\r\n/usr/local/lib/python3.4/dist-packages/theano/gof/link.py in raise_with_op(node, thunk, exc_info, storage_map)\r\n    312         # extra long error message in that case.\r\n    313         pass\r\n--> 314     reraise(exc_type, exc_value, exc_trace)\r\n    315 \r\n    316 \r\n\r\n/usr/local/lib/python3.4/dist-packages/six.py in reraise(tp, value, tb)\r\n    683             value = tp()\r\n    684         if value.__traceback__ is not tb:\r\n--> 685             raise value.with_traceback(tb)\r\n    686         raise value\r\n    687 \r\n\r\n/usr/local/lib/python3.4/dist-packages/theano/gof/link.py in f()\r\n   1010             for i, (thunks, node) in enumerate(izip(thunk_groups, order)):\r\n   1011                 try:\r\n-> 1012                     wrapper(i, node, *thunks)\r\n   1013                 except Exception:\r\n   1014                     raise_with_op(node, *thunks)\r\n\r\n/usr/local/lib/python3.4/dist-packages/theano/compile/nanguardmode.py in nan_check(i, node, fn)\r\n    321                 if var.owner is not None:\r\n    322                     do_check_on(x[0], node, fn, True)\r\n--> 323             fn()\r\n    324             outputs = fn.outputs\r\n    325             for x in outputs:\r\n\r\n/usr/local/lib/python3.4/dist-packages/theano/gof/op.py in rval(p, i, o, n)\r\n    881             # default arguments are stored in the closure of `rval`\r\n    882             def rval(p=p, i=node_input_storage, o=node_output_storage, n=node):\r\n--> 883                 r = p(n, [x[0] for x in i], o)\r\n    884                 for o in node.outputs:\r\n    885                     compute_map[o][0] = True\r\n\r\n/usr/local/lib/python3.4/dist-packages/theano/tensor/nnet/abstract_conv.py in perform(self, node, inp, out_)\r\n    305     def perform(self, node, inp, out_):\r\n    306         raise NotImplementedError(\r\n--> 307             'AbstractConv2d theano optimization failed. '\r\n    308             'Did you exclude both \"conv_dnn\" and \"conv_gemm\" from '\r\n    309             'the optimizer? Is cudnn available and does the GPU support it?')\r\n\r\nNotImplementedError: AbstractConv2d theano optimization failed. Did you exclude both \"conv_dnn\" and \"conv_gemm\" from the optimizer? Is cudnn available and does the GPU support it?\r\nApply node that caused the error: AbstractConv2d{border_mode='valid', subsample=(1, 1), filter_flip=True, imshp=None, kshp=None}(input, <TensorType(float64, 4D)>)\r\nToposort index: 0\r\nInputs types: [TensorType(float64, 4D), TensorType(float64, 4D)]\r\nInputs shapes: [(1, 3, 100, 100), (10, 3, 10, 10)]\r\nInputs strides: [(9223372036854775807, 80000, 800, 8), (2400, 800, 80, 8)]\r\nInputs values: ['not shown', 'not shown']\r\nOutputs clients: [[Sum{acc_dtype=float64}(AbstractConv2d{border_mode='valid', subsample=(1, 1), filter_flip=True, imshp=None, kshp=None}.0)]]\r\n\r\nBacktrace when the node is created:\r\n  File \"<ipython-input-15-d9d7dcba8686>\", line 1, in <module>\r\n    conv = T.nnet.conv2d(input, W)\r\n\r\nHINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.\r\n```\r\n\r\nThe Exception occurs on a normal theano installation without any additional configuration (especially, I did not activate the GPU)."},"comment":{"url":"https://api.github.com/repos/Theano/Theano/issues/comments/173712799","html_url":"https://github.com/Theano/Theano/issues/3910#issuecomment-173712799","issue_url":"https://api.github.com/repos/Theano/Theano/issues/3910","id":173712799,"user":{"login":"matthias-k","id":4093707,"avatar_url":"https://avatars.githubusercontent.com/u/4093707?v=3","gravatar_id":"","url":"https://api.github.com/users/matthias-k","html_url":"https://github.com/matthias-k","followers_url":"https://api.github.com/users/matthias-k/followers","following_url":"https://api.github.com/users/matthias-k/following{/other_user}","gists_url":"https://api.github.com/users/matthias-k/gists{/gist_id}","starred_url":"https://api.github.com/users/matthias-k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthias-k/subscriptions","organizations_url":"https://api.github.com/users/matthias-k/orgs","repos_url":"https://api.github.com/users/matthias-k/repos","events_url":"https://api.github.com/users/matthias-k/events{/privacy}","received_events_url":"https://api.github.com/users/matthias-k/received_events","type":"User","site_admin":false},"created_at":"2016-01-21T21:27:39Z","updated_at":"2016-01-21T21:27:39Z","body":"I tested again with the latest master (fdfbab37146ee475b3fd17d8d104fb09bf3a8d5c). At least it is only one exception now:\r\n\r\n```python\r\nTraceback (most recent call last):\r\n  File \"test_nanguardmode.py\", line 17, in <module>\r\n    print(f2(input_data))\r\n  File \"/gpfs01/bethge/home/mkuemmerer/envs/theano_bug/local/lib/python2.7/site-packages/Theano-0.8.0.dev0-py2.7.egg/theano/compile/function_module.py\", line 859, in __call__\r\n    outputs = self.fn()\r\n  File \"/gpfs01/bethge/home/mkuemmerer/envs/theano_bug/local/lib/python2.7/site-packages/Theano-0.8.0.dev0-py2.7.egg/theano/gof/link.py\", line 1014, in f\r\n    raise_with_op(node, *thunks)\r\n  File \"/gpfs01/bethge/home/mkuemmerer/envs/theano_bug/local/lib/python2.7/site-packages/Theano-0.8.0.dev0-py2.7.egg/theano/gof/link.py\", line 314, in raise_with_op\r\n    reraise(exc_type, exc_value, exc_trace)\r\n  File \"/gpfs01/bethge/home/mkuemmerer/envs/theano_bug/local/lib/python2.7/site-packages/Theano-0.8.0.dev0-py2.7.egg/theano/gof/link.py\", line 1012, in f\r\n    wrapper(i, node, *thunks)\r\n  File \"/gpfs01/bethge/home/mkuemmerer/envs/theano_bug/local/lib/python2.7/site-packages/Theano-0.8.0.dev0-py2.7.egg/theano/compile/nanguardmode.py\", line 323, in nan_check\r\n    fn()\r\n  File \"/gpfs01/bethge/home/mkuemmerer/envs/theano_bug/local/lib/python2.7/site-packages/Theano-0.8.0.dev0-py2.7.egg/theano/gof/op.py\", line 908, in rval\r\n    r = p(n, [x[0] for x in i], o)\r\n  File \"/gpfs01/bethge/home/mkuemmerer/envs/theano_bug/local/lib/python2.7/site-packages/Theano-0.8.0.dev0-py2.7.egg/theano/tensor/nnet/abstract_conv.py\", line 307, in perform\r\n    'AbstractConv2d theano optimization failed. '\r\nNotImplementedError: AbstractConv2d theano optimization failed. Did you exclude both \"conv_dnn\" and \"conv_gemm\" from the optimizer? Is cudnn available and does the GPU support it?\r\nApply node that caused the error: AbstractConv2d{border_mode='valid', subsample=(1, 1), filter_flip=True, imshp=None, kshp=None}(input, <TensorType(float64, 4D)>)\r\nToposort index: 0\r\nInputs types: [TensorType(float64, 4D), TensorType(float64, 4D)]\r\nInputs shapes: [(1, 3, 100, 100), (10, 3, 10, 10)]\r\nInputs strides: [(240000, 80000, 800, 8), (2400, 800, 80, 8)]\r\nInputs values: ['not shown', 'not shown']\r\nOutputs clients: [[Sum{acc_dtype=float64}(AbstractConv2d{border_mode='valid', subsample=(1, 1), filter_flip=True, imshp=None, kshp=None}.0)]]\r\n\r\nBacktrace when the node is created:\r\n  File \"test_nanguardmode.py\", line 8, in <module>\r\n    conv = T.nnet.conv2d(input, W)\r\n\r\n```"}}