{"action":"created","issue":{"url":"https://api.github.com/repos/Theano/Theano/issues/4823","repository_url":"https://api.github.com/repos/Theano/Theano","labels_url":"https://api.github.com/repos/Theano/Theano/issues/4823/labels{/name}","comments_url":"https://api.github.com/repos/Theano/Theano/issues/4823/comments","events_url":"https://api.github.com/repos/Theano/Theano/issues/4823/events","html_url":"https://github.com/Theano/Theano/issues/4823","id":169253109,"number":4823,"title":"Incompatible on nanguardmode and sandbox.rng_mrg?","user":{"login":"lovecambi","id":7774179,"avatar_url":"https://avatars.githubusercontent.com/u/7774179?v=3","gravatar_id":"","url":"https://api.github.com/users/lovecambi","html_url":"https://github.com/lovecambi","followers_url":"https://api.github.com/users/lovecambi/followers","following_url":"https://api.github.com/users/lovecambi/following{/other_user}","gists_url":"https://api.github.com/users/lovecambi/gists{/gist_id}","starred_url":"https://api.github.com/users/lovecambi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lovecambi/subscriptions","organizations_url":"https://api.github.com/users/lovecambi/orgs","repos_url":"https://api.github.com/users/lovecambi/repos","events_url":"https://api.github.com/users/lovecambi/events{/privacy}","received_events_url":"https://api.github.com/users/lovecambi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-08-03T22:51:49Z","updated_at":"2016-08-04T00:17:22Z","closed_at":null,"body":"I try to debug with nanguardmode to find the where my code induces the NAN. However, I find this mode will always throw an NAN error if your random number generator is theano.sandbox.rng_mrg. Thus I cannot locate where the real NAN is.\r\n\r\nBelow is a simple example to show this issue. f1 and f1_nan are basically the same function except for the later one with nanguardmode, and f1 involves random numbers. f2 has no random numbers, and f2_nan is similar but with nanguardmode.\r\n\r\n```python\r\nimport numpy as np\r\nimport theano\r\nfrom theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams\r\nimport theano.tensor as T\r\nfrom theano.compile.nanguardmode import NanGuardMode\r\n\r\n_srng = RandomStreams(np.random.randint(1, 2147462579))\r\n\r\nmu = T.matrix()\r\nlogvar = T.matrix()\r\neps = _srng.normal(mu.shape)\r\n\r\nres1 = (mu + T.exp(0.5*logvar) * eps).mean()\r\nf1 = theano.function([mu, logvar], res1, allow_input_downcast=True)\r\nf1_nan = theano.function([mu, logvar], res1, allow_input_downcast=True, mode=NanGuardMode(nan_is_error=True, inf_is_error=True, big_is_error=True))\r\n\r\nres2 = (mu + T.exp(0.5*logvar)).mean()\r\nf2 = theano.function([mu, logvar], res2, allow_input_downcast=True)\r\nf2_nan = theano.function([mu, logvar], res2, allow_input_downcast=True, mode=NanGuardMode(nan_is_error=True, inf_is_error=True, big_is_error=True))\r\n\r\nv = np.random.normal(size=(100,64))\r\nm = np.random.normal(size=(100,64))\r\n\r\nf1(m,v)\r\n#output: array(0.02410086616873741, dtype=float32)\r\n\r\nf1_nan(m, v)\r\n#output: AssertionError: NaN detected\r\n#        NanGuardMode found an error in an input of the graph.\r\n\r\nf2(m,v)\r\n#output: array(1.1354453563690186, dtype=float32)\r\n\r\nf2_nan(m, v)\r\n#output: array(1.1354453563690186, dtype=float32)\r\n```\r\n\r\nIn addition, I found the CudaNdarray with NAN always has the shape (92160,) regardless of the input shape, which I guess is the pre-allocated vector in GPU for further computation?"},"comment":{"url":"https://api.github.com/repos/Theano/Theano/issues/comments/237414574","html_url":"https://github.com/Theano/Theano/issues/4823#issuecomment-237414574","issue_url":"https://api.github.com/repos/Theano/Theano/issues/4823","id":237414574,"user":{"login":"nouiz","id":180987,"avatar_url":"https://avatars.githubusercontent.com/u/180987?v=3","gravatar_id":"","url":"https://api.github.com/users/nouiz","html_url":"https://github.com/nouiz","followers_url":"https://api.github.com/users/nouiz/followers","following_url":"https://api.github.com/users/nouiz/following{/other_user}","gists_url":"https://api.github.com/users/nouiz/gists{/gist_id}","starred_url":"https://api.github.com/users/nouiz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nouiz/subscriptions","organizations_url":"https://api.github.com/users/nouiz/orgs","repos_url":"https://api.github.com/users/nouiz/repos","events_url":"https://api.github.com/users/nouiz/events{/privacy}","received_events_url":"https://api.github.com/users/nouiz/received_events","type":"User","site_admin":false},"created_at":"2016-08-04T00:17:22Z","updated_at":"2016-08-04T00:17:22Z","body":"Can you try with Theano dev version? I think it was fixed since the last\nrelease.\n\nLe 3 août 2016 18:51, \"lovecambi\" <notifications@github.com> a écrit :\n\n> I try to debug with nanguardmode to find the where my code induces the\n> NAN. However, I find this mode will always throw an NAN error if your\n> random number generator is theano.sandbox.rng_mrg. Below is a simple\n> example to show this issue. f1 and f1_nan are basically the same function\n> except for the later one with nanguardmode, and f1 involves random numbers.\n> f2 has no random numbers, and f2_nan is similar but with nanguardmode.\n>\n> import numpy as npimport theanofrom theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreamsimport theano.tensor as Tfrom theano.compile.nanguardmode import NanGuardMode\n>\n> _srng = RandomStreams(np.random.randint(1, 2147462579))\n>\n> mu = T.matrix()\n> logvar = T.matrix()\n> eps = _srng.normal(mu.shape)\n>\n> res1 = (mu + T.exp(0.5*logvar) * eps).mean()\n> f1 = theano.function([mu, logvar], res1, allow_input_downcast=True)\n> f1_nan = theano.function([mu, logvar], res1, allow_input_downcast=True, mode=NanGuardMode(nan_is_error=True, inf_is_error=True, big_is_error=True))\n>\n> res2 = (mu + T.exp(0.5*logvar)).mean()\n> f2 = theano.function([mu, logvar], res2, allow_input_downcast=True)\n> f2_nan = theano.function([mu, logvar], res2, allow_input_downcast=True, mode=NanGuardMode(nan_is_error=True, inf_is_error=True, big_is_error=True))\n>\n> v = np.random.normal(size=(100,64))\n> m = np.random.normal(size=(100,64))\n>\n> f1(m,v)#output: array(0.02410086616873741, dtype=float32)\n>\n> f1_nan(m, v)#output: AssertionError: NaN detected#        NanGuardMode found an error in an input of the graph.\n>\n> f2(m,v)#output: array(1.1354453563690186, dtype=float32)\n>\n> f2_nan(m, v)#output: array(1.1354453563690186, dtype=float32)\n>\n> —\n> You are receiving this because you are subscribed to this thread.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/Theano/Theano/issues/4823>, or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AALC-6QZ7_SHfGxRrHSnON1wFCblLJdbks5qcRuHgaJpZM4JcLUx>\n> .\n>\n"}}
{"actor":{"display_login":"nouiz"}}