{"action":"created","issue":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/765","repository_url":"https://api.github.com/repos/Lasagne/Lasagne","labels_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/765/labels{/name}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/765/comments","events_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/765/events","html_url":"https://github.com/Lasagne/Lasagne/issues/765","id":187561671,"number":765,"title":"'NaN'  when using nesterov momentum and high learning rates","user":{"login":"andrelopes1705","id":14953758,"avatar_url":"https://avatars.githubusercontent.com/u/14953758?v=3","gravatar_id":"","url":"https://api.github.com/users/andrelopes1705","html_url":"https://github.com/andrelopes1705","followers_url":"https://api.github.com/users/andrelopes1705/followers","following_url":"https://api.github.com/users/andrelopes1705/following{/other_user}","gists_url":"https://api.github.com/users/andrelopes1705/gists{/gist_id}","starred_url":"https://api.github.com/users/andrelopes1705/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrelopes1705/subscriptions","organizations_url":"https://api.github.com/users/andrelopes1705/orgs","repos_url":"https://api.github.com/users/andrelopes1705/repos","events_url":"https://api.github.com/users/andrelopes1705/events{/privacy}","received_events_url":"https://api.github.com/users/andrelopes1705/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-06T13:31:11Z","updated_at":"2016-11-06T13:49:15Z","closed_at":null,"body":"**Introduction**\r\nOn August 21, I reported a bug on the lasagne group forum about this.\r\nBasically, when you are using nesterov momentum to train conv nets, if your learning rate starts too high, \"nans\" are generated. In my case, they start happening near epoch 9.\r\n\r\nThis happens at the most new update of theano and lasagne.\r\n\r\nRight now, im using a current version of lasagne that is old but doesnt have that issue : \r\n\"pip install --upgrade --no-deps git+git://github.com/Lasagne/Lasagne.git@5a009f98cb479c6c39157027de830a83462dabc0\"\r\n\r\n\r\nJan.schluete asked me to bisect the code with git to find the guilty commit.\r\nI did it twice to confirm and found it.\r\n\r\n_4d4e0b0796634c23ad43889685ee4b428fe30f8a is the first bad commit\r\ncommit 4d4e0b0796634c23ad43889685ee4b428fe30f8a\r\nAuthor: Jan Schlüter <jan.schlueter@    .at>\r\nDate:   Thu Jun 30 13:30:29 2016 +0200\r\n\r\n    Have create_param() set the broadcast pattern of created shared variables\r\n\r\n:040000 040000 93ce1ab99b4d4bd14d3825a0b143109aa7d234b2 020a514a738f4fd6d29f538fa2d0fdc5af76555b M      lasagne_\r\n\r\n\r\nThe model im using and the learning rate im using are below :  \r\n\r\n```\r\ndef getModel():\r\n    try:\r\n\r\n        input_var = tensor.tensor4('inputs')\r\n        target_var = tensor.fmatrix('targets')\r\n\r\n        # Input Layer\r\n        network = lasagne.layers.InputLayer(shape=(None, 1, 7, 7), input_var=input_var)\r\n\r\n        # First Convolution Layer\r\n        network = lasagne.layers.Conv2DLayer(network, num_filters=60, filter_size=(3, 3), stride=1, pad=3)\r\n        network = lasagne.layers.Conv2DLayer(network, num_filters=60, filter_size=(3, 3), stride=1, pad=2)\r\n        network = lasagne.layers.MaxPool2DLayer(incoming=network, pool_size=(2, 2))\r\n\r\n        # Second Convolution Layer\r\n        network = lasagne.layers.Conv2DLayer(network, num_filters=120, filter_size=(3, 3), stride=1, pad=3)\r\n        network = lasagne.layers.Conv2DLayer(network, num_filters=120, filter_size=(3, 3), stride=1, pad=2)\r\n        network = lasagne.layers.MaxPool2DLayer(incoming=network, pool_size=(2, 2))\r\n\r\n        # Hidden Layers\r\n        network = lasagne.layers.DenseLayer(lasagne.layers.dropout(network, p=0.5), num_units=96)\r\n        network = lasagne.layers.DenseLayer(lasagne.layers.dropout(network, p=0.5), num_units=32)\r\n        network = lasagne.layers.DenseLayer(lasagne.layers.dropout(network, p=0.5), num_units=1, nonlinearity=lasagne.nonlinearities.sigmoid)\r\n\r\n        return network, input_var, target_var\r\n\r\n    except Exception as inst:\r\n        print (\"Failure to Build NN !\", inst.message, type(inst), (inst.args), (inst))\r\n        sys.exit(1)\r\n\r\ndef getLearningRate(current_epoch=0, max_epochs=1000):\r\n    if current_epoch < (max_epochs * 5) / 100:\r\n        return float32(0.150)\r\n\r\n    if current_epoch < (max_epochs * 10) / 100:\r\n        return float32(0.10)\r\n\r\n    if current_epoch < (max_epochs * 30) / 100:\r\n        return float32(0.010)\r\n\r\n    if current_epoch < (max_epochs * 80) / 100:\r\n        return float32(0.0010)\r\n\r\n    if current_epoch <= (max_epochs * 100) / 100:\r\n        return float32(0.00010)\r\n```\r\n\r\nMomentum is kept at default.\r\n\r\n\r\n\r\n\r\n\r\n> If you suspect you have found a bug**, please first try [updating to the bleeding-edge versions of Theano and Lasagne](http://lasagne.readthedocs.io/en/latest/user/installation.html#bleeding-edge-version). It may have been fixed already.\r\n\r\n**Same issue!**\r\n\r\n\r\n>   If you are not sure whether the problem lies within your code, Theano, or Lasagne, first post on [our mailing list](https://groups.google.com/forum/#!forum/lasagne-users).\r\n\r\n**The problem is in the commit informed above.**\r\n\r\n\r\nThanks! I really need this bug fixed since my thesis is using lasagne and referencing it a lot! \r\n\r\nPlease help!\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/comments/258682031","html_url":"https://github.com/Lasagne/Lasagne/issues/765#issuecomment-258682031","issue_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/765","id":258682031,"user":{"login":"ebenolson","id":544718,"avatar_url":"https://avatars.githubusercontent.com/u/544718?v=3","gravatar_id":"","url":"https://api.github.com/users/ebenolson","html_url":"https://github.com/ebenolson","followers_url":"https://api.github.com/users/ebenolson/followers","following_url":"https://api.github.com/users/ebenolson/following{/other_user}","gists_url":"https://api.github.com/users/ebenolson/gists{/gist_id}","starred_url":"https://api.github.com/users/ebenolson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ebenolson/subscriptions","organizations_url":"https://api.github.com/users/ebenolson/orgs","repos_url":"https://api.github.com/users/ebenolson/repos","events_url":"https://api.github.com/users/ebenolson/events{/privacy}","received_events_url":"https://api.github.com/users/ebenolson/received_events","type":"User","site_admin":false},"created_at":"2016-11-06T13:49:15Z","updated_at":"2016-11-06T13:49:15Z","body":"Hi, good work tracking it down to a commit. My first assumption would have been that your learning rate was just too high, but since it behaved differently in the past we should figure out what's going on. Assuming the bug is not obvious (it's not to me), there are some things you can do to help this get fixed quicker:\r\n\r\n1. Create a fully contained test script that reproduces the error - use randomly generated or constant input data if possible. \r\n\r\n2. Simply the test case as much as possible - is the error dependent on this network structure or can you reduce it to a single layer? Can you replace `getLearningRate` with a constant?"}}
{"actor":{"display_login":"ebenolson"}}