{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/6809","repository_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/6809/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/6809/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/6809/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/6809","id":388082602,"node_id":"MDU6SXNzdWUzODgwODI2MDI=","number":6809,"title":"Setting learningRate dynamicly produces NaN","user":{"login":"liweigu","id":31612697,"node_id":"MDQ6VXNlcjMxNjEyNjk3","avatar_url":"https://avatars3.githubusercontent.com/u/31612697?v=4","gravatar_id":"","url":"https://api.github.com/users/liweigu","html_url":"https://github.com/liweigu","followers_url":"https://api.github.com/users/liweigu/followers","following_url":"https://api.github.com/users/liweigu/following{/other_user}","gists_url":"https://api.github.com/users/liweigu/gists{/gist_id}","starred_url":"https://api.github.com/users/liweigu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liweigu/subscriptions","organizations_url":"https://api.github.com/users/liweigu/orgs","repos_url":"https://api.github.com/users/liweigu/repos","events_url":"https://api.github.com/users/liweigu/events{/privacy}","received_events_url":"https://api.github.com/users/liweigu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-12-06T07:03:58Z","updated_at":"2018-12-12T06:42:46Z","closed_at":null,"author_association":"NONE","body":"DL4J version: 1.0.0-SNAPSHOT\r\n\r\n * In order to implement a GAN model, i should set generator's or discriminator's learningRate to 0 alternately.\r\n * There might be a bug to do it that way, and this is a simple program to test it.\r\n * **When setting 'train1stTime = true', the '2nd predictedValue' is NaN, which should be a valid double value.**\r\n * -- Setting 'train1stTime = true' means to do training first, and to set some layers' learningRate to 0, then to do training again.\r\n * When setting 'train1stTime = false', the '2nd predictedValue' is a valid double value, which is expected.\r\n * -- Setting 'train1stTime = false' means to set some layers' learningRate to 0 first, and to do training.\r\n\r\nThe code:\r\n\r\npackage com.liweigu.dl.study.gan;\r\n\r\nimport org.deeplearning4j.nn.api.Layer;\r\nimport org.deeplearning4j.nn.api.OptimizationAlgorithm;\r\nimport org.deeplearning4j.nn.conf.BackpropType;\r\nimport org.deeplearning4j.nn.conf.ComputationGraphConfiguration.GraphBuilder;\r\nimport org.deeplearning4j.nn.conf.NeuralNetConfiguration;\r\nimport org.deeplearning4j.nn.conf.layers.DenseLayer;\r\nimport org.deeplearning4j.nn.conf.layers.OutputLayer;\r\nimport org.deeplearning4j.nn.graph.ComputationGraph;\r\nimport org.deeplearning4j.nn.layers.BaseLayer;\r\nimport org.deeplearning4j.nn.weights.WeightInit;\r\nimport org.nd4j.linalg.activations.Activation;\r\nimport org.nd4j.linalg.api.ndarray.INDArray;\r\nimport org.nd4j.linalg.dataset.DataSet;\r\nimport org.nd4j.linalg.factory.Nd4j;\r\nimport org.nd4j.linalg.learning.config.Adam;\r\nimport org.nd4j.linalg.lossfunctions.LossFunctions.LossFunction;\r\n\r\n/**\r\n * Set learningRate dynamicly for testing.\r\n * \r\n * In order to implement a GAN model, i should set generator's or discriminator's learningRate to 0 alternately.\r\n * There might be a bug to do it that way, and this is a simple program to test it.\r\n * When setting 'train1stTime = true', the '2nd predictedValue' is NaN, which should be a valid double value.\r\n * -- Setting 'train1stTime = true' means to do training first, and to set some layers' learningRate to 0, then to do training again.\r\n * When setting 'train1stTime = false', the '2nd predictedValue' is a valid double value, which is expected.\r\n * -- Setting 'train1stTime = false' means to set some layers' learningRate to 0 first, and to do training.\r\n * \r\n * @author liweigu714@163.com\r\n *\r\n */\r\npublic class DynamicLR {\r\n\tpublic static double LearningRate = 2e-4;\r\n\r\n\tpublic static void main(String[] args) {\r\n\t\trun();\r\n\t}\r\n\r\n\tpublic static void run() {\r\n\t\tSystem.out.println(\"run DynamicLR\");\r\n\r\n\t\t// Notice:\r\n\t\t// if set train1stTime = true, the 2nd predictedValue is NaN.\r\n\t\t// if set train1stTime = false, the 2nd predictedValue is a valid double value.\r\n\t\tboolean train1stTime = true;\r\n\r\n\t\tComputationGraph multiLayerNetwork = getNetwork();\r\n\r\n\t\tDataSet trainDataSet;\r\n\t\tDataSet testDataSet;\r\n\t\tINDArray[] outputs;\r\n\t\tINDArray output;\r\n\t\tdouble predictedValue;\r\n\r\n\t\t// train for 1st time\r\n\t\tif (train1stTime) {\r\n\t\t\ttrainDataSet = getRandomDataSet();\r\n\t\t\tmultiLayerNetwork.fit(trainDataSet);\r\n\t\t\t// print current output\r\n\t\t\ttestDataSet = getRandomDataSet();\r\n\t\t\toutputs = multiLayerNetwork.output(testDataSet.getFeatures());\r\n\t\t\toutput = outputs[0];\r\n\t\t\tpredictedValue = output.getDouble(0);\r\n\t\t\tSystem.out.println(\"1st predictedValue = \" + predictedValue);\r\n\t\t}\r\n\r\n\t\t// freeze some layer(s)\r\n\t\tString freezeLayerType = \"g\";\r\n\t\tfreeze(multiLayerNetwork, freezeLayerType);\r\n\r\n\t\t// train for 2nd time\r\n\t\ttrainDataSet = getRandomDataSet();\r\n\t\tmultiLayerNetwork.fit(trainDataSet);\r\n\t\t// print current output\r\n\t\ttestDataSet = getRandomDataSet();\r\n\t\toutputs = multiLayerNetwork.output(testDataSet.getFeatures());\r\n\t\toutput = outputs[0];\r\n\t\tpredictedValue = output.getDouble(0);\r\n\t\tSystem.out.println(\"2nd predictedValue = \" + predictedValue);\r\n\t}\r\n\r\n\t// generate random DataSet\r\n\tprivate static DataSet getRandomDataSet() {\r\n\t\tINDArray features = Nd4j.randn(new long[]{1, 100});\r\n\t\t// set label to one, for testing.\r\n\t\tINDArray labels=Nd4j.ones(new long[]{1, 1}) ;\r\n\t\tDataSet dataSet = new DataSet(features, labels);\r\n\t\treturn dataSet;\r\n\t}\r\n\r\n\t/**\r\n\t * Set some layers' learningRate to 0 to 'freeze' it, which means don't change the parameters on the layer durning training.\r\n\t * @param multiLayerNetwork multiLayerNetwork\r\n\t * @param freezeLayerType \"g\" means freeze layers like \"g-*\", \"d\" means freeze layers like \"d-*\".\r\n\t */\r\n\tprivate static void freeze(ComputationGraph multiLayerNetwork, String freezeLayerType) {\r\n\t\tLayer[] layers = multiLayerNetwork.getLayers();\r\n\t\tfor (Layer layer : layers) {\r\n\t\t\tif (layer instanceof BaseLayer) {\r\n\t\t\t\tBaseLayer baseLayer = (BaseLayer) layer;\r\n\t\t\t\tString layerName = baseLayer.getConf().getLayer().getLayerName();\r\n\t\t\t\tif (freezeLayerType.equals(\"g\")) {\r\n\t\t\t\t\tif (layerName.contains(\"g-\")) {\r\n\t\t\t\t\t\tmultiLayerNetwork.setLearningRate(layerName, 0);\r\n\t\t\t\t\t} else if (layerName.contains(\"d-\")) {\r\n\t\t\t\t\t\tmultiLayerNetwork.setLearningRate(layerName, LearningRate);\r\n\t\t\t\t\t}\r\n\t\t\t\t} else if (freezeLayerType.equals(\"d\")) {\r\n\t\t\t\t\tif (layerName.contains(\"g-\")) {\r\n\t\t\t\t\t\tSystem.out.println(layerName + \" = \" + LearningRate);\r\n\t\t\t\t\t\tmultiLayerNetwork.setLearningRate(layerName, LearningRate);\r\n\t\t\t\t\t} else if (layerName.contains(\"d-\")) {\r\n\t\t\t\t\t\tSystem.out.println(layerName + \" = \" + 0);\r\n\t\t\t\t\t\tmultiLayerNetwork.setLearningRate(layerName, 0);\r\n\t\t\t\t\t}\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t}\r\n\t}\r\n\r\n\tprivate static ComputationGraph getNetwork() {\r\n\t\tComputationGraph multiLayerNetwork;\r\n\r\n\t\tNeuralNetConfiguration.Builder builder = new NeuralNetConfiguration.Builder();\r\n\t\tbuilder.seed(140);\r\n\t\tbuilder.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT); \r\n\t\tbuilder.weightInit(WeightInit.XAVIER);\r\n\t\tbuilder.updater(new Adam(LearningRate));\r\n\r\n\t\tGraphBuilder graphBuilder = builder.graphBuilder()\r\n\t\t\t\t.pretrain(false).backprop(true)\r\n\t\t\t\t.backpropType(BackpropType.Standard)\r\n\t\t\t\t.addInputs(\"input\")\r\n\t\t\t\t.setOutputs(\"f-output\");\r\n\r\n\t\t// g-* for generator\r\n\t\tgraphBuilder = graphBuilder.addLayer(\"g-dense\",\r\n        \t\tnew DenseLayer.Builder()\r\n\t\t\t\t.nIn(100)\r\n\t\t\t\t.nOut(10)\r\n\t\t\t\t.updater(new Adam(LearningRate))\r\n\t\t\t\t.weightInit(WeightInit.RELU)\r\n\t\t\t\t.activation(Activation.LEAKYRELU)\r\n\t\t\t\t.build(),\r\n\t\t\t\t\"input\");\r\n\t\t// d-* for discriminator\r\n\t\tgraphBuilder = graphBuilder.addLayer(\"d-dense\",\r\n        \t\tnew DenseLayer.Builder()\r\n\t\t\t\t.nIn(10)\r\n\t\t\t\t.nOut(10)\r\n\t\t\t\t.updater(new Adam(LearningRate))\r\n\t\t\t\t.weightInit(WeightInit.RELU)\r\n\t\t\t\t.activation(Activation.LEAKYRELU).build(),\r\n\t\t\t\t\"g-dense\");\r\n\t\t// final output\r\n\t\tgraphBuilder = graphBuilder.addLayer(\"f-output\",\r\n\t\t\t\tnew OutputLayer.Builder(LossFunction.XENT)\r\n\t\t\t\t.nIn(10)\r\n\t\t\t\t.nOut(1)\r\n\t\t\t\t.updater(new Adam(LearningRate))\r\n\t\t\t\t.weightInit(WeightInit.XAVIER)\r\n\t\t\t\t.activation(Activation.SIGMOID)\r\n\t\t\t\t.build(),\r\n\t\t\t\t\"d-dense\");\r\n\r\n\t\tmultiLayerNetwork = new ComputationGraph(graphBuilder.build());\r\n\t\tmultiLayerNetwork.init();\r\n\t\tSystem.out.println(multiLayerNetwork.summary());\r\n\r\n\t\treturn multiLayerNetwork;\r\n\t}\r\n\r\n}\r\n"},"comment":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/comments/446480052","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/6809#issuecomment-446480052","issue_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/6809","id":446480052,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjQ4MDA1Mg==","user":{"login":"liweigu","id":31612697,"node_id":"MDQ6VXNlcjMxNjEyNjk3","avatar_url":"https://avatars3.githubusercontent.com/u/31612697?v=4","gravatar_id":"","url":"https://api.github.com/users/liweigu","html_url":"https://github.com/liweigu","followers_url":"https://api.github.com/users/liweigu/followers","following_url":"https://api.github.com/users/liweigu/following{/other_user}","gists_url":"https://api.github.com/users/liweigu/gists{/gist_id}","starred_url":"https://api.github.com/users/liweigu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liweigu/subscriptions","organizations_url":"https://api.github.com/users/liweigu/orgs","repos_url":"https://api.github.com/users/liweigu/repos","events_url":"https://api.github.com/users/liweigu/events{/privacy}","received_events_url":"https://api.github.com/users/liweigu/received_events","type":"User","site_admin":false},"created_at":"2018-12-12T06:42:46Z","updated_at":"2018-12-12T06:42:46Z","author_association":"NONE","body":"Another way to produce NaN is to add a BN using RenormalizeL2PerParamType after layer \"g-dense\", like this:\r\n\t\tgraphBuilder = graphBuilder.addLayer(\"g-dense\",\r\n        \t\tnew DenseLayer.Builder()\r\n\t\t\t\t.nIn(100)\r\n\t\t\t\t.nOut(10)\r\n\t\t\t\t.updater(new Adam(LearningRate))\r\n\t\t\t\t.weightInit(WeightInit.RELU)\r\n\t\t\t\t.activation(Activation.LEAKYRELU)\r\n\t\t\t\t.build(),\r\n\t\t\t\t\"input\");\r\n\t\tgraphBuilder = graphBuilder.addLayer(\"g-bn1\",\r\n        \t\tnew BatchNormalization.Builder()\r\n        \t\t**.gradientNormalization(GradientNormalization.RenormalizeL2PerParamType)**\r\n\t\t\t\t.nIn(10)\r\n\t\t\t\t.nOut(10)\r\n\t\t\t\t.updater(new RmsProp(LearningRate))\r\n\t\t\t\t.weightInit(WeightInit.RELU)\r\n\t\t\t\t.activation(Activation.LEAKYRELU)\r\n\t\t\t\t.build(),\r\n\t\t\t\t\"g-dense\");\r\nNotice that using **RenormalizeL2PerLayer** instead of RenormalizeL2PerParamType doesn't produce NaN.\r\n\r\nThese 2 ways producing NaN are alike, please take a look."}}
{"actor":{"display_login":"liweigu"}}