{"action":"opened","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/3424","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/3424/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/3424/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/3424/events","html_url":"https://github.com/dmlc/mxnet/issues/3424","id":180440252,"number":3424,"title":"Training of Imagenet with inception-bn in python gets good training accuracy but low validation after epoch 1","user":{"login":"hoaphumanoid","id":3491412,"avatar_url":"https://avatars.githubusercontent.com/u/3491412?v=3","gravatar_id":"","url":"https://api.github.com/users/hoaphumanoid","html_url":"https://github.com/hoaphumanoid","followers_url":"https://api.github.com/users/hoaphumanoid/followers","following_url":"https://api.github.com/users/hoaphumanoid/following{/other_user}","gists_url":"https://api.github.com/users/hoaphumanoid/gists{/gist_id}","starred_url":"https://api.github.com/users/hoaphumanoid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hoaphumanoid/subscriptions","organizations_url":"https://api.github.com/users/hoaphumanoid/orgs","repos_url":"https://api.github.com/users/hoaphumanoid/repos","events_url":"https://api.github.com/users/hoaphumanoid/events{/privacy}","received_events_url":"https://api.github.com/users/hoaphumanoid/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-10-01T05:53:48Z","updated_at":"2016-10-01T05:53:48Z","closed_at":null,"body":"I'm training Imagenet with inception-bn in python:\r\n\r\n```bash\r\npython train_imagenet.py --batch-size 144 --lr 0.05 --lr-factor .94 --gpus 0,1,2,3 --num-epoch 60 --network inception-bn\r\n```\r\nIn epoch 0 the training and validation accuracy are reasonable, but after that the validation accuracy drops:\r\n```bash\r\n2016-09-29 20:56:15,427 Node[0] Update[8897]: Change learning rate to 4.70000e-02\r\n2016-09-29 20:56:16,214 Node[0] Epoch[0] Resetting Data Iterator\r\n2016-09-29 20:56:16,217 Node[0] Epoch[0] Time cost=7861.295\r\n2016-09-29 20:56:17,215 Node[0] Saved checkpoint to \"inception-bn-python-0-0001.params\"\r\n2016-09-29 20:58:03,494 Node[0] Epoch[0] Validation-accuracy=0.228867\r\n2016-09-29 20:58:03,495 Node[0] Epoch[0] Validation-top_k_accuracy_5=0.464500\r\n2016-09-29 20:58:03,495 Node[0] Epoch[0] Validation-top_k_accuracy_10=0.573894\r\n2016-09-29 20:58:03,495 Node[0] Epoch[0] Validation-top_k_accuracy_20=0.682352\r\n2016-09-29 20:58:47,169 Node[0] Epoch[1] Batch [50]     Speed: 166.24 samples/sec       Train-accuracy=0.354028\r\n2016-09-29 20:58:47,169 Node[0] Epoch[1] Batch [50]     Speed: 166.24 samples/sec       Train-top_k_accuracy_5=      0.607500\r\n2016-09-29 20:58:47,170 Node[0] Epoch[1] Batch [50]     Speed: 166.24 samples/sec       Train-top_k_accuracy_10      =0.706944\r\n2016-09-29 20:58:47,170 Node[0] Epoch[1] Batch [50]     Speed: 166.24 samples/sec       Train-top_k_accuracy_20      =0.791667\r\n...\r\n...\r\n2016-09-29 23:08:14,484 Node[0] Epoch[1] Batch [8850]   Speed: 162.40 samples/sec       Train-accuracy=0.455139\r\n2016-09-29 23:08:14,484 Node[0] Epoch[1] Batch [8850]   Speed: 162.40 samples/sec       Train-top_k_accuracy_5=      0.713472\r\n2016-09-29 23:08:14,484 Node[0] Epoch[1] Batch [8850]   Speed: 162.40 samples/sec       Train-top_k_accuracy_10      =0.795833\r\n2016-09-29 23:08:14,484 Node[0] Epoch[1] Batch [8850]   Speed: 162.40 samples/sec       Train-top_k_accuracy_20      =0.862917\r\n2016-09-29 23:08:53,043 Node[0] Update[17793]: Change learning rate to 4.41800e-02\r\n2016-09-29 23:08:54,986 Node[0] Epoch[1] Resetting Data Iterator\r\n2016-09-29 23:08:54,990 Node[0] Epoch[1] Time cost=7851.495\r\n2016-09-29 23:08:56,070 Node[0] Saved checkpoint to \"inception-bn-python-0-0002.params\"\r\n2016-09-29 23:10:42,088 Node[0] Epoch[1] Validation-accuracy=0.069464\r\n2016-09-29 23:10:42,088 Node[0] Epoch[1] Validation-top_k_accuracy_5=0.181394\r\n2016-09-29 23:10:42,088 Node[0] Epoch[1] Validation-top_k_accuracy_10=0.258162\r\n2016-09-29 23:10:42,088 Node[0] Epoch[1] Validation-top_k_accuracy_20=0.355344\r\n2016-09-29 23:11:26,427 Node[0] Epoch[2] Batch [50]     Speed: 164.00 samples/sec       Train-accuracy=0.466111\r\n2016-09-29 23:11:26,428 Node[0] Epoch[2] Batch [50]     Speed: 164.00 samples/sec       Train-top_k_accuracy_5=      0.723194\r\n2016-09-29 23:11:26,428 Node[0] Epoch[2] Batch [50]     Speed: 164.00 samples/sec       Train-top_k_accuracy_10      =0.806944\r\n2016-09-29 23:11:26,428 Node[0] Epoch[2] Batch [50]     Speed: 164.00 samples/sec       Train-top_k_accuracy_20      =0.869444\r\n...\r\n...\r\n2016-09-30 14:30:36,915 Node[0] Epoch[9] Batch [8850]   Speed: 216.61 samples/sec       Train-accuracy=0.678056\r\n2016-09-30 14:30:36,915 Node[0] Epoch[9] Batch [8850]   Speed: 216.61 samples/sec       Train-top_k_accuracy_5=      0.876250\r\n2016-09-30 14:30:36,915 Node[0] Epoch[9] Batch [8850]   Speed: 216.61 samples/sec       Train-top_k_accuracy_10      =0.920833\r\n2016-09-30 14:30:36,916 Node[0] Epoch[9] Batch [8850]   Speed: 216.61 samples/sec       Train-top_k_accuracy_20      =0.952361\r\n2016-09-30 14:31:01,541 Node[0] Update[88961]: Change learning rate to 2.69308e-02\r\n2016-09-30 14:31:08,212 Node[0] Epoch[9] Resetting Data Iterator\r\n2016-09-30 14:31:08,212 Node[0] Epoch[9] Time cost=5910.135\r\n2016-09-30 14:31:08,754 Node[0] Saved checkpoint to \"inception-bn-python-0-0010.params\"\r\n2016-09-30 14:32:28,038 Node[0] Epoch[9] Validation-accuracy=0.000998\r\n2016-09-30 14:32:28,038 Node[0] Epoch[9] Validation-top_k_accuracy_5=0.005009\r\n2016-09-30 14:32:28,039 Node[0] Epoch[9] Validation-top_k_accuracy_10=0.010317\r\n2016-09-30 14:32:28,039 Node[0] Epoch[9] Validation-top_k_accuracy_20=0.019836\r\n```\r\n\r\nAny suggestion?"}}
{"actor":{"display_login":"hoaphumanoid"}}