{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/gluon-cv/issues/142","repository_url":"https://api.github.com/repos/dmlc/gluon-cv","labels_url":"https://api.github.com/repos/dmlc/gluon-cv/issues/142/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/gluon-cv/issues/142/comments","events_url":"https://api.github.com/repos/dmlc/gluon-cv/issues/142/events","html_url":"https://github.com/dmlc/gluon-cv/issues/142","id":326004411,"number":142,"title":"The training loss is nan in SSD","user":{"login":"1292765944","id":3321128,"avatar_url":"https://avatars3.githubusercontent.com/u/3321128?v=4","gravatar_id":"","url":"https://api.github.com/users/1292765944","html_url":"https://github.com/1292765944","followers_url":"https://api.github.com/users/1292765944/followers","following_url":"https://api.github.com/users/1292765944/following{/other_user}","gists_url":"https://api.github.com/users/1292765944/gists{/gist_id}","starred_url":"https://api.github.com/users/1292765944/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/1292765944/subscriptions","organizations_url":"https://api.github.com/users/1292765944/orgs","repos_url":"https://api.github.com/users/1292765944/repos","events_url":"https://api.github.com/users/1292765944/events{/privacy}","received_events_url":"https://api.github.com/users/1292765944/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-24T07:55:07Z","updated_at":"2018-05-24T18:52:32Z","closed_at":null,"author_association":"NONE","body":"I use the default training script (train_ssd.py) to train SSD300, However, the training loss seems to be large and do not converge. The log file is shown below, and what's the problem for this? Thanks!\r\n\r\n```\r\nNamespace(batch_size=32, data_shape=300, dataset='voc', epochs=240, gpus='0', log_interval=100, lr=0.001, lr_decay=0.1, lr_decay_epoch='160,200', momentum=0.9, network='vgg16_atrous', num_workers=4, resume='', save_interval=10, save_prefix='ssd_300_vgg16_atrous_voc', seed=233, start_epoch=0, wd=0.0005)\r\nStart training from [Epoch 0]\r\n[Epoch 0][Batch 99], Speed: 31.855627 samples/sec, CrossEntropy=12.250115, SmoothL1=nan\r\n[Epoch 0][Batch 199], Speed: 31.908488 samples/sec, CrossEntropy=12.212773, SmoothL1=nan\r\n[Epoch 0][Batch 299], Speed: 31.279352 samples/sec, CrossEntropy=12.199556, SmoothL1=nan\r\n[Epoch 0][Batch 399], Speed: 30.720933 samples/sec, CrossEntropy=12.192253, SmoothL1=nan\r\n[Epoch 0][Batch 499], Speed: 31.459032 samples/sec, CrossEntropy=12.187363, SmoothL1=nan\r\n[Epoch 0] Training cost: 548.135802, CrossEntropy=12.186622, SmoothL1=nan\r\n[Epoch 0] Validation: \r\naeroplane=0.000389\r\nbicycle=0.000102\r\nbird=0.004152\r\nboat=0.000011\r\nbottle=0.000183\r\nbus=0.000084\r\ncar=0.000175\r\ncat=0.004419\r\nchair=0.001955\r\ncow=0.000036\r\ndiningtable=0.000058\r\ndog=0.001009\r\nhorse=0.000108\r\nmotorbike=0.005600\r\nperson=0.000281\r\npottedplant=0.000103\r\nsheep=0.000045\r\nsofa=0.000598\r\ntrain=0.002716\r\ntvmonitor=0.000453\r\nmAP=0.001124\r\n[Epoch 1][Batch 99], Speed: 31.706873 samples/sec, CrossEntropy=12.165724, SmoothL1=nan\r\n```"},"comment":{"url":"https://api.github.com/repos/dmlc/gluon-cv/issues/comments/391821568","html_url":"https://github.com/dmlc/gluon-cv/issues/142#issuecomment-391821568","issue_url":"https://api.github.com/repos/dmlc/gluon-cv/issues/142","id":391821568,"user":{"login":"zhreshold","id":3307514,"avatar_url":"https://avatars2.githubusercontent.com/u/3307514?v=4","gravatar_id":"","url":"https://api.github.com/users/zhreshold","html_url":"https://github.com/zhreshold","followers_url":"https://api.github.com/users/zhreshold/followers","following_url":"https://api.github.com/users/zhreshold/following{/other_user}","gists_url":"https://api.github.com/users/zhreshold/gists{/gist_id}","starred_url":"https://api.github.com/users/zhreshold/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhreshold/subscriptions","organizations_url":"https://api.github.com/users/zhreshold/orgs","repos_url":"https://api.github.com/users/zhreshold/repos","events_url":"https://api.github.com/users/zhreshold/events{/privacy}","received_events_url":"https://api.github.com/users/zhreshold/received_events","type":"User","site_admin":false},"created_at":"2018-05-24T18:52:32Z","updated_at":"2018-05-24T18:52:32Z","author_association":"MEMBER","body":"The smoothl1 is nan at the first batch, suggesting there's some problem. \r\nWhat is the mxnet version you use"}}
{"actor":{"display_login":"zhreshold"}}