{"action":"created","issue":{"url":"https://api.github.com/repos/liuzhuang13/DenseNetCaffe/issues/2","repository_url":"https://api.github.com/repos/liuzhuang13/DenseNetCaffe","labels_url":"https://api.github.com/repos/liuzhuang13/DenseNetCaffe/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/liuzhuang13/DenseNetCaffe/issues/2/comments","events_url":"https://api.github.com/repos/liuzhuang13/DenseNetCaffe/issues/2/events","html_url":"https://github.com/liuzhuang13/DenseNetCaffe/issues/2","id":175984332,"number":2,"title":"Batch normalization with or without learned offset","user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2016-09-09T11:07:15Z","updated_at":"2016-09-14T02:05:58Z","closed_at":"2016-09-12T09:28:03Z","body":"Nice paper! I just have a minor detail question for reimplementing it.\r\n\r\nIn https://github.com/liuzhuang13/DenseNetCaffe/blob/master/make_densenet.py#L8, you use:\r\n`scale = L.Scale(batch_norm, bias_term=False, ...)`\r\nThis would correspond to batch normalization with learned `gamma`, but without `beta`.\r\nIn https://github.com/liuzhuang13/DenseNet/blob/master/densenet.lua#L28, you use:\r\n`convFactory:add(cudnn.SpatialBatchNormalization(nChannels))`\r\nThis includes a learnable `beta`. So I think the Caffe code needs to be adapted to match the Torch implementation.\r\n\r\nOn a side note, the convolutions (both in Caffe and Torch, if I see correctly) all have a bias term, but that will be rendered meaningless by the following batch normalization."},"comment":{"url":"https://api.github.com/repos/liuzhuang13/DenseNetCaffe/issues/comments/246884146","html_url":"https://github.com/liuzhuang13/DenseNetCaffe/issues/2#issuecomment-246884146","issue_url":"https://api.github.com/repos/liuzhuang13/DenseNetCaffe/issues/2","id":246884146,"user":{"login":"liuzhuang13","id":8370623,"avatar_url":"https://avatars.githubusercontent.com/u/8370623?v=3","gravatar_id":"","url":"https://api.github.com/users/liuzhuang13","html_url":"https://github.com/liuzhuang13","followers_url":"https://api.github.com/users/liuzhuang13/followers","following_url":"https://api.github.com/users/liuzhuang13/following{/other_user}","gists_url":"https://api.github.com/users/liuzhuang13/gists{/gist_id}","starred_url":"https://api.github.com/users/liuzhuang13/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liuzhuang13/subscriptions","organizations_url":"https://api.github.com/users/liuzhuang13/orgs","repos_url":"https://api.github.com/users/liuzhuang13/repos","events_url":"https://api.github.com/users/liuzhuang13/events{/privacy}","received_events_url":"https://api.github.com/users/liuzhuang13/received_events","type":"User","site_admin":false},"created_at":"2016-09-14T02:05:58Z","updated_at":"2016-09-14T02:05:58Z","body":"> Both results are worse than before. Maybe the extra transition layer was helpful, although you didn't use it -- you might want to give it a try.\r\n\r\nThanks for your run! The purpose of the last transition layer seems not very clear so we didn't use it. I guess the performance gain by using it is due to more parameters, which can be turned into extra depth or width, so we'll leave the model as before. Thanks anyway!\r\n\r\n> How much did your performance suffer when you used a validation set?\r\n\r\nRight, I guess maybe missing 5000 training examples hurt. When we use a validation set, on C10+ we got about 0.3%-0.5% worse results. On C10 that should be more since C10 is much smaller than C10+ in terms of data size, so 5000 is not a negligible.\r\n\r\n>  Was it the same for you?\r\nYes, in our setting training loss (or error) usually goes near to 0, but the validation or test error doesn't. We didn't monitor the validation loss though, sorry.\r\n\r\n\r\n"}}
{"actor":{"display_login":"liuzhuang13"}}