{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/2246","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/2246/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/2246/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/2246/events","html_url":"https://github.com/dmlc/mxnet/issues/2246","id":156896666,"number":2246,"title":"Help.I write a deepid network using mxnet,but it is always getting the maximum Train-accuracy of 0.405538","user":{"login":"dushoufu","id":8800132,"avatar_url":"https://avatars.githubusercontent.com/u/8800132?v=3","gravatar_id":"","url":"https://api.github.com/users/dushoufu","html_url":"https://github.com/dushoufu","followers_url":"https://api.github.com/users/dushoufu/followers","following_url":"https://api.github.com/users/dushoufu/following{/other_user}","gists_url":"https://api.github.com/users/dushoufu/gists{/gist_id}","starred_url":"https://api.github.com/users/dushoufu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dushoufu/subscriptions","organizations_url":"https://api.github.com/users/dushoufu/orgs","repos_url":"https://api.github.com/users/dushoufu/repos","events_url":"https://api.github.com/users/dushoufu/events{/privacy}","received_events_url":"https://api.github.com/users/dushoufu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-05-26T03:34:49Z","updated_at":"2016-05-26T09:07:57Z","closed_at":null,"body":"I am tasked with transfering deepid using caffe to that using mxnet. All parameters come from a caffe prototxt. But how I train the network,the maximum Train-accuracy is only up  to like 0.40,while caffe can reach at least 80%. I think possibly my mxnet network has something wrong. Could some experts help check my programm and give some advice? Thank very much.\r\n\r\nIn this program, I changed a source code that belongs to mxnet original sources,which is python\\mxnet\\initializer.py. My aim is to initialize every layers with different initial weight and bias.\r\nIn initializer.py I add a class,as following:\r\n\r\n```python\r\nclass Custom(Initializer):\r\n    \"\"\"Initialize the weight and bias with custom requirements\r\n\r\n    Parameters\r\n    ----------\r\n    {\"layername\":{\"initialiseMethod\":(para1,para2)}}\r\n    initialiseMethod:\r\n    for weight: \"normal\",\"uniform\",\"orthogonal\",\"xavier\"\r\n    for bias:\"costant\"\r\n    dict : , optional\r\n        Standard deviation for gaussian distribution.\r\n    \"\"\"\r\n    weightMethods=[\"normal\",\"uniform\",\"orthogonal\",\"xavier\"]\r\n    biasMethods=[\"costant\"]\r\n    \r\n    def __init__(self, idict):\r\n        self.dict=idict\r\n\r\n    def _init_weight(self, name, arr):\r\n        if name in self.dict.keys():\r\n        \tdictPara=self.dict[name]\r\n        \tfor(k, v) in dictPara.items():\r\n        \t\tif k.lower()==\"normal\":\r\n        \t\t\trandom.normal(0, v, out=arr)\r\n        \t\telif k.lower()==\"uniform\":\r\n        \t\t\traise NotImplementedError(\"Not support at the moment\")\r\n        \t\telif k.lower()==\"orthogonal\":\r\n        \t\t\traise NotImplementedError(\"Not support at the moment\")\r\n        \t\telif k.lower()==\"xavier\":\r\n        \t\t\txa = Xavier(v[0],v[1],v[2])\r\n        \t\t\txa(name,arr)\r\n        else:\r\n        \traise NotImplementedError(\"Not support\")\r\n        \r\n    def _init_bias(self, name, arr):\r\n        if name in self.dict.keys():\r\n        \tdictPara=self.dict[name]\r\n        \tfor(k, v) in dictPara.items():\r\n        \t\tif k.lower()==\"costant\":\r\n        \t\t\tarr[:]=v\r\n        else\r\n        \traise NotImplementedError(\"Not support\")\r\n```\r\n\r\nmy data.py is:\r\n```python\r\n# pylint: skip-file\r\n\"\"\" data iterator for deepid \"\"\"\r\nimport sys\r\nimport os\r\nimport mxnet as mx\r\n\r\ndef deepid_iterator(batch_size, input_shape):\r\n    \"\"\"return train and val iterators for deepid\"\"\"\r\n    \r\n    train = mx.io.ImageRecordIter(\r\n        path_imgrec = \"./train.rec\",\r\n        data_shape  = input_shape,\r\n        batch_size  = batch_size,\r\n        rand_crop   = True,\r\n        rand_mirror = True,\r\n        shuffle     = True,\r\n\t\t\t\tmirror      = True,\r\n#        scale       = 0.00390625,\r\n        round_batch = True)\r\n    val = mx.io.ImageRecordIter(\r\n        path_imgrec = \"./test.rec\",\r\n        data_shape  = input_shape,\r\n        batch_size  = batch_size,\r\n        rand_crop   = False,\r\n        rand_mirror = False,\r\n        shuffle     = False,\r\n\t\t\t\tmirror      = False,\r\n#        scale       = 0.00390625,\r\n        round_batch = False)\r\n\r\n    return (train, val)\r\n\r\nmy deepid_py is:\r\n# pylint: skip-file\r\nfrom data import deepid_iterator\r\nimport mxnet as mx\r\nimport logging\r\nimport datetime\r\n\r\n#logging the network running\r\nprestr=datetime.datetime.now().strftime(\"%Y%m%d%H%M\")\r\noutfile=prestr+\".txt\"\r\n\r\n## define deepid\r\ninput_data = mx.symbol.Variable(name=\"data\")\r\n\r\n# stage 1\r\nconv1 = mx.symbol.Convolution(data=input_data, kernel=(4, 4), stride=(1, 1), num_filter=20,workspace=1024)\r\nrelu1 = mx.symbol.Activation(data=conv1, act_type=\"relu\")\r\npool1 = mx.symbol.Pooling(\r\n    data=relu1, pool_type=\"max\", kernel=(2, 2), stride=(2,2))\r\n\r\n#arg_shape, out_shape, aux_shape = conv1.infer_shape(date=(256,1,100,100))\r\n\r\n# stage 2\r\nconv2 = mx.symbol.Convolution(data=pool1, kernel=(3, 3), num_group=2, num_filter=40,workspace=1024)\r\nrelu2 = mx.symbol.Activation(data=conv2, act_type=\"relu\")\r\npool2 = mx.symbol.Pooling(data=relu2, kernel=(2, 2), stride=(2, 2), pool_type=\"max\")\r\n\r\n# stage 3\r\nconv3 = mx.symbol.Convolution(data=pool2, kernel=(3, 3),num_filter=60,workspace=1024)\r\npool3 = mx.symbol.Pooling(data=conv3, kernel=(2, 2), stride=(2, 2), pool_type=\"max\")\r\n\r\n# stage 4\r\nconv4 = mx.symbol.Convolution(data=pool3, kernel=(2, 2),stride=(2, 2), num_filter=80,workspace=1024)\r\n\r\n# stage 51\r\nflatten_conv4 = mx.symbol.Flatten(conv4)\r\n\r\n# stage 52\r\nflatten_pool3 = mx.symbol.Flatten(pool3)\r\n\r\n# stage 6\r\nconcat6=mx.symbol.Concat(*[flatten_conv4,flatten_pool3])\r\n\r\n# stage 7\r\nfc1 = mx.symbol.FullyConnected(data=concat6, num_hidden=160)\r\nbn1 = mx.symbol.BatchNorm(data=fc1)\r\n\r\n# stage 8\r\n#dropout1 = mx.symbol.Dropout(data=fc1, p=0.5)\r\n\r\n# stage 9\r\n#fc2 = mx.symbol.FullyConnected(data=dropout1, num_hidden=15529)\r\nfc2 = mx.symbol.FullyConnected(data=bn1, num_hidden=15529)\r\n#bn2 = mx.symbol.BatchNorm(data=fc2)\r\n\r\n# stage 9\r\nsoftmax = mx.symbol.SoftmaxOutput(data=fc2, name='softmax')\r\n#softmax = mx.symbol.SoftmaxOutput(data=bn2, name='softmax')\r\narg_names = softmax.list_arguments()\r\nprint arg_names\r\n\r\ninitDict={'convolution0_weight':{'normal':0.125}, 'convolution0_bias':{'costant':0}, \\\r\n\t\t'convolution1_weight':{'normal':0.2}, 'convolution1_bias':{'costant':0.1}, \\\r\n                'convolution2_weight':{'normal':0.2}, 'convolution2_bias':{'costant':0}, \\\r\n                'convolution3_weight':{'normal':0.5}, 'convolution3_bias':{'costant':0.1}, \\\r\n                'fullyconnected0_weight':{'normal':0.0001}, 'fullyconnected0_bias':{'costant':0.1}, \\\r\n                'fullyconnected1_weight':{'xavier':(\"uniform\", \"avg\", 3)}, 'fullyconnected1_bias':{'costant':0}} \r\n## data\r\nbatch_size = 256 \r\ntrain, val = deepid_iterator(batch_size=batch_size, input_shape=(1,100,100))\r\n\r\nlr = mx.lr_scheduler.FactorScheduler(step=10000, factor=.9)\r\noptimizer = mx.optimizer.SGD(\r\nlearning_rate = 0.001,\r\nmomentum = 0.9,\r\nwd = 0.005,\r\nlr_scheduler = lr,\r\nclip_gradient = 10)\r\n\r\n## train\r\nnum_gpus = 2 \r\ngpus = [mx.gpu(i) for i in range(num_gpus)]\r\n\r\nmodel = mx.model.FeedForward(\r\n    ctx           = mx.gpu(0),\r\n    symbol        = softmax,\r\n    num_epoch     = 20,\r\n    initializer   = mx.initializer.Custom(idict=initDict),\r\n    optimizer=optimizer)\r\n    \r\n\"\"\"\r\nmodel = mx.model.FeedForward(\r\n    ctx           = gpus,\r\n    symbol        = softmax,\r\n    num_epoch     = 20,\r\n    learning_rate = 0.0001,\r\n    momentum      = 0.9,\r\n    initializer   = mx.initializer.Custom(idict=initDict),\r\n    wd            = 0.00001)\r\n\"\"\"\r\n\r\nlogging.basicConfig(level = logging.DEBUG)\r\n\r\nsave_model_prefix = './model/deepid'+prestr\r\ncheckpoint = mx.callback.do_checkpoint(save_model_prefix)\r\n\r\nmodel.fit(X = train, eval_data = val,\r\n          batch_end_callback = mx.callback.Speedometer(batch_size=batch_size),\r\n          epoch_end_callback = checkpoint)\r\n```"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/221817721","html_url":"https://github.com/dmlc/mxnet/issues/2246#issuecomment-221817721","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/2246","id":221817721,"user":{"login":"dushoufu","id":8800132,"avatar_url":"https://avatars.githubusercontent.com/u/8800132?v=3","gravatar_id":"","url":"https://api.github.com/users/dushoufu","html_url":"https://github.com/dushoufu","followers_url":"https://api.github.com/users/dushoufu/followers","following_url":"https://api.github.com/users/dushoufu/following{/other_user}","gists_url":"https://api.github.com/users/dushoufu/gists{/gist_id}","starred_url":"https://api.github.com/users/dushoufu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dushoufu/subscriptions","organizations_url":"https://api.github.com/users/dushoufu/orgs","repos_url":"https://api.github.com/users/dushoufu/repos","events_url":"https://api.github.com/users/dushoufu/events{/privacy}","received_events_url":"https://api.github.com/users/dushoufu/received_events","type":"User","site_admin":false},"created_at":"2016-05-26T09:07:57Z","updated_at":"2016-05-26T09:07:57Z","body":"Mine converged normally, the main problem is that the train accuracy is too low only at like 40%. But with the suggestion of @piiswrong. At the moment it seems to be much better. I am waiting the training result now."}}
{"actor":{"display_login":"dushoufu"}}