{"action":"opened","issue":{"url":"https://api.github.com/repos/vlfeat/matconvnet/issues/491","repository_url":"https://api.github.com/repos/vlfeat/matconvnet","labels_url":"https://api.github.com/repos/vlfeat/matconvnet/issues/491/labels{/name}","comments_url":"https://api.github.com/repos/vlfeat/matconvnet/issues/491/comments","events_url":"https://api.github.com/repos/vlfeat/matconvnet/issues/491/events","html_url":"https://github.com/vlfeat/matconvnet/issues/491","id":147160892,"number":491,"title":"training in different steps","user":{"login":"alevillard","id":18304893,"avatar_url":"https://avatars.githubusercontent.com/u/18304893?v=3","gravatar_id":"","url":"https://api.github.com/users/alevillard","html_url":"https://github.com/alevillard","followers_url":"https://api.github.com/users/alevillard/followers","following_url":"https://api.github.com/users/alevillard/following{/other_user}","gists_url":"https://api.github.com/users/alevillard/gists{/gist_id}","starred_url":"https://api.github.com/users/alevillard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alevillard/subscriptions","organizations_url":"https://api.github.com/users/alevillard/orgs","repos_url":"https://api.github.com/users/alevillard/repos","events_url":"https://api.github.com/users/alevillard/events{/privacy}","received_events_url":"https://api.github.com/users/alevillard/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-04-09T20:27:39Z","updated_at":"2016-04-09T20:27:39Z","closed_at":null,"body":"Hi, I try to finetune vggD, but I saw that if I make for example a training of 10 epochs in one time it works at the test. But if I train for example the first 5 epoch, and then stop it (for problems of time for example), and then start again to train other 5 epoch, resuming the previous five, I will see that the test fail because all the scores output are close to zero. Do you know if there is some option to add to the code for which I can stop a train and then restart from that point avoiding this problem? Maybe I see that stopping and then resuming, the code re-reinizialized the parameters and maybe that's why of this problem I have. Someone could help me? Thank you, I attached the code I use for finetuning:\r\n\r\n function [net, info] = vggD_train(imdb, expDir, varargin)\r\n\r\n\tvl_setupnn;\r\n\r\n\t% some common options\r\n\topts.train.batchSize = 20;\r\n\topts.train.numEpochs = 10 ;\r\n\topts.train.continue = true ;\r\n\topts.train.gpus = [1] ;\r\n\topts.train.learningRate = 1e-3;\r\n\topts.train.weightDecay = 2e-4;\r\n\topts.train.momentum = 0.9;\r\n\topts.train.expDir = expDir;\r\n\topts.train.numSubBatches = 1;\r\n\t% getBatch options\r\n\tbopts.useGpu = numel(opts.train.gpus) >  0 ;\r\n\topts.optMethod = 'gradient'; % ['adagrad', 'gradient']\r\n\r\n     if(numel(varargin) > 0)\r\n\t\tnetPre = varargin{1};\r\n    end\r\n% network definition!\r\n\t% MATLAB handle, passed by reference\r\n\tnet = dagnn.DagNN() ;\r\n\r\n    net.addLayer('conv1_1', dagnn.Conv('size', [3 3 3 64], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'input'}, {'conv1'},  {'conv1_1f'  'conv1_1b'});\r\n\tnet.addLayer('relu1_1', dagnn.ReLU(), {'conv1'}, {'relu1'}, {});\r\n\tnet.addLayer('conv1_2', dagnn.Conv('size', [3 3 64 64], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'relu1'}, {'conv2'},  {'conv1_2f'  'conv1_2b'});\r\n\tnet.addLayer('relu1_2', dagnn.ReLU(), {'conv2'}, {'relu2'}, {});\r\n\t%net.addLayer('lrn1', dagnn.LRN('param', [5 1 0.0001/5 0.75]), {'relu1'}, {'lrn1'}, {});\r\n\tnet.addLayer('pool1', dagnn.Pooling('method', 'max', 'poolSize', [2, 2], 'stride', [2 2], 'pad', [0 0 0 0]), {'relu2'}, {'pool1'}, {});\r\n    \r\n    net.addLayer('conv2_1', dagnn.Conv('size', [3 3 64 128], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'pool1'}, {'conv3'},  {'conv2_1f'  'conv2_1b'});\r\n\tnet.addLayer('relu2_1', dagnn.ReLU(), {'conv3'}, {'relu3'}, {});\r\n\tnet.addLayer('conv2_2', dagnn.Conv('size', [3 3 128 128], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'relu3'}, {'conv4'},  {'conv2_2f'  'conv2_2b'});\r\n\tnet.addLayer('relu2_2', dagnn.ReLU(), {'conv4'}, {'relu4'}, {});\r\n\t%net.addLayer('lrn2', dagnn.LRN('param', [5 1 0.0001/5 0.75]), {'relu2'}, {'lrn2'}, {});\r\n\tnet.addLayer('pool2', dagnn.Pooling('method', 'max', 'poolSize', [2, 2], 'stride', [2 2], 'pad', [0 0 0 0]), {'relu4'}, {'pool2'}, {});\r\n\r\n    net.addLayer('conv3_1', dagnn.Conv('size', [3 3 128 256], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'pool2'}, {'conv5'},  {'conv3_1f'  'conv3_1b'});\r\n\tnet.addLayer('relu3_1', dagnn.ReLU(), {'conv5'}, {'relu5'}, {});\r\n    net.addLayer('conv3_2', dagnn.Conv('size', [3 3 256 256], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'relu5'}, {'conv6'},  {'conv3_2f'  'conv3_2b'});\r\n\tnet.addLayer('relu3_2', dagnn.ReLU(), {'conv6'}, {'relu6'}, {});\r\n\tnet.addLayer('conv3_3', dagnn.Conv('size', [3 3 256 256], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'relu6'}, {'conv7'},  {'conv3_3f'  'conv3_3b'});\r\n\tnet.addLayer('relu3_3', dagnn.ReLU(), {'conv7'}, {'relu7'}, {});\r\n\t%net.addLayer('lrn3', dagnn.LRN('param', [5 1 0.0001/5 0.75]), {'relu2'}, {'lrn2'}, {});\r\n\tnet.addLayer('pool3', dagnn.Pooling('method', 'max', 'poolSize', [2, 2], 'stride', [2 2], 'pad', [0 0 0 0]), {'relu7'}, {'pool3'}, {});\r\n    \r\n    net.addLayer('conv4_1', dagnn.Conv('size', [3 3 256 512], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'pool3'}, {'conv8'},  {'conv4_1f'  'conv4_1b'});\r\n\tnet.addLayer('relu4_1', dagnn.ReLU(), {'conv8'}, {'relu8'}, {});\r\n\tnet.addLayer('conv4_2', dagnn.Conv('size', [3 3 512 512], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'relu8'}, {'conv9'},  {'conv4_2f'  'conv4_2b'});\r\n\tnet.addLayer('relu4_2', dagnn.ReLU(), {'conv9'}, {'relu9'}, {});\r\n\tnet.addLayer('conv4_3', dagnn.Conv('size', [3 3 512 512], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'relu9'}, {'conv10'},  {'conv4_3f'  'conv4_3b'});\r\n\tnet.addLayer('relu4_3', dagnn.ReLU(), {'conv10'}, {'relu10'}, {});\r\n\t%net.addLayer('lrn4', dagnn.LRN('param', [5 1 0.0001/5 0.75]), {'relu2'}, {'lrn2'}, {});\r\n\tnet.addLayer('pool4', dagnn.Pooling('method', 'max', 'poolSize', [2, 2], 'stride', [2 2], 'pad', [0 0 0 0]), {'relu10'}, {'pool4'}, {});\r\n  \r\n    net.addLayer('conv5_1', dagnn.Conv('size', [3 3 512 512], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'pool4'}, {'conv11'},  {'conv5_1f'  'conv5_1b'});\r\n\tnet.addLayer('relu5_1', dagnn.ReLU(), {'conv11'}, {'relu11'}, {});\r\n\tnet.addLayer('conv5_2', dagnn.Conv('size', [3 3 512 512], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'relu11'}, {'conv12'},  {'conv5_2f'  'conv5_2b'});\r\n\tnet.addLayer('relu5_2', dagnn.ReLU(), {'conv12'}, {'relu12'}, {});\r\n\tnet.addLayer('conv5_3', dagnn.Conv('size', [3 3 512 512], 'hasBias', true, 'stride', [1, 1], 'pad', [1 1 1 1]), {'relu12'}, {'conv13'},  {'conv5_3f'  'conv5_3b'});\r\n\tnet.addLayer('relu5_3', dagnn.ReLU(), {'conv13'}, {'relu13'}, {});\r\n\tnet.addLayer('pool5', dagnn.Pooling('method', 'max', 'poolSize', [2, 2], 'stride', [2 2], 'pad', [0 0 0 0]), {'relu13'}, {'pool5'}, {});\r\n\r\n\tnet.addLayer('conv6', dagnn.Conv('size', [1 1 512 4096], 'hasBias', true, 'stride', [1, 1], 'pad', [0 0 0 0]), {'pool5'}, {'conv14'},  {'conv6f'  'conv6b'});\r\n\tnet.addLayer('relu6', dagnn.ReLU(), {'conv14'}, {'relu14'}, {});\r\n\tnet.addLayer('drop6', dagnn.DropOut('rate', 0.5), {'relu14'}, {'drop6'}, {});\r\n\r\n\tnet.addLayer('conv7', dagnn.Conv('size', [1 1 4096 4096], 'hasBias', true, 'stride', [1, 1], 'pad', [0 0 0 0]), {'drop6'}, {'conv15'},  {'conv7f'  'conv7b'});\r\n\tnet.addLayer('relu7', dagnn.ReLU(), {'conv15'}, {'relu15'}, {});\r\n\tnet.addLayer('drop7', dagnn.DropOut('rate', 0.5), {'relu15'}, {'drop7'}, {});\r\n\r\n\tnet.addLayer('classifier', dagnn.Conv('size', [1 1 4096 263], 'hasBias', true, 'stride', [1, 1], 'pad', [0 0 0 0]), {'drop7'}, {'classifier'},  {'conv8f'  'conv8b'});\r\n\tnet.addLayer('prob', dagnn.SoftMax(), {'classifier'}, {'prob'}, {});\r\n\tnet.addLayer('objective', dagnn.Loss('loss', 'log'), {'prob', 'label'}, {'objective'}, {});\r\n\tnet.addLayer('error', dagnn.Loss('loss', 'classerror'), {'prob','label'}, 'error') ;\r\n\t% -- end of the network\r\n\r\n    % initialization of the weights (CRITICAL!!!!)\r\n    if(numel(varargin) > 0)\r\n\t\tinitNet_FineTuning(net, netPre);\r\n        \r\n\telse\r\n\t\t initNet_He(net);\r\n\tend\t\r\n   \r\n    %train\r\n    info = cnn_train_dag(net, imdb, @(i,b) getBatchDisk(bopts,i,b), opts.train, 'val', find(imdb.images.set == 2)) ;\r\nend\r\n\r\nfunction initNet_FineTuning(net, netPre)\r\n\tnet.initParams();\r\n\r\n\tind = 1;\r\n\tfor l=1:length(net.layers)-4\r\n\t\t% is a convolution layer?\r\n\t\tif(strcmp(class(net.layers(l).block), 'dagnn.Conv'))\r\n\t\t\tf_ind = net.layers(l).paramIndexes(1);\r\n\t\t\tb_ind = net.layers(l).paramIndexes(2);\r\n            \r\n            net.params(f_ind).value = netPre.params(ind).value\r\n\t\t\tind = ind + 1;\r\n\t\t\tnet.params(b_ind).value = netPre.params(ind).value\r\n\t\t\tind = ind + 1;\r\n          if l<= length(net.layers)-10  %freeze the pararamiters\r\n\t\t\tnet.params(f_ind).learningRate = 0;\r\n\t\t\tnet.params(f_ind).weightDecay = 1;\r\n\t\t\tnet.params(b_ind).learningRate = 0;\r\n\t\t\tnet.params(b_ind).weightDecay = 1;\r\n\t\t\t\r\n          else  %let paramiters to change in last layers\r\n            net.params(f_ind).learningRate = 1;\r\n\t\t\tnet.params(f_ind).weightDecay = 1;\r\n\t\t\tnet.params(b_ind).learningRate = 1;\r\n\t\t\tnet.params(b_ind).weightDecay = 1;   \r\n          end\r\n\t\tend\r\n\tend\r\n\r\n\tfor l=length(net.layers)-3:length(net.layers)\r\n\t\t% is a convolution layer?\r\n\t\tif(strcmp(class(net.layers(l).block), 'dagnn.Conv'))\r\n\t\t\tf_ind = net.layers(l).paramIndexes(1)\r\n\t\t\tb_ind = net.layers(l).paramIndexes(2)\r\n\r\n            \r\n\t\t\t[h,w,in,out] = size(net.params(f_ind).value)\r\n\t\t\the_gain = 1e-2*sqrt(2/(h*w*in)) % sqrt(1/fan_in)\r\n\t\t\tnet.params(f_ind).value = (1/5)*he_gain*randn(size(net.params(f_ind).value), 'single');\r\n\t\t\tnet.params(f_ind).learningRate = 1;\r\n\t\t\tnet.params(f_ind).weightDecay = 1;\r\n\r\n\t\t\tnet.params(b_ind).value = zeros(size(net.params(b_ind).value), 'single');\r\n\t\t\tnet.params(b_ind).learningRate = 0.5;\r\n\t\t\tnet.params(b_ind).weightDecay = 1;\r\n\t\t\t\r\n\t\tend\r\n\tend\r\nend\r\n\r\n\r\n% getBatch for IMDBs that are too big to be in RAM\r\nfunction inputs = getBatchDisk(opts, imdb, batch)\r\n\tcell_images = vl_imreadjpeg(imdb.images.filenames(batch),'numThreads', 2, 'Resize', [224 224]) ;\r\n\t%[H, W, CH] = size(cell_images{1});\r\n\r\n\timages = zeros(224, 224, 3, 'single');\r\n    %size(images)\r\n\tfor i=1:numel(cell_images)\r\n\t\tim = cell_images{i};\r\n        %im=imresize(im, [224 224]);\r\n        %size(im)\r\n\t\timages(:,:,:, i) = im - imdb.images.data_mean;\r\n\tend\r\n\r\n\tlabels = imdb.images.labels(1,batch) ;\r\n\tif opts.useGpu > 0\r\n  \t\timages = gpuArray(images) ;\r\n\tend\r\n\r\n\tinputs = {'input', images, 'label', labels} ;\r\nend\r\n\r\nfunction initNet_He(net)\r\n\tnet.initParams();\r\n\tfor l=1:length(net.layers)\r\n\t\t% is a convolution layer?\r\n\t\tif(strcmp(class(net.layers(l).block), 'dagnn.Conv'))\r\n\t\t\tf_ind = net.layers(l).paramIndexes(1);\r\n\t\t\tb_ind = net.layers(l).paramIndexes(2);\r\n\r\n\t\t\t[h,w,in,out] = size(net.params(f_ind).value);\r\n\t\t\the_gain = 1e-2*sqrt(2/(h*w*in)); % sqrt(1/fan_in)\r\n\t\t\tnet.params(f_ind).value = (1/2)*he_gain*randn(size(net.params(f_ind).value), 'single')\r\n\t\t\tnet.params(f_ind).learningRate = 1;\r\n\t\t\tnet.params(f_ind).weightDecay = 1;\r\n\r\n\t\t\tnet.params(b_ind).value = zeros(size(net.params(b_ind).value), 'single');\r\n\t\t\tnet.params(b_ind).learningRate = 0.5;\r\n\t\t\tnet.params(b_ind).weightDecay = 1;\r\n\t\tend\r\n\tend\r\nend"}}