{"action":"created","issue":{"url":"https://api.github.com/repos/NervanaSystems/neon/issues/194","labels_url":"https://api.github.com/repos/NervanaSystems/neon/issues/194/labels{/name}","comments_url":"https://api.github.com/repos/NervanaSystems/neon/issues/194/comments","events_url":"https://api.github.com/repos/NervanaSystems/neon/issues/194/events","html_url":"https://github.com/NervanaSystems/neon/issues/194","id":129118092,"number":194,"title":"Cannot repeat experiment, when using model checkpoint ","user":{"login":"ilija139","id":1022717,"avatar_url":"https://avatars.githubusercontent.com/u/1022717?v=3","gravatar_id":"","url":"https://api.github.com/users/ilija139","html_url":"https://github.com/ilija139","followers_url":"https://api.github.com/users/ilija139/followers","following_url":"https://api.github.com/users/ilija139/following{/other_user}","gists_url":"https://api.github.com/users/ilija139/gists{/gist_id}","starred_url":"https://api.github.com/users/ilija139/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilija139/subscriptions","organizations_url":"https://api.github.com/users/ilija139/orgs","repos_url":"https://api.github.com/users/ilija139/repos","events_url":"https://api.github.com/users/ilija139/events{/privacy}","received_events_url":"https://api.github.com/users/ilija139/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/NervanaSystems/neon/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":{"login":"nervetumer","id":12916726,"avatar_url":"https://avatars.githubusercontent.com/u/12916726?v=3","gravatar_id":"","url":"https://api.github.com/users/nervetumer","html_url":"https://github.com/nervetumer","followers_url":"https://api.github.com/users/nervetumer/followers","following_url":"https://api.github.com/users/nervetumer/following{/other_user}","gists_url":"https://api.github.com/users/nervetumer/gists{/gist_id}","starred_url":"https://api.github.com/users/nervetumer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nervetumer/subscriptions","organizations_url":"https://api.github.com/users/nervetumer/orgs","repos_url":"https://api.github.com/users/nervetumer/repos","events_url":"https://api.github.com/users/nervetumer/events{/privacy}","received_events_url":"https://api.github.com/users/nervetumer/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2016-01-27T11:39:06Z","updated_at":"2016-01-28T03:20:20Z","closed_at":null,"body":"Example: \r\nCase 1: I train a network for 10 epochs and save the model state. Then I load the model state and train for another 5 epochs (in total 15 epochs) I get `Misclassification error = 2.4%`. \r\nCase 2: I train the same network for 15 epochs from start to finish (i.e. without making a checkpoint and loading the model state from there) I get `Misclassification error = 2.5%`. \r\n\r\nShouldn't both of them be the same? Saving the model state and loading it should not make a difference IMO. \r\n\r\nFollowing is the experiment log:\r\n\r\n```\r\n(.venv)ilija@deep27:~/hyper/neon-1.1.5/examples$ ./mnist_mlp.py -r 139 -b gpu --serialize 1 -s checkpoint.pkl\r\nEpoch 0   [Train |████████████████████|  469/469  batches, 0.26 cost, 1.63s]\r\nEpoch 1   [Train |████████████████████|  469/469  batches, 0.20 cost, 1.18s]\r\nEpoch 2   [Train |████████████████████|  469/469  batches, 0.17 cost, 1.19s]\r\nEpoch 3   [Train |████████████████████|  468/468  batches, 0.14 cost, 1.20s]\r\nEpoch 4   [Train |████████████████████|  469/469  batches, 0.13 cost, 1.20s]\r\nEpoch 5   [Train |████████████████████|  469/469  batches, 0.12 cost, 1.20s]\r\nEpoch 6   [Train |████████████████████|  469/469  batches, 0.11 cost, 1.41s]\r\nEpoch 7   [Train |████████████████████|  468/468  batches, 0.09 cost, 1.18s]\r\nEpoch 8   [Train |████████████████████|  469/469  batches, 0.08 cost, 1.35s]\r\nEpoch 9   [Train |████████████████████|  469/469  batches, 0.07 cost, 1.19s]\r\nMisclassification error = 2.5%\r\n(.venv)ilija@deep27:~/hyper/neon-1.1.5/examples$ ./mnist_mlp.py -r 139 -b gpu --model_file checkpoint.pkl -e 15\r\nEpoch 10  [Train |████████████████████|  469/469  batches, 0.07 cost, 1.58s]\r\nEpoch 11  [Train |████████████████████|  469/469  batches, 0.06 cost, 1.19s]\r\nEpoch 12  [Train |████████████████████|  469/469  batches, 0.06 cost, 1.19s]\r\nEpoch 13  [Train |████████████████████|  468/468  batches, 0.05 cost, 1.18s]\r\nEpoch 14  [Train |████████████████████|  469/469  batches, 0.05 cost, 1.18s]\r\nMisclassification error = 2.4%\r\n(.venv)ilija@deep27:~/hyper/neon-1.1.5/examples$ ./mnist_mlp.py -r 139 -b gpu -e 15\r\nEpoch 0   [Train |████████████████████|  469/469  batches, 0.26 cost, 1.41s]\r\nEpoch 1   [Train |████████████████████|  469/469  batches, 0.20 cost, 1.66s]\r\nEpoch 2   [Train |████████████████████|  469/469  batches, 0.17 cost, 1.67s]\r\nEpoch 3   [Train |████████████████████|  468/468  batches, 0.14 cost, 1.66s]\r\nEpoch 4   [Train |████████████████████|  469/469  batches, 0.13 cost, 1.23s]\r\nEpoch 5   [Train |████████████████████|  469/469  batches, 0.12 cost, 1.17s]\r\nEpoch 6   [Train |████████████████████|  469/469  batches, 0.11 cost, 1.20s]\r\nEpoch 7   [Train |████████████████████|  468/468  batches, 0.09 cost, 1.19s]\r\nEpoch 8   [Train |████████████████████|  469/469  batches, 0.08 cost, 1.17s]\r\nEpoch 9   [Train |████████████████████|  469/469  batches, 0.07 cost, 1.19s]\r\nEpoch 10  [Train |████████████████████|  469/469  batches, 0.07 cost, 1.21s]\r\nEpoch 11  [Train |████████████████████|  468/468  batches, 0.05 cost, 1.17s]\r\nEpoch 12  [Train |████████████████████|  469/469  batches, 0.05 cost, 1.17s]\r\nEpoch 13  [Train |████████████████████|  469/469  batches, 0.05 cost, 1.19s]\r\nEpoch 14  [Train |████████████████████|  469/469  batches, 0.05 cost, 1.17s]\r\nMisclassification error = 2.5%\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/NervanaSystems/neon/issues/comments/175938564","html_url":"https://github.com/NervanaSystems/neon/issues/194#issuecomment-175938564","issue_url":"https://api.github.com/repos/NervanaSystems/neon/issues/194","id":175938564,"user":{"login":"ilija139","id":1022717,"avatar_url":"https://avatars.githubusercontent.com/u/1022717?v=3","gravatar_id":"","url":"https://api.github.com/users/ilija139","html_url":"https://github.com/ilija139","followers_url":"https://api.github.com/users/ilija139/followers","following_url":"https://api.github.com/users/ilija139/following{/other_user}","gists_url":"https://api.github.com/users/ilija139/gists{/gist_id}","starred_url":"https://api.github.com/users/ilija139/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilija139/subscriptions","organizations_url":"https://api.github.com/users/ilija139/orgs","repos_url":"https://api.github.com/users/ilija139/repos","events_url":"https://api.github.com/users/ilija139/events{/privacy}","received_events_url":"https://api.github.com/users/ilija139/received_events","type":"User","site_admin":false},"created_at":"2016-01-28T03:20:20Z","updated_at":"2016-01-28T03:20:20Z","body":"I can confirm that the issue is due to partial mini batches. \r\nI think this is ok, as long we are aware of it. Since we can always adjust the batch size or save the model on exact multiples. \r\n\r\nfull log:\r\n```\r\n(.venv)ilija@deep29:~/hyper/neon-1.1.5/examples$ ./mnist_mlp.py -z 100 -e 5 -r 139 --serialize 1\r\n2016-01-28 11:15:25,220 - neon.util.argparser - WARNING - No path given for model serialization, usin\r\ng default \"neon_model.pkl\"\r\nEpoch 0   [Train |████████████████████|  600/600  batches, 0.27 cost, 2.81s]\r\nEpoch 1   [Train |████████████████████|  600/600  batches, 0.20 cost, 2.00s]\r\nEpoch 2   [Train |████████████████████|  600/600  batches, 0.17 cost, 1.73s]\r\nEpoch 3   [Train |████████████████████|  600/600  batches, 0.16 cost, 1.97s]\r\nEpoch 4   [Train |████████████████████|  600/600  batches, 0.13 cost, 1.84s]\r\nMisclassification error = 3.0%\r\n(.venv)ilija@deep29:~/hyper/neon-1.1.5/examples$ ./mnist_mlp.py -z 100 -e 10 -r 139 --model_file neon_model.pkl\r\nEpoch 5   [Train |████████████████████|  600/600  batches, 0.12 cost, 4.52s]\r\nEpoch 6   [Train |████████████████████|  600/600  batches, 0.10 cost, 1.77s]\r\nEpoch 7   [Train |████████████████████|  600/600  batches, 0.10 cost, 1.96s]\r\nEpoch 8   [Train |████████████████████|  600/600  batches, 0.08 cost, 2.02s]\r\nEpoch 9   [Train |████████████████████|  600/600  batches, 0.09 cost, 1.71s]\r\nMisclassification error = 2.8%\r\n(.venv)ilija@deep29:~/hyper/neon-1.1.5/examples$ ./mnist_mlp.py -z 100 -e 10 -r 139                  \r\nEpoch 0   [Train |████████████████████|  600/600  batches, 0.27 cost, 2.19s]\r\nEpoch 1   [Train |████████████████████|  600/600  batches, 0.20 cost, 1.83s]\r\nEpoch 2   [Train |████████████████████|  600/600  batches, 0.17 cost, 1.94s]\r\nEpoch 3   [Train |████████████████████|  600/600  batches, 0.16 cost, 1.81s]\r\nEpoch 4   [Train |████████████████████|  600/600  batches, 0.13 cost, 1.94s]\r\nEpoch 5   [Train |████████████████████|  600/600  batches, 0.12 cost, 1.99s]\r\nEpoch 6   [Train |████████████████████|  600/600  batches, 0.10 cost, 1.76s]\r\nEpoch 7   [Train |████████████████████|  600/600  batches, 0.10 cost, 1.98s]\r\nEpoch 8   [Train |████████████████████|  600/600  batches, 0.08 cost, 1.76s]\r\nEpoch 9   [Train |████████████████████|  600/600  batches, 0.09 cost, 1.94s]\r\nMisclassification error = 2.8%\r\n```"}}