{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/4189","repository_url":"https://api.github.com/repos/BVLC/caffe","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/4189/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/4189/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/4189/events","html_url":"https://github.com/BVLC/caffe/issues/4189","id":156022623,"number":4189,"title":"random_seed doesnt create identical loss/accuracy in different runs","user":{"login":"Coderx7","id":5382892,"avatar_url":"https://avatars.githubusercontent.com/u/5382892?v=3","gravatar_id":"","url":"https://api.github.com/users/Coderx7","html_url":"https://github.com/Coderx7","followers_url":"https://api.github.com/users/Coderx7/followers","following_url":"https://api.github.com/users/Coderx7/following{/other_user}","gists_url":"https://api.github.com/users/Coderx7/gists{/gist_id}","starred_url":"https://api.github.com/users/Coderx7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Coderx7/subscriptions","organizations_url":"https://api.github.com/users/Coderx7/orgs","repos_url":"https://api.github.com/users/Coderx7/repos","events_url":"https://api.github.com/users/Coderx7/events{/privacy}","received_events_url":"https://api.github.com/users/Coderx7/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-05-20T18:23:27Z","updated_at":"2016-05-24T20:05:27Z","closed_at":"2016-05-24T20:05:27Z","body":"I tired to use random_seed in the solver to create a deterministic output result, but to no avail. \r\nhere is a very simple solver demonstrating the cause : \r\n```\r\ntest_iter: 200\r\ntest_interval: 500\r\nbase_lr: 0.1\r\ndisplay: 100\r\nmax_iter: 40000\r\nlr_policy: \"multistep\"\r\ngamma: 0.1\r\nmomentum: 0.9\r\nweight_decay: 0.001\r\nsnapshot: 500\r\nsnapshot_prefix: \"examples/cifar10/cif\"\r\nsolver_mode: GPU\r\ndevice_id: 0\r\nrandom_seed: 1201\r\nnet: \"examples/cifar10/cifar10_full_relu_train_test_bn.prototxt\"\r\ndelta: 0.001\r\nstepvalue: 4000\r\nstepvalue: 6500\r\nstepvalue: 9500\r\nstepvalue: 12000\r\nstepvalue: 15000\r\nstepvalue: 18000\r\ntype: \"AdaDelta\"\r\n```\r\nAnd this is how the result differ for each run: \r\n![figure_1-3](https://cloud.githubusercontent.com/assets/5382892/15437658/a238f654-1edd-11e6-8919-162e71cb6170.png)\r\n\r\n\r\nlog1:\r\n```\r\nI0520 22:56:57.964998  8952 solver.cpp:280] Learning Rate Policy: multistep\r\nI0520 22:56:57.965998  8952 solver.cpp:337] Iteration 0, Testing net (#0)\r\nI0520 22:56:58.506774  8952 blocking_queue.cpp:50] Data layer prefetch queue empty\r\nI0520 22:56:58.529290 10292 blocking_queue.cpp:50] Waiting for data\r\nI0520 22:56:58.765458  8952 solver.cpp:404]     Test net output #0: accuracy = 0.0994001\r\nI0520 22:56:58.765458  8952 solver.cpp:404]     Test net output #1: loss = 78.6553 (* 1 = 78.6553 loss)\r\nI0520 22:56:58.817082  8952 solver.cpp:228] Iteration 0, loss = 2.37327\r\nI0520 22:56:58.817082  8952 solver.cpp:244]     Train net output #0: loss = 2.37327 (* 1 = 2.37327 loss)\r\nI0520 22:56:58.817082  8952 sgd_solver.cpp:106] Iteration 0, lr = 0.1\r\nI0520 22:57:01.425292  8952 solver.cpp:228] Iteration 100, loss = 1.84951\r\nI0520 22:57:01.425292  8952 solver.cpp:244]     Train net output #0: loss = 1.84951 (* 1 = 1.84951 loss)\r\nI0520 22:57:01.425292  8952 sgd_solver.cpp:106] Iteration 100, lr = 0.1\r\nI0520 22:57:04.037649  8952 solver.cpp:228] Iteration 200, loss = 1.83825\r\nI0520 22:57:04.037649  8952 solver.cpp:244]     Train net output #0: loss = 1.83825 (* 1 = 1.83825 loss)\r\nI0520 22:57:04.037649  8952 sgd_solver.cpp:106] Iteration 200, lr = 0.1\r\nI0520 22:57:06.640002  8952 solver.cpp:228] Iteration 300, loss = 1.65452\r\nI0520 22:57:06.640002  8952 solver.cpp:244]     Train net output #0: loss = 1.65452 (* 1 = 1.65452 loss)\r\nI0520 22:57:06.640002  8952 sgd_solver.cpp:106] Iteration 300, lr = 0.1\r\nI0520 22:57:09.276876  8952 solver.cpp:228] Iteration 400, loss = 1.49125\r\nI0520 22:57:09.276876  8952 solver.cpp:244]     Train net output #0: loss = 1.49125 (* 1 = 1.49125 loss)\r\nI0520 22:57:09.276876  8952 sgd_solver.cpp:106] Iteration 400, lr = 0.1\r\nI0520 22:57:11.879228  8952 solver.cpp:454] Snapshotting to binary proto file examples/cifar10/cif_iter_500.caffemodel\r\nI0520 22:57:11.891237  8952 sgd_solver.cpp:273] Snapshotting solver state to binary proto file examples/cifar10/cif_iter_500.solverstate\r\nI0520 22:57:11.892738  8952 solver.cpp:337] Iteration 500, Testing net (#0)\r\nI0520 22:57:12.583729  8952 solver.cpp:404]     Test net output #0: accuracy = 0.4375\r\nI0520 22:57:12.583729  8952 solver.cpp:404]     Test net output #1: loss = 1.51292 (* 1 = 1.51292 loss)\r\nI0520 22:57:12.594238  8952 solver.cpp:228] Iteration 500, loss = 1.59907\r\nI0520 22:57:12.594238  8952 solver.cpp:244]     Train net output #0: loss = 1.59907 (* 1 = 1.59907 loss)\r\nI0520 22:57:12.594238  8952 sgd_solver.cpp:106] Iteration 500, lr = 0.1\r\nI0520 22:57:15.189584  8952 solver.cpp:228] Iteration 600, loss = 1.40691\r\nI0520 22:57:15.189584  8952 solver.cpp:244]     Train net output #0: loss = 1.40691 (* 1 = 1.40691 loss)\r\nI0520 22:57:15.189584  8952 sgd_solver.cpp:106] Iteration 600, lr = 0.1\r\nI0520 22:57:17.790434  8952 solver.cpp:228] Iteration 700, loss = 1.48115\r\nI0520 22:57:17.790434  8952 solver.cpp:244]     Train net output #0: loss = 1.48115 (* 1 = 1.48115 loss)\r\nI0520 22:57:17.790434  8952 sgd_solver.cpp:106] Iteration 700, lr = 0.1\r\nI0520 22:57:20.370769  8952 solver.cpp:228] Iteration 800, loss = 1.35165\r\nI0520 22:57:20.370769  8952 solver.cpp:244]     Train net output #0: loss = 1.35165 (* 1 = 1.35165 loss)\r\nI0520 22:57:20.370769  8952 sgd_solver.cpp:106] Iteration 800, lr = 0.1\r\nI0520 22:57:22.956110  8952 solver.cpp:228] Iteration 900, loss = 1.20441\r\nI0520 22:57:22.956110  8952 solver.cpp:244]     Train net output #0: loss = 1.20441 (* 1 = 1.20441 loss)\r\nI0520 22:57:22.956110  8952 sgd_solver.cpp:106] Iteration 900, lr = 0.1\r\nI0520 22:57:25.603274  8952 solver.cpp:454] Snapshotting to binary proto file examples/cifar10/cif_iter_1000.caffemodel\r\nI0520 22:57:25.611498  8952 sgd_solver.cpp:273] Snapshotting solver state to binary proto file examples/cifar10/cif_iter_1000.solverstate\r\nI0520 22:57:25.612999  8952 solver.cpp:337] Iteration 1000, Testing net (#0)\r\nI0520 22:57:26.383548  8952 solver.cpp:404]     Test net output #0: accuracy = 0.5235\r\nI0520 22:57:26.383548  8952 solver.cpp:404]     Test net output #1: loss = 1.30647 (* 1 = 1.30647 loss)\r\nI0520 22:57:26.395056  8952 solver.cpp:228] Iteration 1000, loss = 1.24928\r\nI0520 22:57:26.395056  8952 solver.cpp:244]     Train net output #0: loss = 1.24928 (* 1 = 1.24928 loss)\r\nI0520 22:57:26.395056  8952 sgd_solver.cpp:106] Iteration 1000, lr = 0.1\r\nI0520 22:57:29.094476  8952 solver.cpp:228] Iteration 1100, loss = 1.15243\r\nI0520 22:57:29.094976  8952 solver.cpp:244]     Train net output #0: loss = 1.15243 (* 1 = 1.15243 loss)\r\n```\r\nlog2:\r\n```\r\nI0520 23:00:23.213491  9848 solver.cpp:280] Learning Rate Policy: multistep\r\nI0520 23:00:23.213491  9848 solver.cpp:337] Iteration 0, Testing net (#0)\r\nI0520 23:00:23.616646  9848 blocking_queue.cpp:50] Data layer prefetch queue empty\r\nI0520 23:00:23.618648 10420 blocking_queue.cpp:50] Waiting for data\r\nI0520 23:00:23.970897  9848 solver.cpp:404]     Test net output #0: accuracy = 0.0994001\r\nI0520 23:00:23.970897  9848 solver.cpp:404]     Test net output #1: loss = 78.6553 (* 1 = 78.6553 loss)\r\nI0520 23:00:24.019619  9848 solver.cpp:228] Iteration 0, loss = 2.37327\r\nI0520 23:00:24.019619  9848 solver.cpp:244]     Train net output #0: loss = 2.37327 (* 1 = 2.37327 loss)\r\nI0520 23:00:24.019619  9848 sgd_solver.cpp:106] Iteration 0, lr = 0.1\r\nI0520 23:00:26.558173  9848 solver.cpp:228] Iteration 100, loss = 1.79718\r\nI0520 23:00:26.558173  9848 solver.cpp:244]     Train net output #0: loss = 1.79718 (* 1 = 1.79718 loss)\r\nI0520 23:00:26.558173  9848 sgd_solver.cpp:106] Iteration 100, lr = 0.1\r\nI0520 23:00:29.119212  9848 solver.cpp:228] Iteration 200, loss = 1.79569\r\nI0520 23:00:29.119212  9848 solver.cpp:244]     Train net output #0: loss = 1.79569 (* 1 = 1.79569 loss)\r\nI0520 23:00:29.119212  9848 sgd_solver.cpp:106] Iteration 200, lr = 0.1\r\nI0520 23:00:31.674723  9848 solver.cpp:228] Iteration 300, loss = 1.62069\r\nI0520 23:00:31.674723  9848 solver.cpp:244]     Train net output #0: loss = 1.62069 (* 1 = 1.62069 loss)\r\nI0520 23:00:31.674723  9848 sgd_solver.cpp:106] Iteration 300, lr = 0.1\r\nI0520 23:00:34.230991  9848 solver.cpp:228] Iteration 400, loss = 1.42211\r\nI0520 23:00:34.230991  9848 solver.cpp:244]     Train net output #0: loss = 1.42211 (* 1 = 1.42211 loss)\r\nI0520 23:00:34.230991  9848 sgd_solver.cpp:106] Iteration 400, lr = 0.1\r\nI0520 23:00:36.764794  9848 solver.cpp:454] Snapshotting to binary proto file examples/cifar10/cif_iter_500.caffemodel\r\nI0520 23:00:36.780421  9848 sgd_solver.cpp:273] Snapshotting solver state to binary proto file examples/cifar10/cif_iter_500.solverstate\r\nI0520 23:00:36.780421  9848 solver.cpp:337] Iteration 500, Testing net (#0)\r\nI0520 23:00:37.483541  9848 solver.cpp:404]     Test net output #0: accuracy = 0.4521\r\nI0520 23:00:37.483541  9848 solver.cpp:404]     Test net output #1: loss = 1.48877 (* 1 = 1.48877 loss)\r\nI0520 23:00:37.499166  9848 solver.cpp:228] Iteration 500, loss = 1.59037\r\nI0520 23:00:37.499166  9848 solver.cpp:244]     Train net output #0: loss = 1.59037 (* 1 = 1.59037 loss)\r\nI0520 23:00:37.499166  9848 sgd_solver.cpp:106] Iteration 500, lr = 0.1\r\nI0520 23:00:40.057332  9848 solver.cpp:228] Iteration 600, loss = 1.29694\r\nI0520 23:00:40.057332  9848 solver.cpp:244]     Train net output #0: loss = 1.29694 (* 1 = 1.29694 loss)\r\nI0520 23:00:40.057332  9848 sgd_solver.cpp:106] Iteration 600, lr = 0.1\r\nI0520 23:00:42.600935  9848 solver.cpp:228] Iteration 700, loss = 1.42686\r\nI0520 23:00:42.600935  9848 solver.cpp:244]     Train net output #0: loss = 1.42686 (* 1 = 1.42686 loss)\r\nI0520 23:00:42.600935  9848 sgd_solver.cpp:106] Iteration 700, lr = 0.1\r\nI0520 23:00:45.152452  9848 solver.cpp:228] Iteration 800, loss = 1.38337\r\nI0520 23:00:45.152452  9848 solver.cpp:244]     Train net output #0: loss = 1.38337 (* 1 = 1.38337 loss)\r\nI0520 23:00:45.152452  9848 sgd_solver.cpp:106] Iteration 800, lr = 0.1\r\nI0520 23:00:47.699352  9848 solver.cpp:228] Iteration 900, loss = 1.06408\r\nI0520 23:00:47.699352  9848 solver.cpp:244]     Train net output #0: loss = 1.06408 (* 1 = 1.06408 loss)\r\nI0520 23:00:47.699352  9848 sgd_solver.cpp:106] Iteration 900, lr = 0.1\r\nI0520 23:00:50.236433  9848 solver.cpp:454] Snapshotting to binary proto file examples/cifar10/cif_iter_1000.caffemodel\r\nI0520 23:00:50.252058  9848 sgd_solver.cpp:273] Snapshotting solver state to binary proto file examples/cifar10/cif_iter_1000.solverstate\r\nI0520 23:00:50.252058  9848 solver.cpp:337] Iteration 1000, Testing net (#0)\r\nI0520 23:00:50.955193  9848 solver.cpp:404]     Test net output #0: accuracy = 0.5476\r\nI0520 23:00:50.955193  9848 solver.cpp:404]     Test net output #1: loss = 1.23861 (* 1 = 1.23861 loss)\r\nI0520 23:00:50.955193  9848 solver.cpp:228] Iteration 1000, loss = 1.27772\r\nI0520 23:00:50.955193  9848 solver.cpp:244]     Train net output #0: loss = 1.27772 (* 1 = 1.27772 loss)\r\nI0520 23:00:50.955193  9848 sgd_solver.cpp:106] Iteration 1000, lr = 0.1\r\n```"},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/221386107","html_url":"https://github.com/BVLC/caffe/issues/4189#issuecomment-221386107","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/4189","id":221386107,"user":{"login":"ajtulloch","id":1121581,"avatar_url":"https://avatars.githubusercontent.com/u/1121581?v=3","gravatar_id":"","url":"https://api.github.com/users/ajtulloch","html_url":"https://github.com/ajtulloch","followers_url":"https://api.github.com/users/ajtulloch/followers","following_url":"https://api.github.com/users/ajtulloch/following{/other_user}","gists_url":"https://api.github.com/users/ajtulloch/gists{/gist_id}","starred_url":"https://api.github.com/users/ajtulloch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajtulloch/subscriptions","organizations_url":"https://api.github.com/users/ajtulloch/orgs","repos_url":"https://api.github.com/users/ajtulloch/repos","events_url":"https://api.github.com/users/ajtulloch/events{/privacy}","received_events_url":"https://api.github.com/users/ajtulloch/received_events","type":"User","site_admin":false},"created_at":"2016-05-24T20:05:27Z","updated_at":"2016-05-24T20:05:27Z","body":"CuDNN max pooling and convolutions are nondeterministic (IIRC algo_t = 1 for for fwd/bwd_data/bwd_filter is guaranteed deterministic). This is documented in the PDF documentation distributed with CuDNN. You should switch to `engine: CAFFE` for all layers where you need determinism."}}
{"actor":{"display_login":"ajtulloch"}}