{"action":"created","issue":{"url":"https://api.github.com/repos/mikegashler/waffles/issues/33","repository_url":"https://api.github.com/repos/mikegashler/waffles","labels_url":"https://api.github.com/repos/mikegashler/waffles/issues/33/labels{/name}","comments_url":"https://api.github.com/repos/mikegashler/waffles/issues/33/comments","events_url":"https://api.github.com/repos/mikegashler/waffles/issues/33/events","html_url":"https://github.com/mikegashler/waffles/issues/33","id":186177520,"number":33,"title":"uninitialized values","user":{"login":"mikegashler","id":1940715,"avatar_url":"https://avatars.githubusercontent.com/u/1940715?v=3","gravatar_id":"","url":"https://api.github.com/users/mikegashler","html_url":"https://github.com/mikegashler","followers_url":"https://api.github.com/users/mikegashler/followers","following_url":"https://api.github.com/users/mikegashler/following{/other_user}","gists_url":"https://api.github.com/users/mikegashler/gists{/gist_id}","starred_url":"https://api.github.com/users/mikegashler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikegashler/subscriptions","organizations_url":"https://api.github.com/users/mikegashler/orgs","repos_url":"https://api.github.com/users/mikegashler/repos","events_url":"https://api.github.com/users/mikegashler/events{/privacy}","received_events_url":"https://api.github.com/users/mikegashler/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-10-31T00:39:13Z","updated_at":"2016-11-01T15:09:20Z","closed_at":null,"body":"It looks like there are some uninitialized values somewhere causing semi-nondeterministic behavior with the new neural net optimizer code. The NeuralDecomposition test sometimes passes and sometimes fails with a floating point exception, depending on which tests are run before it. Here's a call stack:\n# 1 GClasses::GAssertFailed() at /home/mike/waffles/src/GClasses/GError.cpp:149\n# 2 GClasses::GLayerClassic::feedForward() at /home/mike/waffles/src/GClasses/GLayer.cpp:496\n# 3 GClasses::GNeuralNetLayer::feedForward() at /home/mike/waffles/src/GClasses/GLayer.h:93\n# 4 GClasses::GNeuralNet::forwardProp() at /home/mike/waffles/src/GClasses/GNeuralNet.cpp:401\n# 5 GClasses::GNeuralNet::predict() at /home/mike/waffles/src/GClasses/GNeuralNet.cpp:423\n# 6 GClasses::GNeuralNetFunction::evaluate() at /home/mike/waffles/src/GClasses/GOptimizer.cpp:85\n# 7 GClasses::GSGDOptimizer::updateDeltas() at /home/mike/waffles/src/GClasses/GOptimizer.cpp:279\n# 8 GClasses::GDifferentiableOptimizer::optimizeIncremental() at /home/mike/waffles/src/GClasses/GOptimizer.cpp:146\n# 9 GClasses::GNeuralDecomposition::trainIncremental() at /home/mike/waffles/src/GClasses/GNeuralDecomposition.cpp:303\n# 10 GClasses::GNeuralDecomposition::trainInner() at /home/mike/waffles/src/GClasses/GNeuralDecomposition.cpp:196\n# 11 GClasses::GSupervisedLearner::train() at /home/mike/waffles/src/GClasses/GLearner.cpp:478\n# 12 GClasses::GNeuralDecomposition::trainOnSeries() at /home/mike/waffles/src/GClasses/GNeuralDecomposition.cpp:68\n# 13 GClasses::GNeuralDecomposition::test() at /home/mike/waffles/src/GClasses/GNeuralDecomposition.cpp:335\n# 14 GTestHarness::runTest() at /home/mike/waffles/src/test/main.cpp:863\n# 15 GTestHarness::runAllTests() at /home/mike/waffles/src/test/main.cpp:950\n# 16 main() at /home/mike/waffles/src/test/main.cpp:1046\n\nAnd when I run this test in Valgrind, it reports lots of issues like this:\n\n==10133== Conditional jump or move depends on uninitialised value(s)\n==10133==    at 0x4C4AB1: GClasses::GLayerClassic::feedForward(GClasses::GVec const&) (GLayer.cpp:482)\n==10133==    by 0x4CA9AE: GClasses::GLayerMixed::feedForward(GClasses::GVec const&) (GLayer.cpp:1419)\n==10133==    by 0x5465B1: GClasses::GNeuralNet::forwardProp(GClasses::GVec const&, unsigned long) (GNeuralNet.cpp:396)\n==10133==    by 0x5467E5: GClasses::GNeuralNet::predict(GClasses::GVec const&, GClasses::GVec&) (GNeuralNet.cpp:423)\n==10133==    by 0x551B21: GClasses::GNeuralNetFunction::evaluate(GClasses::GVec const&, GClasses::GVec&) (GOptimizer.cpp:85)\n==10133==    by 0x55305F: GClasses::GSGDOptimizer::updateDeltas(GClasses::GVec const&, GClasses::GVec const&) (GOptimizer.cpp:279)\n==10133==    by 0x5521B8: GClasses::GDifferentiableOptimizer::optimizeIncremental(GClasses::GVec const&, GClasses::GVec const&) (GOptimizer.cpp:146)\n==10133==    by 0x543560: GClasses::GNeuralDecomposition::trainIncremental(GClasses::GVec const&, GClasses::GVec const&) (GNeuralDecomposition.cpp:303)\n==10133==    by 0x5429E0: GClasses::GNeuralDecomposition::trainInner(GClasses::GMatrix const&, GClasses::GMatrix const&) (GNeuralDecomposition.cpp:196)\n==10133==    by 0x4D76D9: GClasses::GSupervisedLearner::train(GClasses::GMatrix const&, GClasses::GMatrix const&) (GLearner.cpp:478)\n==10133==    by 0x541C7C: GClasses::GNeuralDecomposition::trainOnSeries(GClasses::GMatrix const&) (GNeuralDecomposition.cpp:68)\n==10133==    by 0x5438D9: GClasses::GNeuralDecomposition::test() (GNeuralDecomposition.cpp:335)\n"},"comment":{"url":"https://api.github.com/repos/mikegashler/waffles/issues/comments/257591725","html_url":"https://github.com/mikegashler/waffles/issues/33#issuecomment-257591725","issue_url":"https://api.github.com/repos/mikegashler/waffles/issues/33","id":257591725,"user":{"login":"mikegashler","id":1940715,"avatar_url":"https://avatars.githubusercontent.com/u/1940715?v=3","gravatar_id":"","url":"https://api.github.com/users/mikegashler","html_url":"https://github.com/mikegashler","followers_url":"https://api.github.com/users/mikegashler/followers","following_url":"https://api.github.com/users/mikegashler/following{/other_user}","gists_url":"https://api.github.com/users/mikegashler/gists{/gist_id}","starred_url":"https://api.github.com/users/mikegashler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikegashler/subscriptions","organizations_url":"https://api.github.com/users/mikegashler/orgs","repos_url":"https://api.github.com/users/mikegashler/repos","events_url":"https://api.github.com/users/mikegashler/events{/privacy}","received_events_url":"https://api.github.com/users/mikegashler/received_events","type":"User","site_admin":false},"created_at":"2016-11-01T15:09:20Z","updated_at":"2016-11-01T15:09:20Z","body":"I narrowed it down. The problem goes away if you add\r\n\tm_out.fill(0.0);\r\nin GLayerClassic::resize. Does that mean neural decomposition begins training without initializing one of its layers? Could we add a check somewhere, at least in debug mode, to catch this issue?"}}
{"actor":{"display_login":"mikegashler"}}