{"action":"created","issue":{"url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/1017","repository_url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit","labels_url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/1017/labels{/name}","comments_url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/1017/comments","events_url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/1017/events","html_url":"https://github.com/JohnLangford/vowpal_wabbit/issues/1017","id":161555583,"number":1017,"title":"--OjaNewton numerical instability and slowness","user":{"login":"arielf","id":159664,"avatar_url":"https://avatars.githubusercontent.com/u/159664?v=3","gravatar_id":"","url":"https://api.github.com/users/arielf","html_url":"https://github.com/arielf","followers_url":"https://api.github.com/users/arielf/followers","following_url":"https://api.github.com/users/arielf/following{/other_user}","gists_url":"https://api.github.com/users/arielf/gists{/gist_id}","starred_url":"https://api.github.com/users/arielf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arielf/subscriptions","organizations_url":"https://api.github.com/users/arielf/orgs","repos_url":"https://api.github.com/users/arielf/repos","events_url":"https://api.github.com/users/arielf/events{/privacy}","received_events_url":"https://api.github.com/users/arielf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-06-21T22:35:32Z","updated_at":"2016-07-25T02:41:26Z","closed_at":null,"body":"Tried to use the new `--OjaNewton` (2nd order learning) on `test/train-sets/0002.dat`\r\n\r\nObservations:\r\n\r\n1) With vanilla run (all defaults) results are surprisingly much worse than without it (final mean loss is 0.127483 vs 0.003847.  About ~30x greater).  Note that this data-set labels are all around 0.5.  Predictions with `--OjaNewton` are way off, e.g. rather than roughly converge on the 0.5 average, many (not all) of the examples cause `--OjaNewton` to predict either 0.0 or 1.0 and the mean loss to jump sharply upwards.\r\n\r\n2) Adding `-b 21` (or any greater value) makes `--OjaNewton` run many times slower (`-b 24` takes half a minute to complete one pass, about 100 times slower). At `-b 21` I see NaNs on all examples and final average loss more than doubling to 0.283062.  This is weird, why would changing `-b` affect the convergence?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/comments/234824328","html_url":"https://github.com/JohnLangford/vowpal_wabbit/issues/1017#issuecomment-234824328","issue_url":"https://api.github.com/repos/JohnLangford/vowpal_wabbit/issues/1017","id":234824328,"user":{"login":"haipengl","id":8459966,"avatar_url":"https://avatars.githubusercontent.com/u/8459966?v=3","gravatar_id":"","url":"https://api.github.com/users/haipengl","html_url":"https://github.com/haipengl","followers_url":"https://api.github.com/users/haipengl/followers","following_url":"https://api.github.com/users/haipengl/following{/other_user}","gists_url":"https://api.github.com/users/haipengl/gists{/gist_id}","starred_url":"https://api.github.com/users/haipengl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haipengl/subscriptions","organizations_url":"https://api.github.com/users/haipengl/orgs","repos_url":"https://api.github.com/users/haipengl/repos","events_url":"https://api.github.com/users/haipengl/events{/privacy}","received_events_url":"https://api.github.com/users/haipengl/received_events","type":"User","site_admin":false},"created_at":"2016-07-25T02:41:26Z","updated_at":"2016-07-25T02:41:26Z","body":"Sorry for the delay. It seems like the problem is in the learning rate. If you run OjaNewton with --alpha_inverse=0.1, you will get similar loss, although indeed it doesn't seem like second order information is helping here..."}}
{"actor":{"display_login":"haipengl"}}