{"action":"opened","issue":{"url":"https://api.github.com/repos/AtheMathmo/rusty-machine/issues/67","repository_url":"https://api.github.com/repos/AtheMathmo/rusty-machine","labels_url":"https://api.github.com/repos/AtheMathmo/rusty-machine/issues/67/labels{/name}","comments_url":"https://api.github.com/repos/AtheMathmo/rusty-machine/issues/67/comments","events_url":"https://api.github.com/repos/AtheMathmo/rusty-machine/issues/67/events","html_url":"https://github.com/AtheMathmo/rusty-machine/issues/67","id":153301840,"number":67,"title":"Model Regularization (in neural networks)","user":{"login":"AtheMathmo","id":10150986,"avatar_url":"https://avatars.githubusercontent.com/u/10150986?v=3","gravatar_id":"","url":"https://api.github.com/users/AtheMathmo","html_url":"https://github.com/AtheMathmo","followers_url":"https://api.github.com/users/AtheMathmo/followers","following_url":"https://api.github.com/users/AtheMathmo/following{/other_user}","gists_url":"https://api.github.com/users/AtheMathmo/gists{/gist_id}","starred_url":"https://api.github.com/users/AtheMathmo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AtheMathmo/subscriptions","organizations_url":"https://api.github.com/users/AtheMathmo/orgs","repos_url":"https://api.github.com/users/AtheMathmo/repos","events_url":"https://api.github.com/users/AtheMathmo/events{/privacy}","received_events_url":"https://api.github.com/users/AtheMathmo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-05-05T19:10:15Z","updated_at":"2016-05-05T19:10:15Z","closed_at":null,"body":"### The problem\r\n\r\nWe need regularization in the relevant models, especially neural networks.\r\n\r\nThe initial problem discovery is described in issue #61, briefly: unstable neural network weights. In addition to the solution described in #61 this could also be improved by adding Regularization.\r\n\r\n### The solution\r\n\r\nWe should add some basic regularization to the neural network models. We would like this regularization to be general enough that it can be reused somewhat in other models. Here is one possible implementation we can use as a starting point.\r\n\r\nIn learning/toolkit/regularization.rs\r\n\r\n```rust\r\n/// Model regularization\r\npub enum Regularization {\r\n    /// L1 Regularization\r\n    L1(f64),\r\n    /// L2 Regularization\r\n    L2(f64),\r\n    /// L1 AND L2 Regularization\r\n    ElasticNet(f64, f64),\r\n}\r\n```\r\n\r\n_Note that the above would only support `f64` - this is the same as the models for now and will be changed in the future._\r\n\r\nThen for neural networks we can allow the Criterion to have a regularization field. This will allow us to modify the cost and gradient computations to utilize the regularization.\r\n\r\n#### Note\r\n\r\nWe may have some additional work on our hands if the existing code is not generic enough for us to include these changes. For example, the bias term weights should not be regularized. However, this shouldn't pose a significant problem.\r\n\r\n### Criteria for success\r\n\r\n- All existing tests should pass.\r\n- New tests for regularization should pass.\r\n- Should be able to verify that the neural network is behaving as expected with regularization. This will be documented in this issue.\r\n"}}