{"action":"created","issue":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/570","labels_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/570/labels{/name}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/570/comments","events_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/570/events","html_url":"https://github.com/Lasagne/Lasagne/pull/570","id":125974586,"number":570,"title":"Add StandardizationLayer","user":{"login":"craffel","id":417568,"avatar_url":"https://avatars.githubusercontent.com/u/417568?v=3","gravatar_id":"","url":"https://api.github.com/users/craffel","html_url":"https://github.com/craffel","followers_url":"https://api.github.com/users/craffel/followers","following_url":"https://api.github.com/users/craffel/following{/other_user}","gists_url":"https://api.github.com/users/craffel/gists{/gist_id}","starred_url":"https://api.github.com/users/craffel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craffel/subscriptions","organizations_url":"https://api.github.com/users/craffel/orgs","repos_url":"https://api.github.com/users/craffel/repos","events_url":"https://api.github.com/users/craffel/events{/privacy}","received_events_url":"https://api.github.com/users/craffel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-01-11T15:55:13Z","updated_at":"2016-01-11T17:51:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Lasagne/Lasagne/pulls/570","html_url":"https://github.com/Lasagne/Lasagne/pull/570","diff_url":"https://github.com/Lasagne/Lasagne/pull/570.diff","patch_url":"https://github.com/Lasagne/Lasagne/pull/570.patch"},"body":"The motivation behind this layer is this: Usually, part of your ML pipeline is to compute some statistics on the training set (mean and standard deviation, or min and max) and then use them to normalize/standardize all inputs to the network.  Currently, this needs to be done \"outside of the model\", i.e. when you save your model you must also save the training set statistics to use later when passing data through the model.  There's no convenient way to store this information in the model itself.  This layer simply stores these statistics, and then applies them in `get_output` in a smart way which hopefully covers most use-cases.  See the example in the docstring for an example.\r\n\r\nAs another (personally relevant) example, say you are passing sequences of feature tensors into a convolutional network.  Your input data is of shape `(n_batch, n_filters, n_time_steps, n_feature)`.  You want to normalize over the `n_filters` and `n_feature` (feature) dimensions, not the `n_batch` or `n_time_steps` (sample) dimensions.  After reshaping/dimshuffling your training data (call it `X`) to shape `(n_batch*n_time_steps, n_filters, n_feature)`, you could compute the mean and standard deviation over these dimensions as `offset = X.mean(axis=0)`, `scale = X.std(axis=0)` and then apply them using the `StandardizationLayer` as follows:\r\n\r\n```Python\r\nl_in = lasagne.layers.InputLayer((None, n_filters, None, n_features))\r\nl_std = lasagne.layers.normalization.StandardizationLayer(\r\n    l_in, scale, offset, axis=(1, 3))\r\n```\r\n\r\nIf this makes sense and seems useful, I will create some unit tests for it and/or add/remove/change features."},"comment":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/comments/170632876","html_url":"https://github.com/Lasagne/Lasagne/pull/570#issuecomment-170632876","issue_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/570","id":170632876,"user":{"login":"craffel","id":417568,"avatar_url":"https://avatars.githubusercontent.com/u/417568?v=3","gravatar_id":"","url":"https://api.github.com/users/craffel","html_url":"https://github.com/craffel","followers_url":"https://api.github.com/users/craffel/followers","following_url":"https://api.github.com/users/craffel/following{/other_user}","gists_url":"https://api.github.com/users/craffel/gists{/gist_id}","starred_url":"https://api.github.com/users/craffel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craffel/subscriptions","organizations_url":"https://api.github.com/users/craffel/orgs","repos_url":"https://api.github.com/users/craffel/repos","events_url":"https://api.github.com/users/craffel/events{/privacy}","received_events_url":"https://api.github.com/users/craffel/received_events","type":"User","site_admin":false},"created_at":"2016-01-11T17:51:22Z","updated_at":"2016-01-11T17:51:22Z","body":"> No no, it doesn't. It was just a question of what dimensionality the parameters are defined as in the first place. My idea was that a BiasLayer on 4D input should always have a 4D bias tensor with broadcastable singleton dimensions for all shared axes, instead of creating, say, a 2D tensor and adding the correct broadcastable dimensions in get_output_for(). It does the latter now to be compatible with existing biases, but this also means it's got to have the somewhat complex dimshuffle pattern generator.\r\n\r\nI'll follow the existing convention, if it's something which we decide to change later on I'll leave it to then.\r\n\r\n> Do you want to add the ScaleLayer and standardize() macro (the latter could reuse a lot of your current docstring)? The ScaleLayer should probably go into layers/special.py after the BiasLayer, about standardize() I'm not sure.\r\n\r\nSure, I will do this now."}}