{"action":"created","issue":{"url":"https://api.github.com/repos/dnouri/skorch/issues/342","repository_url":"https://api.github.com/repos/dnouri/skorch","labels_url":"https://api.github.com/repos/dnouri/skorch/issues/342/labels{/name}","comments_url":"https://api.github.com/repos/dnouri/skorch/issues/342/comments","events_url":"https://api.github.com/repos/dnouri/skorch/issues/342/events","html_url":"https://github.com/dnouri/skorch/issues/342","id":359125953,"node_id":"MDU6SXNzdWUzNTkxMjU5NTM=","number":342,"title":"Introduce a parameter initialization stage","user":{"login":"ottonemo","id":29305596,"node_id":"MDQ6VXNlcjI5MzA1NTk2","avatar_url":"https://avatars2.githubusercontent.com/u/29305596?v=4","gravatar_id":"","url":"https://api.github.com/users/ottonemo","html_url":"https://github.com/ottonemo","followers_url":"https://api.github.com/users/ottonemo/followers","following_url":"https://api.github.com/users/ottonemo/following{/other_user}","gists_url":"https://api.github.com/users/ottonemo/gists{/gist_id}","starred_url":"https://api.github.com/users/ottonemo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ottonemo/subscriptions","organizations_url":"https://api.github.com/users/ottonemo/orgs","repos_url":"https://api.github.com/users/ottonemo/repos","events_url":"https://api.github.com/users/ottonemo/events{/privacy}","received_events_url":"https://api.github.com/users/ottonemo/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-09-11T16:31:41Z","updated_at":"2018-09-18T15:57:37Z","closed_at":"2018-09-18T15:57:37Z","author_association":"COLLABORATOR","body":"Since there is still discussion around how to properly and conveniently freeze parameters (#261, #337) I'd like to present a generic way of approaching this problem while avoiding callbacks (which may seem an unnatural choice for such an action, but that might also be personal preference) and opening up a number of modifications that were not easily possible.\r\n\r\nThe general idea is to introduce a parameter initialization stage that applies user-specified functions to certain parameters, e.g.:\r\n\r\n```python\r\nnet = Net(\r\n\tMyModule,\r\n\tparameter_map={\r\n\t\t'linear[0-9].weight': skorch.helper.freeze_param,\r\n\t\t'linear[0-9].bias': skorch.helper.freeze_param,\r\n\t\t'rnn*.h2h': torch.init.xavier_normal_,\r\n\t}\r\n\tparameter_map__linear0__weight=torch.init.xavier_normal_,\r\n)\r\n```\r\n\r\nthis approach has the following benefits:\r\n\r\n- works with both uninitialized and initialized modules\r\n- compatible with grid search (`parameter_map__linear0__weight` notation)\r\n- easily extensible to multiple optimizers (basically no change needed under the assumption of non-overlapping parameter sets between optimizers)\r\n- allows easy test of different initialization schemes\r\n- compatible with torch utility functions\r\n\r\nwhile there are a few benefits it has the drawback of introducing a new specialized parameter that needs documentation and introduces a new workflow besides callbacks.\r\n\r\nI'd like to hear your feedback.\r\n\r\nIdea sketch:\r\n```python\r\nclass _untrainable_marker: pass\r\n\r\ndef freeze_parameter(param):\r\n\tparam.requires_grad = False\r\n\treturn _untrainable_marker\r\n\r\ndef apply_parameter_map(self, map, named_parameters):\r\n\tfor name, param in named_parameters:\r\n\t\tfor map_name, map_fn in map.items():\r\n\t\t\tif not fnmatch.fnmatch(name, map_name):\r\n\t\t\t\tcontinue\r\n\t\t\tif map_fn(param) is _untrainable_marker:\r\n\t\t\t\tbreak\r\n\t\telse:\r\n\t\t\tcontinue\r\n\t\tyield name, param\r\n\r\nclass Net:\r\n\tdef __init__(self, parameter_map=None, **kwargs):\r\n\t\tself.parameter_map = self._collect_parameter_map(parameter_map, kwargs)\r\n\r\n\tdef collect_parameter_map(self, pmap, kwargs):\r\n\t\tfor k, fn in self.get_params_for('parameter_map', kwargs).items():\r\n\t\t\tpmap[k.replace('__', '.')] = fn\r\n\t\treturn pmap\r\n\r\n\tdef initialize_parameters(self):\r\n\t\tself.trainable_parameters_ = apply_parameter_map(self.parameter_map, self.module_.named_parameters())\r\n\r\n\tdef initialize_optimizer(self):\r\n\t\tself.optimizer_ = self.optimizer(self.trainable_parameters_)\r\n\r\n\tdef initialize(self):\r\n\t\tself.init_callbacks()\r\n\t\tself.init_module()\r\n\t\tself.init_parameters()\r\n\t\tself.init_optimizer()\r\n\r\nnet = Net(\r\n\tMyModule,\r\n\tparameter_map={\r\n\t\t'linear[0-9].weight': skorch.helper.freeze_param,\r\n\t\t'linear[0-9].bias': skorch.helper.freeze_param,\r\n\t\t'rnn*.h2h': torch.init.xavier_normal_,\r\n\t}\r\n\tparameter_map__linear0__weight=torch.init.xavier_normal_,\r\n)\r\n```"},"comment":{"url":"https://api.github.com/repos/dnouri/skorch/issues/comments/422450398","html_url":"https://github.com/dnouri/skorch/issues/342#issuecomment-422450398","issue_url":"https://api.github.com/repos/dnouri/skorch/issues/342","id":422450398,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMjQ1MDM5OA==","user":{"login":"ottonemo","id":29305596,"node_id":"MDQ6VXNlcjI5MzA1NTk2","avatar_url":"https://avatars2.githubusercontent.com/u/29305596?v=4","gravatar_id":"","url":"https://api.github.com/users/ottonemo","html_url":"https://github.com/ottonemo","followers_url":"https://api.github.com/users/ottonemo/followers","following_url":"https://api.github.com/users/ottonemo/following{/other_user}","gists_url":"https://api.github.com/users/ottonemo/gists{/gist_id}","starred_url":"https://api.github.com/users/ottonemo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ottonemo/subscriptions","organizations_url":"https://api.github.com/users/ottonemo/orgs","repos_url":"https://api.github.com/users/ottonemo/repos","events_url":"https://api.github.com/users/ottonemo/events{/privacy}","received_events_url":"https://api.github.com/users/ottonemo/received_events","type":"User","site_admin":false},"created_at":"2018-09-18T15:57:37Z","updated_at":"2018-09-18T15:57:37Z","author_association":"COLLABORATOR","body":"Since #343 is merged now, we have a way to (hopefully) conveniently freeze/unfreeze parameters and do all sorts of other things along the way e.g. initialize parameters, periodically freeze/unfreeze parameters, re-initialize under some condition, normalize parameters if their norm is getting our of hand, etc."}}
{"actor":{"display_login":"ottonemo"}}