{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19903","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19903/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19903/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19903/events","html_url":"https://github.com/tensorflow/tensorflow/issues/19903","id":331092939,"node_id":"MDU6SXNzdWUzMzEwOTI5Mzk=","number":19903,"title":"Warm-starting moving averages (batch norm) as part of an estimator not possible","user":{"login":"jonasrauber","id":5837385,"node_id":"MDQ6VXNlcjU4MzczODU=","avatar_url":"https://avatars1.githubusercontent.com/u/5837385?v=4","gravatar_id":"","url":"https://api.github.com/users/jonasrauber","html_url":"https://github.com/jonasrauber","followers_url":"https://api.github.com/users/jonasrauber/followers","following_url":"https://api.github.com/users/jonasrauber/following{/other_user}","gists_url":"https://api.github.com/users/jonasrauber/gists{/gist_id}","starred_url":"https://api.github.com/users/jonasrauber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonasrauber/subscriptions","organizations_url":"https://api.github.com/users/jonasrauber/orgs","repos_url":"https://api.github.com/users/jonasrauber/repos","events_url":"https://api.github.com/users/jonasrauber/events{/privacy}","received_events_url":"https://api.github.com/users/jonasrauber/received_events","type":"User","site_admin":false},"labels":[{"id":386191887,"node_id":"MDU6TGFiZWwzODYxOTE4ODc=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20response","name":"stat:awaiting response","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":{"login":"ispirmustafa","id":19293677,"node_id":"MDQ6VXNlcjE5MjkzNjc3","avatar_url":"https://avatars1.githubusercontent.com/u/19293677?v=4","gravatar_id":"","url":"https://api.github.com/users/ispirmustafa","html_url":"https://github.com/ispirmustafa","followers_url":"https://api.github.com/users/ispirmustafa/followers","following_url":"https://api.github.com/users/ispirmustafa/following{/other_user}","gists_url":"https://api.github.com/users/ispirmustafa/gists{/gist_id}","starred_url":"https://api.github.com/users/ispirmustafa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ispirmustafa/subscriptions","organizations_url":"https://api.github.com/users/ispirmustafa/orgs","repos_url":"https://api.github.com/users/ispirmustafa/repos","events_url":"https://api.github.com/users/ispirmustafa/events{/privacy}","received_events_url":"https://api.github.com/users/ispirmustafa/received_events","type":"User","site_admin":false},"assignees":[{"login":"ispirmustafa","id":19293677,"node_id":"MDQ6VXNlcjE5MjkzNjc3","avatar_url":"https://avatars1.githubusercontent.com/u/19293677?v=4","gravatar_id":"","url":"https://api.github.com/users/ispirmustafa","html_url":"https://github.com/ispirmustafa","followers_url":"https://api.github.com/users/ispirmustafa/followers","following_url":"https://api.github.com/users/ispirmustafa/following{/other_user}","gists_url":"https://api.github.com/users/ispirmustafa/gists{/gist_id}","starred_url":"https://api.github.com/users/ispirmustafa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ispirmustafa/subscriptions","organizations_url":"https://api.github.com/users/ispirmustafa/orgs","repos_url":"https://api.github.com/users/ispirmustafa/repos","events_url":"https://api.github.com/users/ispirmustafa/events{/privacy}","received_events_url":"https://api.github.com/users/ispirmustafa/received_events","type":"User","site_admin":false},{"login":"eddie-zhou","id":35045937,"node_id":"MDQ6VXNlcjM1MDQ1OTM3","avatar_url":"https://avatars3.githubusercontent.com/u/35045937?v=4","gravatar_id":"","url":"https://api.github.com/users/eddie-zhou","html_url":"https://github.com/eddie-zhou","followers_url":"https://api.github.com/users/eddie-zhou/followers","following_url":"https://api.github.com/users/eddie-zhou/following{/other_user}","gists_url":"https://api.github.com/users/eddie-zhou/gists{/gist_id}","starred_url":"https://api.github.com/users/eddie-zhou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eddie-zhou/subscriptions","organizations_url":"https://api.github.com/users/eddie-zhou/orgs","repos_url":"https://api.github.com/users/eddie-zhou/repos","events_url":"https://api.github.com/users/eddie-zhou/events{/privacy}","received_events_url":"https://api.github.com/users/eddie-zhou/received_events","type":"User","site_admin":false}],"milestone":null,"comments":13,"created_at":"2018-06-11T08:34:56Z","updated_at":"2018-09-25T16:11:56Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I want to use a `tf.estimators.Estimator` to fine-tune a model that contains batch normalization (e.g. ResNet). To initialize the model, I use the new `WarmStartSettings` and pass it to the estimator's `warm_start_from` argument. Unfortunately, this will only warm-start trainable variables, and the `moving_mean` and `moving_variance` created by the batch normalization layer are not part of the trainable variables collection. Thus, the moving averages will not be warm-started.\r\n\r\nBecause of this problem, TensorFlow 1.9 (rc0) introduced the possibility to pass a list of variables to the `WarmStartSettings`. Unfortunately, when using estimators, this does not help because the variables are recreated all the time and not know at the position where the warm start settings have to be defined.\r\n\r\nA possible solution might be to make it possible to pass a function to the `warm_start_from` argument of estimators that has access to the current graph and returns a `WarmStartSettings` object.\r\n\r\nMy current workaround is to specify a list of variable names rather than variables in the `WarmStartSettings`, but this is not exactly how it's supposed to be and comes with its own problems (e.g. getting the list of variable names before the model was built; I just use all the variable names that are saved in the checkpoint and exlude e.g. `global_step`, but this is problematic because it circumvents certain checks and assertions).\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/424406781","html_url":"https://github.com/tensorflow/tensorflow/issues/19903#issuecomment-424406781","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19903","id":424406781,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNDQwNjc4MQ==","user":{"login":"eddie-zhou","id":35045937,"node_id":"MDQ6VXNlcjM1MDQ1OTM3","avatar_url":"https://avatars3.githubusercontent.com/u/35045937?v=4","gravatar_id":"","url":"https://api.github.com/users/eddie-zhou","html_url":"https://github.com/eddie-zhou","followers_url":"https://api.github.com/users/eddie-zhou/followers","following_url":"https://api.github.com/users/eddie-zhou/following{/other_user}","gists_url":"https://api.github.com/users/eddie-zhou/gists{/gist_id}","starred_url":"https://api.github.com/users/eddie-zhou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eddie-zhou/subscriptions","organizations_url":"https://api.github.com/users/eddie-zhou/orgs","repos_url":"https://api.github.com/users/eddie-zhou/repos","events_url":"https://api.github.com/users/eddie-zhou/events{/privacy}","received_events_url":"https://api.github.com/users/eddie-zhou/received_events","type":"User","site_admin":false},"created_at":"2018-09-25T16:11:56Z","updated_at":"2018-09-25T16:11:56Z","author_association":"CONTRIBUTOR","body":"Thanks for your thoughts, @jonasrauber!  I do agree that it can be useful in many settings, but by not being prescriptive in the API, we intentionally force users to explicitly think about exactly what variables they want to warm-start, and whether or not it is appropriate for their task.  Does that make sense?"}}
{"actor":{"display_login":"eddie-zhou"}}