{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/17257","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/17257/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/17257/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/17257/events","html_url":"https://github.com/tensorflow/tensorflow/issues/17257","id":300021191,"number":17257,"title":"Saver: Don't fail on restoring variables not present in a checkpoint.","user":{"login":"sleighsoft","id":9438971,"avatar_url":"https://avatars3.githubusercontent.com/u/9438971?v=4","gravatar_id":"","url":"https://api.github.com/users/sleighsoft","html_url":"https://github.com/sleighsoft","followers_url":"https://api.github.com/users/sleighsoft/followers","following_url":"https://api.github.com/users/sleighsoft/following{/other_user}","gists_url":"https://api.github.com/users/sleighsoft/gists{/gist_id}","starred_url":"https://api.github.com/users/sleighsoft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sleighsoft/subscriptions","organizations_url":"https://api.github.com/users/sleighsoft/orgs","repos_url":"https://api.github.com/users/sleighsoft/repos","events_url":"https://api.github.com/users/sleighsoft/events{/privacy}","received_events_url":"https://api.github.com/users/sleighsoft/received_events","type":"User","site_admin":false},"labels":[{"id":299643928,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:contributions%20welcome","name":"stat:contributions welcome","color":"009800","default":false},{"id":473173272,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:feature","name":"type:feature","color":"c2e0c6","default":false}],"state":"closed","locked":false,"assignee":{"login":"allenlavoie","id":3731025,"avatar_url":"https://avatars3.githubusercontent.com/u/3731025?v=4","gravatar_id":"","url":"https://api.github.com/users/allenlavoie","html_url":"https://github.com/allenlavoie","followers_url":"https://api.github.com/users/allenlavoie/followers","following_url":"https://api.github.com/users/allenlavoie/following{/other_user}","gists_url":"https://api.github.com/users/allenlavoie/gists{/gist_id}","starred_url":"https://api.github.com/users/allenlavoie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allenlavoie/subscriptions","organizations_url":"https://api.github.com/users/allenlavoie/orgs","repos_url":"https://api.github.com/users/allenlavoie/repos","events_url":"https://api.github.com/users/allenlavoie/events{/privacy}","received_events_url":"https://api.github.com/users/allenlavoie/received_events","type":"User","site_admin":false},"assignees":[{"login":"allenlavoie","id":3731025,"avatar_url":"https://avatars3.githubusercontent.com/u/3731025?v=4","gravatar_id":"","url":"https://api.github.com/users/allenlavoie","html_url":"https://github.com/allenlavoie","followers_url":"https://api.github.com/users/allenlavoie/followers","following_url":"https://api.github.com/users/allenlavoie/following{/other_user}","gists_url":"https://api.github.com/users/allenlavoie/gists{/gist_id}","starred_url":"https://api.github.com/users/allenlavoie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allenlavoie/subscriptions","organizations_url":"https://api.github.com/users/allenlavoie/orgs","repos_url":"https://api.github.com/users/allenlavoie/repos","events_url":"https://api.github.com/users/allenlavoie/events{/privacy}","received_events_url":"https://api.github.com/users/allenlavoie/received_events","type":"User","site_admin":false}],"milestone":null,"comments":18,"created_at":"2018-02-25T12:12:37Z","updated_at":"2018-05-11T09:28:46Z","closed_at":"2018-04-04T16:19:51Z","author_association":"CONTRIBUTOR","body":"This is a feature request.\r\n\r\nLet's consider a scenario where one trains multiple models and uses them in combination (like it is done in GANs).\r\n\r\nTo simplify the process of saving and restoring variables that are partly shared across the models (Pretraining Model, Training Model, Evaluation Model, Infer Model) one could instantiate the whole graph, containing all operations and variables, and save it.\r\n\r\nThen in order to do pre-training only the subset of graph elements that is required for pretraining is used.\r\n\r\nThis results in the overhead of having to build the whole model (which might consist of multiple sub models) the first time the model is run even though only a smaller part (e.g. Pretraining Model) is required.\r\n\r\nAnother issue arises when using different optimizers in different training stages (e.g. SGD first then Adam). As Adam creates additional variables Adam has to be instantiated during the first training stage so restoring from a checkpoint does not fail when restoring with Adam instead of SGD.\r\n\r\nThis restriction of having to build everything despite parts not being required results in more complicated code. If it would be possible to silently fail when a variable is not found in a checkpoint, so it can be initialized with `tf.global_variables_initializer()` instead, would allow better structuring of code.\r\n\r\nI have looked through all current issues regarding this problem and I have found a couple that face a similar problem and where a `QuietlyFailRestoringSaver` could solve this problem:\r\nhttps://github.com/tensorflow/tensorflow/issues/12032\r\nhttps://github.com/tensorflow/tensorflow/issues/16781\r\n\r\nI might consider building this if there is enough support for it. I am open for feedback."},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/388312273","html_url":"https://github.com/tensorflow/tensorflow/issues/17257#issuecomment-388312273","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/17257","id":388312273,"user":{"login":"sleighsoft","id":9438971,"avatar_url":"https://avatars3.githubusercontent.com/u/9438971?v=4","gravatar_id":"","url":"https://api.github.com/users/sleighsoft","html_url":"https://github.com/sleighsoft","followers_url":"https://api.github.com/users/sleighsoft/followers","following_url":"https://api.github.com/users/sleighsoft/following{/other_user}","gists_url":"https://api.github.com/users/sleighsoft/gists{/gist_id}","starred_url":"https://api.github.com/users/sleighsoft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sleighsoft/subscriptions","organizations_url":"https://api.github.com/users/sleighsoft/orgs","repos_url":"https://api.github.com/users/sleighsoft/repos","events_url":"https://api.github.com/users/sleighsoft/events{/privacy}","received_events_url":"https://api.github.com/users/sleighsoft/received_events","type":"User","site_admin":false},"created_at":"2018-05-11T09:28:46Z","updated_at":"2018-05-11T09:28:46Z","author_association":"CONTRIBUTOR","body":"Small example:\r\n```python\r\nimport tensorflow as tf\r\nimport tensorflow.contrib.eager as tfe\r\nfrom tensorflow.python.training import checkpointable\r\nfrom tensorflow.contrib.eager.python import checkpointable_utils\r\n\r\n\r\nclass MyModel(checkpointable.Checkpointable):\r\n\r\n  def __init__(self):\r\n    self.optimizer = tf.train.GradientDescentOptimizer(0.1)\r\n    self.var = tf.get_variable('var', [])\r\n\r\n  def optimizer_to_adam(self):\r\n    self.optimizer = tf.train.AdamOptimizer()\r\n\r\n\r\ncheckpoint_path = 'modeldir/test/'\r\n\r\nwith tf.Graph().as_default():\r\n  model = MyModel()\r\n  checkpoint = tfe.Checkpoint(model=model)\r\n  train_op = model.optimizer.minimize(model.var * model.var)\r\n  print(checkpointable_utils._serialize_object_graph(checkpoint))\r\n  session = tf.Session()\r\n  session.run(tf.global_variables_initializer())\r\n  checkpoint.save(checkpoint_path, session)\r\n  session.run(train_op)\r\n  print('Finished Model with SGD')\r\n\r\nwith tf.Graph().as_default():\r\n  model = MyModel()\r\n  model.optimizer_to_adam()\r\n  latest = tf.train.latest_checkpoint(checkpoint_path)\r\n  train_op = model.optimizer.minimize(model.var * model.var)\r\n  checkpoint = tfe.Checkpoint(model=model)\r\n  print(checkpointable_utils._serialize_object_graph(checkpoint))\r\n  session = tf.Session()\r\n  checkpoint.restore(latest).initialize_or_restore(session)\r\n  session.run(train_op)\r\n  print('Finished Model with Adam')\r\n```\r\n\r\nI would expect that the second graph (`Finished Model with Adam`) works.\r\nBut it throws `Attempting to use uninitialized value beta2_power`.\r\nI expected the `.initialize_or_restore` to initialize the `Adam` variables but that does not happen."}}
{"actor":{"display_login":"sleighsoft"}}