{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21405","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21405/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21405/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21405/events","html_url":"https://github.com/tensorflow/tensorflow/issues/21405","id":347873634,"node_id":"MDU6SXNzdWUzNDc4NzM2MzQ=","number":21405,"title":"Tensorflow Moving Average Optimizer and Distribution Strategy","user":{"login":"iliTheFallen","id":8805234,"node_id":"MDQ6VXNlcjg4MDUyMzQ=","avatar_url":"https://avatars1.githubusercontent.com/u/8805234?v=4","gravatar_id":"","url":"https://api.github.com/users/iliTheFallen","html_url":"https://github.com/iliTheFallen","followers_url":"https://api.github.com/users/iliTheFallen/followers","following_url":"https://api.github.com/users/iliTheFallen/following{/other_user}","gists_url":"https://api.github.com/users/iliTheFallen/gists{/gist_id}","starred_url":"https://api.github.com/users/iliTheFallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iliTheFallen/subscriptions","organizations_url":"https://api.github.com/users/iliTheFallen/orgs","repos_url":"https://api.github.com/users/iliTheFallen/repos","events_url":"https://api.github.com/users/iliTheFallen/events{/privacy}","received_events_url":"https://api.github.com/users/iliTheFallen/received_events","type":"User","site_admin":false},"labels":[{"id":996845227,"node_id":"MDU6TGFiZWw5OTY4NDUyMjc=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/comp:dist-strat","name":"comp:dist-strat","color":"0052cc","default":false}],"state":"open","locked":false,"assignee":{"login":"josh11b","id":15258583,"node_id":"MDQ6VXNlcjE1MjU4NTgz","avatar_url":"https://avatars0.githubusercontent.com/u/15258583?v=4","gravatar_id":"","url":"https://api.github.com/users/josh11b","html_url":"https://github.com/josh11b","followers_url":"https://api.github.com/users/josh11b/followers","following_url":"https://api.github.com/users/josh11b/following{/other_user}","gists_url":"https://api.github.com/users/josh11b/gists{/gist_id}","starred_url":"https://api.github.com/users/josh11b/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josh11b/subscriptions","organizations_url":"https://api.github.com/users/josh11b/orgs","repos_url":"https://api.github.com/users/josh11b/repos","events_url":"https://api.github.com/users/josh11b/events{/privacy}","received_events_url":"https://api.github.com/users/josh11b/received_events","type":"User","site_admin":false},"assignees":[{"login":"josh11b","id":15258583,"node_id":"MDQ6VXNlcjE1MjU4NTgz","avatar_url":"https://avatars0.githubusercontent.com/u/15258583?v=4","gravatar_id":"","url":"https://api.github.com/users/josh11b","html_url":"https://github.com/josh11b","followers_url":"https://api.github.com/users/josh11b/followers","following_url":"https://api.github.com/users/josh11b/following{/other_user}","gists_url":"https://api.github.com/users/josh11b/gists{/gist_id}","starred_url":"https://api.github.com/users/josh11b/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josh11b/subscriptions","organizations_url":"https://api.github.com/users/josh11b/orgs","repos_url":"https://api.github.com/users/josh11b/repos","events_url":"https://api.github.com/users/josh11b/events{/privacy}","received_events_url":"https://api.github.com/users/josh11b/received_events","type":"User","site_admin":false},{"login":"anj-s","id":32556631,"node_id":"MDQ6VXNlcjMyNTU2NjMx","avatar_url":"https://avatars1.githubusercontent.com/u/32556631?v=4","gravatar_id":"","url":"https://api.github.com/users/anj-s","html_url":"https://github.com/anj-s","followers_url":"https://api.github.com/users/anj-s/followers","following_url":"https://api.github.com/users/anj-s/following{/other_user}","gists_url":"https://api.github.com/users/anj-s/gists{/gist_id}","starred_url":"https://api.github.com/users/anj-s/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anj-s/subscriptions","organizations_url":"https://api.github.com/users/anj-s/orgs","repos_url":"https://api.github.com/users/anj-s/repos","events_url":"https://api.github.com/users/anj-s/events{/privacy}","received_events_url":"https://api.github.com/users/anj-s/received_events","type":"User","site_admin":false}],"milestone":null,"comments":10,"created_at":"2018-08-06T10:49:17Z","updated_at":"2018-10-05T00:42:31Z","closed_at":null,"author_association":"NONE","body":"- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: No\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: 16.04.4 LTS (Xenial Xerus)\r\n- **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device**: No\r\n- **TensorFlow installed from (source or binary)**: From source\r\n- **TensorFlow version (use command below)**: 1.9.0\r\n- **Python version**: 3.5\r\n- **Bazel version (if compiling from source)**: 0.11.0\r\n- **GCC/Compiler version (if compiling from source)**: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609\r\n- **CUDA/cuDNN version**: 7.1.4.18\r\n- **GPU model and memory**: Quadro K620 and Tesla K40c -- 2GB and 11.5GB respectively.\r\n- **Exact command to reproduce**:\r\n```\r\nwith tf.Graph().as_default() as graph:\r\n    config = tf.estimator.RunConfig(\r\n        model_dir=\"./output\",\r\n        save_summary_steps=FLAGS.saving_summ_freq, \r\n        save_checkpoints_steps=FLAGS.saving_ckpt_freq,\r\n        keep_checkpoint_max=3,\r\n        train_distribute=tf.contrib.distribute.MirroredStrategy()\r\n    )\r\n    classifier = tf.estimator.Estimator(\r\n        model_fn, \r\n        config=config)\r\n    train_spec = tf.estimator.TrainSpec(gen_input_fn('train', FLAGS.num_epochs))\r\n    valid_spec = tf.estimator.EvalSpec(gen_input_fn('valid', 1))\r\n    tf.estimator.train_and_evaluate(classifier, train_spec, valid_spec)\r\n```\r\nHello Tensorflow Devs,\r\n\r\nI try to add exponential moving average support to the optimization step. However, this new Estimator API backed by the \"Mirrored Distrubution Strategy\" fails due to a tensor conversion method specific to this strategy.\r\n\r\nWhen I call ema.apply_gradients(...) it ends up with the following exception:\r\n```\r\n\r\nINFO:tensorflow:Using config: {'_model_dir': './output', 1    365       saving_listeners = _check_listeners_type(saving_listeners)\r\n--> 366       loss = self._train_model(input_fn, hooks, saving_listeners)\r\n    367       logging.info('Loss for final step: %s.', loss)\r\n    368       return self\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py in _train_model(self, input_fn, hooks, saving_listeners)\r\n   1115   def _train_model(self, input_fn, hooks, saving_listeners):\r\n   1116     if self._distribution:\r\n-> 1117       return self._train_model_distributed(input_fn, hooks, saving_listeners)\r\n   1118     else:\r\n   1119       return self._train_model_default(input_fn, hooks, saving_listeners)\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py in _train_model_distributed(self, input_fn, hooks, saving_listeners)\r\n   1158             labels,  # although this will be None it seems\r\n   1159             model_fn_lib.ModeKeys.TRAIN,\r\n-> 1160             self.config)\r\n   1161 \r\n   1162         # TODO(anjalisridhar): Figure out how to resolve the following scaffold\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/distribute.py in call_for_each_tower(self, fn, *args, **kwargs)\r\n    792     \"\"\"\r\n    793     _require_cross_tower_context(self)\r\n--> 794     return self._call_for_each_tower(fn, *args, **kwargs)\r\n    795 \r\n    796   def _call_for_each_tower(self, fn, *args, **kwargs):\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py in _call_for_each_tower(self, fn, *args, **kwargs)\r\n    267       for t in threads:\r\n    268         t.should_run.set()\r\n--> 269       coord.join(threads)\r\n    270 \r\n    271     return values.regroup({t.device: t.main_result for t in threads})\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/coordinator.py in join(self, threads, stop_grace_period_secs, ignore_live_threads)\r\n    387       self._registered_threads = set()\r\n    388       if self._exc_info_to_raise:\r\n--> 389         six.reraise(*self._exc_info_to_raise)\r\n    390       elif stragglers:\r\n    391         if ignore_live_threads:\r\n\r\n/usr/local/lib/python3.5/dist-packages/six.py in reraise(tp, value, tb)\r\n    691             if value.__traceback__ is not tb:\r\n    692                 raise value.with_traceback(tb)\r\n--> 693             raise value\r\n    694         finally:\r\n    695             value = None\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/coordinator.py in stop_on_exception(self)\r\n    295     \"\"\"\r\n    296     try:\r\n--> 297       yield\r\n    298     except:  # pylint: disable=bare-except\r\n    299       self.request_stop(ex=sys.exc_info())\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py in run(self)\r\n    477                 self._captured_var_scope, reuse=self.tower_id > 0), \\\r\n    478             variable_scope.variable_creator_scope(self.variable_creator_fn):\r\n--> 479           self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)\r\n    480           self.done = True\r\n    481       finally:\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py in _call_model_fn(self, features, labels, mode, config)\r\n   1105 \r\n   1106     logging.info('Calling model_fn.')\r\n-> 1107     model_fn_results = self._model_fn(features=features, **kwargs)\r\n   1108     logging.info('Done calling model_fn.')\r\n   1109 \r\n\r\n<ipython-input-9-2239e101f763> in model_fn(features, labels, mode)\r\n      3     loss = tfsi_model(features)\r\n      4     if mode == tf.estimator.ModeKeys.TRAIN:\r\n----> 5         train_op, grads, saver = minimize(loss)\r\n      6         writer, merged = prepare_summary(tf.get_default_graph(), loss, grads)\r\n      7         chkpt_hook = tf.train.CheckpointSaverHook(\r\n\r\n<ipython-input-7-8dbd2a0df6d6> in minimize(loss)\r\n     17         train_op = ema.apply_gradients(\r\n     18             grads,\r\n---> 19             global_step=tf.train.get_or_create_global_step()\r\n     20         )\r\n     21         return train_op, grads, ema.swapping_saver()\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/opt/python/training/moving_average_optimizer.py in apply_gradients(self, grads_and_vars, global_step, name)\r\n     97     if self._sequential_update:\r\n     98       with ops.control_dependencies([train_op]):\r\n---> 99         ma_op = self._ema.apply(var_list)\r\n    100     else:\r\n    101       ma_op = self._ema.apply(var_list)\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/moving_averages.py in apply(self, var_list)\r\n    428         zero_debias = self._averages[var] in zero_debias_true\r\n    429         updates.append(assign_moving_average(\r\n--> 430             self._averages[var], var, decay, zero_debias=zero_debias))\r\n    431       return control_flow_ops.group(*updates, name=scope)\r\n    432 \r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/moving_averages.py in assign_moving_average(variable, value, decay, zero_debias, name)\r\n     82   with ops.name_scope(name, \"AssignMovingAvg\",\r\n     83                       [variable, value, decay]) as scope:\r\n---> 84     with ops.colocate_with(variable):\r\n     85       decay = ops.convert_to_tensor(1.0 - decay, name=\"decay\")\r\n     86       if decay.dtype != variable.dtype.base_dtype:\r\n\r\n/usr/lib/python3.5/contextlib.py in __enter__(self)\r\n     57     def __enter__(self):\r\n     58         try:\r\n---> 59             return next(self.gen)\r\n     60         except StopIteration:\r\n     61             raise RuntimeError(\"generator didn't yield\") from None\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py in _colocate_with_for_gradient(self, op, gradient_uid, ignore_existing)\r\n   4217   def _colocate_with_for_gradient(self, op, gradient_uid,\r\n   4218                                   ignore_existing=False):\r\n-> 4219     with self.colocate_with(op, ignore_existing):\r\n   4220       if gradient_uid is not None and self._control_flow_context is not None:\r\n   4221         try:\r\n\r\n/usr/lib/python3.5/contextlib.py in __enter__(self)\r\n     57     def __enter__(self):\r\n     58         try:\r\n---> 59             return next(self.gen)\r\n     60         except StopIteration:\r\n     61             raise RuntimeError(\"generator didn't yield\") from None\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py in colocate_with(self, op, ignore_existing)\r\n   4270     if op is not None and not isinstance(op, Operation):\r\n   4271       # We always want to colocate with the reference op.\r\n-> 4272       op = internal_convert_to_tensor_or_indexed_slices(op, as_ref=True).op\r\n   4273 \r\n   4274     # By default, colocate_with resets the device function stack,\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor_or_indexed_slices(value, dtype, name, as_ref)\r\n   1266   else:\r\n   1267     return internal_convert_to_tensor(\r\n-> 1268         value, dtype=dtype, name=name, as_ref=as_ref)\r\n   1269 \r\n   1270 \r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, ctx)\r\n   1105 \r\n   1106     if ret is None:\r\n-> 1107       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)\r\n   1108 \r\n   1109     if ret is NotImplemented:\r\n\r\n/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/distribute/python/values.py in _tensor_conversion(var, dtype, name, as_ref)\r\n    243   # Try to avoid assignments to and other mutations of MirroredVariable\r\n    244   # state except through a DistributionStrategy.update() call.\r\n--> 245   assert not as_ref\r\n    246   return ops.internal_convert_to_tensor(\r\n    247       var.get(), dtype=dtype, name=name, as_ref=as_ref)\r\n\r\nAssertionError: \r\n```\r\nHere is the code for creating optimizer and applying backpropagation to the specified loss function:\r\n```\r\n\r\ndef minimize(loss):\r\n\r\n    lr = tf.constant(learning_rate_schedule[0], dtype=tf.float32)\r\n    for key, val in learning_rate_schedule.items():\r\n        lr = tf.cond(\r\n            tf.less(tf.train.get_or_create_global_step(), key), \r\n            lambda : lr,\r\n            lambda : tf.constant(val, dtype=tf.float32)\r\n        )\r\n    opt = tf.train.AdamOptimizer(learning_rate=lr, epsilon=FLAGS.epsilon)\r\n    if FLAGS.is_ema_enabled:\r\n        ema = tf.contrib.opt.MovingAverageOptimizer(\r\n            opt, \r\n            num_updates=tf.train.get_or_create_global_step()\r\n        )\r\n        grads = ema.compute_gradients(loss)\r\n        train_op = ema.apply_gradients(\r\n            grads,\r\n            global_step=tf.train.get_or_create_global_step()\r\n        )\r\n        return train_op, grads, ema.swapping_saver()\r\n    else:\r\n        grads = opt.compute_gradients(loss)\r\n        train_op = opt.apply_gradients(\r\n            grads, \r\n            global_step=tf.train.get_or_create_global_step()\r\n        )\r\n        return train_op, grads, tf.train.Saver()\r\n```\r\nIt seems that it causes a trouble when \"internal_tensor_conversion\" receives a reference variable though I am not sure of it. Am I doing something wrong or is it a bug?\r\n\r\nThank you for the help in advance."},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/427212610","html_url":"https://github.com/tensorflow/tensorflow/issues/21405#issuecomment-427212610","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21405","id":427212610,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNzIxMjYxMA==","user":{"login":"facaiy","id":1112263,"node_id":"MDQ6VXNlcjExMTIyNjM=","avatar_url":"https://avatars3.githubusercontent.com/u/1112263?v=4","gravatar_id":"","url":"https://api.github.com/users/facaiy","html_url":"https://github.com/facaiy","followers_url":"https://api.github.com/users/facaiy/followers","following_url":"https://api.github.com/users/facaiy/following{/other_user}","gists_url":"https://api.github.com/users/facaiy/gists{/gist_id}","starred_url":"https://api.github.com/users/facaiy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facaiy/subscriptions","organizations_url":"https://api.github.com/users/facaiy/orgs","repos_url":"https://api.github.com/users/facaiy/repos","events_url":"https://api.github.com/users/facaiy/events{/privacy}","received_events_url":"https://api.github.com/users/facaiy/received_events","type":"User","site_admin":false},"created_at":"2018-10-05T00:42:31Z","updated_at":"2018-10-05T00:42:31Z","author_association":"MEMBER","body":"tf.contrib.layers is deprecated and will be removed later."}}
{"actor":{"display_login":"facaiy"}}