{"action":"opened","issue":{"url":"https://api.github.com/repos/tensorflow/models/issues/3323","repository_url":"https://api.github.com/repos/tensorflow/models","labels_url":"https://api.github.com/repos/tensorflow/models/issues/3323/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/models/issues/3323/comments","events_url":"https://api.github.com/repos/tensorflow/models/issues/3323/events","html_url":"https://github.com/tensorflow/models/issues/3323","id":294521532,"number":3323,"title":"[object_detection] scope names of \"slim.conv2d\" calls.","user":{"login":"vs-zhehangd","id":29877977,"avatar_url":"https://avatars1.githubusercontent.com/u/29877977?v=4","gravatar_id":"","url":"https://api.github.com/users/vs-zhehangd","html_url":"https://github.com/vs-zhehangd","followers_url":"https://api.github.com/users/vs-zhehangd/followers","following_url":"https://api.github.com/users/vs-zhehangd/following{/other_user}","gists_url":"https://api.github.com/users/vs-zhehangd/gists{/gist_id}","starred_url":"https://api.github.com/users/vs-zhehangd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vs-zhehangd/subscriptions","organizations_url":"https://api.github.com/users/vs-zhehangd/orgs","repos_url":"https://api.github.com/users/vs-zhehangd/repos","events_url":"https://api.github.com/users/vs-zhehangd/events{/privacy}","received_events_url":"https://api.github.com/users/vs-zhehangd/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-05T19:54:13Z","updated_at":"2018-02-05T19:54:13Z","closed_at":null,"author_association":"NONE","body":"I am trying to parallelize object detection inference on multiple GPUs using pretrained checkpoints from the model zoo. I parallelize the model by doing like this:\r\n```python\r\nwith tf.variable_scope(tf.get_variable_scope(), reuse=tf.AUTO_REUSE):\r\n  with tf.name_scope('dev1'):\r\n    with tf.device('gpu:0'):\r\n      preprocessed_inputs   = model.preprocess(input1)\r\n      output_tensors        = model.predict(preprocessed_inputs)\r\n      postprocessed_tensors = model.postprocess(output_tensors)\r\n  \r\n  with tf.name_scope('dev2'):    \r\n    with tf.device('gpu:1'):\r\n      preprocessed_inputs   = model.preprocess(input2)\r\n      output_tensors        = model.predict(preprocessed_inputs)\r\n      postprocessed_tensors = model.postprocess(output_tensors)\r\n```\r\nDuring checkpoint restoration, I got the following error:\r\n```\r\nNotFoundError (see above for traceback): Key Conv_1/biases not found in checkpoint\r\n         [[Node: save/RestoreV2_2 = RestoreV2[dtypes=[DT_FLOAT], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](_arg_save/Const_0_0, save/RestoreV2_2/tensor_names, save/RestoreV2_2/shape_and_slices)]]\r\n         [[Node: save/RestoreV2_2/_1 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:GPU:1\", send_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device_incarnation=1, tensor_name=\"edge_1820_save/RestoreV2_2\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:GPU:1\"]()]]\r\n```\r\nThis is because some variables are not properly shared. I managed to identify the error source, it is in `FasterRCNNMetaArch._extract_rpn_feature_maps` ( `meta_architectures/faster_rcnn_meta_arch.py`) \r\n```python\r\nrpn_box_predictor_features = slim.conv2d(\r\n    rpn_features_to_crop,\r\n    self._first_stage_box_predictor_depth,\r\n    kernel_size=[kernel_size, kernel_size],\r\n    rate=self._first_stage_atrous_rate,\r\n    activation_fn=tf.nn.relu6)\r\n```\r\nThe scope name is not explicitly given, as a result we will get `Conv_1` instead of `Conv` in the second run. By adding the scope name to the function call\r\n```python\r\nrpn_box_predictor_features = slim.conv2d(\r\n    rpn_features_to_crop,\r\n    self._first_stage_box_predictor_depth,\r\n    kernel_size=[kernel_size, kernel_size],\r\n    rate=self._first_stage_atrous_rate,\r\n    activation_fn=tf.nn.relu6),\r\n    scope='Conv')\r\n```\r\nthe problem is solved and fully compatible with the original one. I did not investigate the other architectures, but I casually ran `grep` to get all `slim.conv2d` calls and found there were a few other calls in `core/box_predictor.py` that missed the scope names as well. Maybe someone would like to test and fix all this kind of problems at once? \r\n"}}
{"actor":{"display_login":"vs-zhehangd"}}