{"action":"closed","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/10535","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/10535/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/10535/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/10535/events","html_url":"https://github.com/tensorflow/tensorflow/issues/10535","id":234534261,"number":10535,"title":"tf.nn.conv3d_transpose really slow on i7 CPU with 100+G free memory","user":{"login":"laoreja","id":9369143,"avatar_url":"https://avatars2.githubusercontent.com/u/9369143?v=4","gravatar_id":"","url":"https://api.github.com/users/laoreja","html_url":"https://github.com/laoreja","followers_url":"https://api.github.com/users/laoreja/followers","following_url":"https://api.github.com/users/laoreja/following{/other_user}","gists_url":"https://api.github.com/users/laoreja/gists{/gist_id}","starred_url":"https://api.github.com/users/laoreja/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laoreja/subscriptions","organizations_url":"https://api.github.com/users/laoreja/orgs","repos_url":"https://api.github.com/users/laoreja/repos","events_url":"https://api.github.com/users/laoreja/events{/privacy}","received_events_url":"https://api.github.com/users/laoreja/received_events","type":"User","site_admin":false},"labels":[{"id":386191887,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20response","name":"stat:awaiting response","color":"5319e7","default":false},{"id":473172988,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:bug/performance","name":"type:bug/performance","color":"fbca04","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2017-06-08T13:50:49Z","updated_at":"2018-02-08T22:29:30Z","closed_at":"2018-02-08T22:29:29Z","author_association":"NONE","body":"\r\n### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: Yes\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 16.04\r\n- **TensorFlow installed from (source or binary)**: virtualenv pip\r\n- **TensorFlow version (use command below)**: 1.1.0\r\n- **Bazel version (if compiling from source)**:\r\n- **CUDA/cuDNN version**: 8.0\r\n- **GPU model and memory**: on CPU\r\n- **Exact command to reproduce**:\r\n\r\n### Describe the problem\r\nIt takes about 10 mins to do the tf.nn.conv3d_transpose computation.\r\nThe input size is [1, 97, 128, 256, 32], kernel size is 3*3*3, strides is [1, 2, 2, 2, 1], and the output size is [1, 193, 256, 512, 1].\r\n\r\nAt first my model runs with normal speed. Before this layer, there is several tf.nn.conv3d and tf.nn.conv3d_transpose computation.\r\nAfter this step's computation, it seems the computation become really slow down.\r\n\r\nBut my model runs smoothly on 12G GPU, if using 4 GPUs, 1.8 examples/sec.\r\n\r\nI notice a similar issue #3128. That issue also reports some problem on CPU, but not on GPU. Is that issue resolved?\r\n\r\n### Source code / logs\r\nSince I am training a large model on a large dataset. I will only include the model's code. If is needed, I will include more codes. \r\n\r\nThe computationally cost layer is the last layer in the variable_scope learning_regularization, right before the scope soft_argmin in `def _build_model`.\r\n\r\nFor more details, I am reimplementing https://arxiv.org/pdf/1703.04309.pdf.\r\n\r\n```python\r\nfrom collections import namedtuple\r\n\r\nimport numpy as np\r\nimport tensorflow as tf\r\nimport six\r\n\r\nfrom tensorflow.python.training import moving_averages\r\n\r\n# If a model is trained using multiple GPUs, prefix all Op names with tower_name\r\n# to differentiate the operations. Note that this prefix is removed from the\r\n# names of the summaries when visualizing a model.\r\nTOWER_NAME = 'tower'\r\n\r\n# Batch normalization. Constant governing the exponential moving average of\r\n# the 'global' mean and variance for all activations.\r\nBATCHNORM_MOVING_AVERAGE_DECAY = 0.9997\r\n\r\n# The decay to use for the moving average.\r\nMOVING_AVERAGE_DECAY = 0.9999\r\n\r\n\r\nHParams = namedtuple('HParams',\r\n                     ['batch_size', 'lrn_rate',\r\n                     'weight_decay_rate',\r\n                     'relu_leakiness', 'optimizer', 'max_disparity'])\r\n\r\n\r\nclass GCNet(object):\r\n  \"\"\"GCNet model.\"\"\"\r\n\r\n  def __init__(self, hps, left_images, right_images, gt_disparity, mask, mode): \r\n    \"\"\"ResNet constructor.\r\n\r\n    Args:\r\n      hps: Hyperparameters.\r\n      images: Batches of images. [batch_size, image_size, image_size, 3]\r\n      labels: Batches of labels. [batch_size, num_classes]\r\n      mode: One of 'train', 'eval' and 'predict'.\r\n    \"\"\"\r\n    self.hps = hps\r\n    self._left_images = left_images\r\n    self._right_images = right_images\r\n    self.gt_disparity = gt_disparity\r\n    self.mask = mask\r\n    self.mode = mode\r\n    self.debug_op_list = []  \r\n\r\n    self._extra_train_ops = []\r\n    \r\n  def build_graph_to_loss(self):\r\n    self._build_model()\r\n    self._build_loss_op()\r\n\r\n  def _stride_arr(self, stride):\r\n    \"\"\"Map a stride scalar to the stride array for tf.nn.conv2d.\"\"\"\r\n    return [1, stride, stride, 1]\r\n    \r\n  def _stride_3d_arr(self, stride):\r\n    \"\"\"Map a stride scalar to the stride array for tf.nn.conv2d.\"\"\"\r\n    return [1, stride, stride, stride, 1]\r\n\r\n  def _build_model(self):\r\n    \"\"\"Build the core model within the graph.\"\"\"\r\n\r\n    layer_idx = 1\r\n    with tf.variable_scope('unary_features', reuse=False):\r\n      with tf.variable_scope('layer_'+str(layer_idx)):\r\n        layer_idx += 1\r\n        left_x = self._left_images\r\n        left_x = self._conv('conv', left_x, 5, 3, 32, self._stride_arr(2))\r\n        left_x = self._relu(left_x, self.hps.relu_leakiness)\r\n        left_x = self._batch_norm('bn', left_x)\r\n      tf.add_to_collection('shapes', tf.shape(left_x))\r\n        \r\n      for i in six.moves.range(8):\r\n        left_x, layer_idx = self._unary_feat_residual(left_x, 3, 32, 32, self._stride_arr(1), layer_idx)\r\n        tf.add_to_collection('shapes', tf.shape(left_x))\r\n    \r\n      with tf.variable_scope('layer_'+str(layer_idx)):\r\n        layer_idx += 1\r\n        left_x = self._conv('conv', left_x, 3, 32, 32, self._stride_arr(1))\r\n      tf.add_to_collection('shapes', tf.shape(left_x))\r\n    \r\n    layer_idx = 1    \r\n    with tf.variable_scope('unary_features', reuse=True):\r\n      with tf.variable_scope('layer_'+str(layer_idx)):\r\n        layer_idx += 1\r\n        right_x = self._left_images\r\n        right_x = self._conv('conv', right_x, 5, 3, 32, self._stride_arr(2))\r\n        right_x = self._relu(right_x, self.hps.relu_leakiness)\r\n        right_x = self._batch_norm('bn', right_x)\r\n        \r\n      for i in six.moves.range(8):\r\n        right_x, layer_idx = self._unary_feat_residual(right_x, 3, 32, 32, self._stride_arr(1), layer_idx)\r\n\r\n      with tf.variable_scope('layer_'+str(layer_idx)):\r\n        layer_idx += 1\r\n        right_x = self._conv('conv', right_x, 3, 32, 32, self._stride_arr(1))\r\n      \r\n    with tf.variable_scope('cost_volumn'):\r\n      left_cost_volume = tf.stack([tf.identity(left_x)] * (self.hps.max_disparity/2+1), axis=1, name='left_stack')\r\n      right_cost_volume = []\r\n      cur_width = tf.shape(right_x)[2]\r\n\r\n      for depth in six.moves.range(self.hps.max_disparity/2+1):\r\n        right_cost_volume.append(tf.pad(tf.slice(right_x, [0, 0, 0, 0], [-1, -1, cur_width - depth, -1], name='right_slice_'+str(depth)),\r\n                                        [[0, 0], [0, 0], [depth, 0], [0, 0]],\r\n                                        name='right_pad_'+str(depth)\r\n                                        ))\r\n      right_cost_volume = tf.stack(right_cost_volume, axis=1, name='right_stack')\r\n      x = tf.concat([left_cost_volume, right_cost_volume], 4)\r\n      tf.add_to_collection('shapes', tf.shape(x))\r\n      \r\n          \r\n    with tf.variable_scope('learning_regularization'):\r\n      stored_features = []\r\n\r\n      in_filters = [64, 64, 64, 64]\r\n      out_filters = [32, 64, 64, 64]\r\n      in_filters_stride_2 = [64, 64, 64, 64]\r\n      out_filters_stride_2 = [64, 64, 64, 128]\r\n      for i in six.moves.range(4):\r\n        tmp_x, layer_idx = self._regularization_subsample(x, 3, in_filters[i], out_filters[i], self._stride_3d_arr(1), layer_idx)\r\n        tf.add_to_collection('shapes', tf.shape(tmp_x))\r\n        stored_features.append(tmp_x)\r\n        \r\n        with tf.variable_scope('layer_'+str(layer_idx)):\r\n          layer_idx += 1\r\n          x = self._conv3d('conv3d', x, 3, in_filters_stride_2[i], out_filters_stride_2[i], self._stride_3d_arr(2))\r\n          x = self._relu(x, self.hps.relu_leakiness)\r\n          x = self._batch_norm('bn', x)\r\n          tf.add_to_collection('shapes', tf.shape(x))\r\n\r\n      \r\n      assert stored_features[0] is not stored_features[1]\r\n\r\n      for i in six.moves.range(2):\r\n        with tf.variable_scope('layer_'+str(layer_idx)):\r\n          layer_idx += 1\r\n          x = self._conv3d('conv3d', x, 3, 128, 128, self._stride_3d_arr(1))\r\n          x = self._relu(x, self.hps.relu_leakiness)\r\n          x = self._batch_norm('bn', x)\r\n          tf.add_to_collection('shapes', tf.shape(x))\r\n\r\n      transposed_in_filters = [128, 64, 64, 64]\r\n      transposed_out_filters = [64, 64, 64, 32]\r\n      \r\n      for i in six.moves.range(4):\r\n        x, layer_idx = self._regularization_upsample(x, stored_features[-i-1], 3, transposed_in_filters[i], transposed_out_filters[i], self._stride_3d_arr(2), layer_idx)\r\n        tf.add_to_collection('shapes', tf.shape(x))\r\n      \r\n      with tf.variable_scope('layer_'+str(layer_idx)):\r\n        layer_idx += 1\r\n        input_shape = tf.shape(self.gt_disparity)\r\n        x = self._conv3d_trans('conv_trans', x, 3, 32, 1, self._stride_3d_arr(2), [input_shape[0], self.hps.max_disparity+1, input_shape[1], input_shape[2], 1])\r\n        tf.add_to_collection('shapes', tf.shape(x))\r\n        self.debug_op_list.append(tf.shape(x))\r\n\r\n    \r\n    with tf.variable_scope('soft_argmin'):\r\n        x = tf.squeeze(x, squeeze_dims=[4], name='squeeze')\r\n        tf.add_to_collection('shapes', tf.shape(x))\r\n        x = tf.transpose(x, perm=[0, 2, 3, 1], name='transpose')\r\n        tf.add_to_collection('shapes', tf.shape(x))\r\n        x = tf.nn.softmax(x, dim=-1, name='softmax')\r\n        tf.add_to_collection('shapes', tf.shape(x))\r\n\r\n        multiplier = tf.range(0, self.hps.max_disparity+1, dtype=tf.float32, name='depth_range')\r\n        x = tf.multiply(x, multiplier, name='softmax_mul_depth')\r\n        tf.add_to_collection('shapes', tf.shape(x))\r\n        self.predicted_disparity = tf.reduce_sum(x, axis=3, name='reduce_sum')       \r\n        tf.add_to_collection('shapes', tf.shape(self.predicted_disparity))\r\n    self.shapes = tf.get_collection('shapes')\r\n    self.debug_op_list.append(self.shapes)\r\n\r\n  def _build_loss_op(self):\r\n    with tf.variable_scope('loss'):\r\n      self.abs_loss = tf.reduce_mean(tf.abs((self.gt_disparity - self.predicted_disparity) * self.mask), name='abs_loss')\r\n      self.total_loss = self.abs_loss + self._decay()\r\n\r\n      \r\n  def _add_loss_summaries(self):\r\n    \"\"\"Add summaries for losses in CIFAR-10 model.\r\n\r\n    Generates moving average for all losses and associated summaries for\r\n    visualizing the performance of the network.\r\n\r\n    Args:\r\n      total_loss: Total loss from loss().\r\n    Returns:\r\n      loss_averages_op: op for generating moving averages of losses.\r\n    \"\"\"\r\n    # Compute the moving average of all individual losses and the total loss.\r\n    loss_averages = tf.train.ExponentialMovingAverage(0.9, name='loss_avg')\r\n    self.loss_averages_op = loss_averages.apply([self.abs_loss, self.total_loss])\r\n\r\n    # Attach a scalar summary to all individual losses and the total loss; do the\r\n    # same for the averaged version of the losses.\r\n    for l in [self.abs_loss, self.total_loss]:\r\n      # Name each loss as '(raw)' and name the moving average version of the loss\r\n      # as the original loss name.\r\n      tf.summary.scalar(l.op.name + ' (raw)', l)\r\n      tf.summary.scalar(l.op.name, loss_averages.average(l))\r\n    \r\n  def _build_train_op(self, global_step):\r\n    \"\"\"Build training specific ops for the graph.\"\"\"\r\n    self.lrn_rate = tf.constant(self.hps.lrn_rate, tf.float32)\r\n    tf.summary.scalar('learning_rate', self.lrn_rate)\r\n\r\n    loss_averages_op = self._add_loss_summaries()\r\n    \r\n    with tf.control_dependencies([loss_averages_op]):\r\n      if self.hps.optimizer == 'sgd':\r\n        optimizer = tf.train.GradientDescentOptimizer(self.lrn_rate)\r\n      elif self.hps.optimizer == 'mom':\r\n        optimizer = tf.train.MomentumOptimizer(self.lrn_rate, 0.9)\r\n      elif self.hps.optimizer == 'RMSProp':\r\n        optimizer = tf.train.RMSPropOptimizer(self.lrn_rate, decay=0.9, momentum=0.9, epsilon=1)\r\n        \r\n        trainable_variables = tf.trainable_variables()\r\n        grads = optimizer.compute_gradients(self.total_loss, trainable_variables)\r\n\r\n\r\n    apply_op = optimizer.apply_gradients(\r\n        grads,\r\n        global_step=global_step, \r\n        name='train_step')\r\n        \r\n    # Track the moving averages of all trainable variables.\r\n    variable_averages = tf.train.ExponentialMovingAverage(\r\n        MOVING_AVERAGE_DECAY, global_step)\r\n    variables_averages_op = variable_averages.apply(tf.trainable_variables())\r\n\r\n    with tf.control_dependencies([apply_op, variables_averages_op]):\r\n      self.train_op = tf.no_op(name='train')\r\n\r\n  def _regularization_upsample(self, x, feature, filter_size, in_filter, out_filter, stride, layer_idx):\r\n    with tf.variable_scope('layer_'+str(layer_idx)):\r\n      layer_idx += 1\r\n      x = self._conv3d_trans('conv_trans', x, filter_size, in_filter, out_filter, stride, tf.shape(feature))\r\n      x = self._relu(x, self.hps.relu_leakiness)\r\n      x = self._batch_norm('bn', x)\r\n      \r\n    with tf.variable_scope('residual_after_'+str(layer_idx-1)):\r\n      x += feature\r\n\r\n    tf.logging.debug('image after unit %s', x.get_shape())\r\n    return x, layer_idx\r\n\r\n  def _regularization_subsample(self, x, filter_size, in_filter, out_filter, stride, layer_idx):\r\n\r\n    with tf.variable_scope('layer_'+str(layer_idx)):\r\n      layer_idx += 1\r\n      x = self._conv3d('conv3d', x, filter_size, in_filter, out_filter, stride)\r\n      x = self._relu(x, self.hps.relu_leakiness)\r\n      x = self._batch_norm('bn', x)\r\n\r\n    with tf.variable_scope('layer_'+str(layer_idx)):\r\n      layer_idx += 1\r\n      x = self._conv3d('conv3d', x, filter_size, out_filter, out_filter, stride)\r\n      x = self._relu(x, self.hps.relu_leakiness)\r\n      x = self._batch_norm('bn', x)\r\n      \r\n    tf.logging.debug('image after unit %s', x.get_shape())\r\n    return x, layer_idx\r\n\r\n  def _unary_feat_residual(self, x, filter_size, in_filter, out_filter, stride, layer_idx):\r\n    orig_x = x\r\n    orig_layer_idx = layer_idx - 1\r\n    \r\n    for i in six.moves.range(2):\r\n      with tf.variable_scope('layer_'+str(layer_idx)):\r\n        layer_idx += 1\r\n        x = self._conv('conv', x, 3, in_filter, out_filter, stride)\r\n        x = self._relu(x, self.hps.relu_leakiness)\r\n        x = self._batch_norm('bn', x)\r\n          \r\n    with tf.variable_scope('residual_btw_'+str(layer_idx-1)+'_'+str(orig_layer_idx)):\r\n      x += orig_x\r\n\r\n    tf.logging.debug('image after unit %s', x.get_shape())\r\n    return x, layer_idx\r\n\r\n\r\n  def _decay(self):\r\n    \"\"\"L2 weight decay loss.\"\"\"\r\n    costs = []\r\n    for var in tf.trainable_variables():\r\n      if var.op.name.find(r'DW') > 0:\r\n        costs.append(tf.nn.l2_loss(var))\r\n        # tf.summary.histogram(var.op.name, var)\r\n\r\n    return tf.multiply(self.hps.weight_decay_rate, tf.add_n(costs))\r\n\r\n  def _conv(self, name, x, filter_size, in_filters, out_filters, strides):\r\n    \"\"\"Convolution.\"\"\"\r\n    with tf.variable_scope(name):\r\n      n = filter_size * filter_size * out_filters\r\n      kernel = self._variable_on_cpu(\r\n          'DW', [filter_size, filter_size, in_filters, out_filters],\r\n          initializer=tf.random_normal_initializer(\r\n              stddev=np.sqrt(2.0/n)))\r\n      return tf.nn.conv2d(x, kernel, strides, padding='SAME')\r\n      \r\n  def _conv3d(self, name, x, filter_size, in_filters, out_filters, strides):\r\n    \"\"\"Convolution.\"\"\"\r\n    with tf.variable_scope(name):\r\n      n = filter_size * filter_size * filter_size * out_filters\r\n      kernel = self._variable_on_cpu(\r\n          'DW', [filter_size, filter_size, filter_size, in_filters, out_filters],\r\n           initializer=tf.random_normal_initializer(\r\n              stddev=np.sqrt(2.0/n)))\r\n      return tf.nn.conv3d(x, kernel, strides, padding='SAME')\r\n      \r\n  def _conv3d_trans(self, name, x, filter_size, in_filters, out_filters, strides, output_shape):\r\n    \"\"\"Convolution.\"\"\"\r\n    with tf.variable_scope(name):\r\n      n = filter_size * filter_size * filter_size * out_filters\r\n      kernel = self._variable_on_cpu(\r\n          'DW', [filter_size, filter_size, filter_size, out_filters, in_filters],\r\n            initializer=tf.random_normal_initializer(\r\n              stddev=np.sqrt(2.0/n)))\r\n      x_shape = tf.shape(x)\r\n      self.debug_op_list.append(tf.shape(kernel))\r\n      return tf.nn.conv3d_transpose(\r\n                x, \r\n                kernel, \r\n                output_shape,\r\n                strides, \r\n                padding='SAME')\r\n\r\n  def _relu(self, x, leakiness=0.0):\r\n    \"\"\"Relu, with optional leaky support.\"\"\"\r\n    return tf.where(tf.less(x, 0.0), leakiness * x, x, name='leaky_relu')\r\n\r\n  # TODO(xpan): Consider batch_norm in contrib/layers/python/layers/layers.py\r\n  def _batch_norm(self, name, x):\r\n    \"\"\"Batch normalization.\"\"\"\r\n    with tf.variable_scope(name):\r\n      params_shape = [x.get_shape()[-1]]\r\n\r\n      beta = self._variable_on_cpu(\r\n          'beta', params_shape,\r\n          initializer=tf.constant_initializer(0.0, tf.float32))\r\n      gamma = self._variable_on_cpu(\r\n          'gamma', params_shape,\r\n          initializer=tf.constant_initializer(1.0, tf.float32))\r\n\r\n      if self.mode == 'train':\r\n        mean, variance = tf.nn.moments(x, range(len(x.get_shape())-1), name='moments')\r\n\r\n        moving_mean = self._variable_on_cpu(\r\n            'moving_mean', params_shape,\r\n            initializer=tf.constant_initializer(0.0, tf.float32),\r\n            trainable=False)\r\n        moving_variance = self._variable_on_cpu(\r\n            'moving_variance', params_shape,\r\n            initializer=tf.constant_initializer(1.0, tf.float32),\r\n            trainable=False)\r\n\r\n        self._extra_train_ops.append(moving_averages.assign_moving_average(\r\n            moving_mean, mean, BATCHNORM_MOVING_AVERAGE_DECAY))\r\n        self._extra_train_ops.append(moving_averages.assign_moving_average(\r\n            moving_variance, variance, BATCHNORM_MOVING_AVERAGE_DECAY))\r\n      else:\r\n        mean = self._variable_on_cpu(\r\n            'moving_mean', params_shape,\r\n            initializer=tf.constant_initializer(0.0, tf.float32),\r\n            trainable=False)\r\n        variance = self._variable_on_cpu(\r\n            'moving_variance', params_shape,\r\n            initializer=tf.constant_initializer(1.0, tf.float32),\r\n            trainable=False)\r\n      y = tf.nn.batch_normalization(\r\n          x, mean, variance, beta, gamma, 0.001)\r\n      y.set_shape(x.get_shape())\r\n      return y\r\n\r\n  def _variable_on_cpu(self, name, shape, initializer, dtype=tf.float32, trainable=True):\r\n    \"\"\"Helper to create a Variable stored on CPU memory.\r\n\r\n    Args:\r\n      name: name of the variable\r\n      shape: list of ints\r\n      initializer: initializer for Variable\r\n\r\n    Returns:\r\n      Variable Tensor\r\n    \"\"\"\r\n    with tf.device('/cpu:0'):\r\n      var = tf.get_variable(name, shape, initializer=initializer, dtype=dtype, trainable=trainable)\r\n    return var\r\n```"}}
{"actor":{"display_login":"michaelisard"}}