<p>Depthwise convolutions and grouped convolutions are very similar. Grouped convolutions apply a set of independent kernels across a number of channel groups, whereas depthwise convolutions apply a set of independent kernels for every input channel. Crucially, in both cases individual connections between input and output channels use weights that are not shared with any other input-output channel pair in both cases. As a result, we can apply (as the man said!) a reshape and sum to emulate a grouped convolution with a depthwise convolution. This approach comes at the expense of memory, as we must allocate a tensor that is multiple times larger to perform the intermediary computation.</p>

<p>The depthwise convolutions map individual input channels to multiple output channels, and the grouped convolution maps blocks of input channels to blocks of output channels. If we want a to apply a grouped convolution with 32 groups ot 128-channel input, we can instead apply a depthwise convolution with a channel multiplier of 128/32=4. The output tensor represents a decomposed version of the equivalent grouped convolution output - the first sixteen channels of the depthwise convolution output correspond to the first four channels of the grouped convolution output. We can reshape these channels into a set of 4x4 spaces, and sum along one of the new axes to realize the equivalent of the grouped convolution output. Across all the output channels, we just reshape by adding two new axes with a dimensionality of 4, sum, and reshape back to 128 channels.</p>

<pre><code># one resnext block per figure 3c
# see also https://arxiv.org/pdf/1611.05431.pdf
def bottleneck(x, strides, dim, is_training):
  input_channels = x.shape.as_list()[-1]
  bottleneck_depth = input_channels // 2
  x = tf.layers.conv2d(x, filters=bottleneck_depth, kernel_size=1, strides=strides)
  x = tf.layers.batch_normalization(x, training=is_training)
  x = tf.nn.relu(x)

  group_size = bottleneck_depth // cardinality
  w = tf.get_variable(name='depthwise_filter', shape=[3, 3, bottleneck_depth, group_size])
  x = tf.nn.depthwise_conv2d_native(x, w, strides=1, padding='same')
  depthwise_shape = x.shape.as_list()
  x = tf.reshape(x, depthwise_shape[:3] + [cardinality, group_size, group_size])
  x = tf.reduce_sum(x, axis=4)
  x = tf.reshape(x, depthwise_shape[:3] + [bottleneck_depth])

  x = tf.layers.batch_normalization(x, training=is_training)
  x = tf.nn.relu(x)
  x = tf.layers.conv2d(x, filters=dim, kernel_size=1, strides=1)
  x = tf.layers.batch_normalization(x, training=is_training)
  return tf.nn.relu(x)
</code></pre>

<p><strong>EDIT:</strong> It seems I didn't formulate the reshape/sum correctly. I've updated the code sample above to reflect what I now believe is the correct transformation. The older version was reducible to a depthwise convolution with a <code>channel_multiplier</code> of 1.</p>

<p>I'll illustrate the incorrect and correct behavior using numpy with weights fixed at 1 to better understand the difference. We'll look at a simpler 8-channel input with two groups.</p>

<pre><code>input = np.arange(8)
# =&gt; [0, 1, 2, 3, 4, 5, 6, 7]
# the result of applying a depthwise convolution with a channel multiplier of 2 and weights fixed at 1
depthwise_output = output.repeat(input, 4)
# =&gt; [0, 0, 0, 0, 1, 1, 1, 1, 2, 2, ..., 6, 6, 7, 7, 7, 7]
</code></pre>

<p>Incorrect transformation:</p>

<pre><code>x = depthwise_output.reshape((8, 4))
# =&gt; [[0, 0, 0, 0],
#     [1, 1, 1, 1],
#     [2, 2, 2, 2],
#     [3, 3, 3, 3],
#     [4, 4, 4, 4],
#     [5, 5, 5, 5],
#     [6, 6, 6, 6],
#     [7, 7, 7, 7]]
x = x.sum(axis=1)
# =&gt; [ 0,  4,  8, 12, 16, 20, 24, 28]
</code></pre>

<p>Correct transformation:</p>

<pre><code>x = depthwise_output.reshape((2, 4, 4))
# =&gt; [[[0, 0, 0, 0],
#      [1, 1, 1, 1],
#      [2, 2, 2, 2],
#      [3, 3, 3, 3]],
# 
#     [[4, 4, 4, 4],
#      [5, 5, 5, 5],
#      [6, 6, 6, 6],
#      [7, 7, 7, 7]]]
x = x.sum(axis=1)
# =&gt; [[ 6,  6,  6,  6],
#     [22, 22, 22, 22]])
x = x.reshape((8,))
# =&gt; [ 6,  6,  6,  6, 22, 22, 22, 22]
</code></pre>
Grouped depthwise convolution performance
<p>I'm attempting to improve the performance of my ResNeXt implementation in Tensorflow. David Berthelot mentioned a potential improvement over <a href="https://twitter.com/D_Berthelot_ML/status/908200291274711040" rel="nofollow noreferrer">on twitter</a>. I'd like to apply this to my implementation - how does reshape+sum fit into this?</p>

<pre><code># one resnext block per figure 3c
# see also https://arxiv.org/pdf/1611.05431.pdf
def bottleneck(x, strides, dim):
  x = tf.layers.conv2d(x, filters=64, kernel_size=1, strides=strides)
  x = tf.layers.batch_normalization(x, training=is_training)
  x = tf.nn.relu(x)
  w = tf.get_variable(name='depthwise_filter', shape=[3, 3, 64, cardinality])
  x = tf.nn.depthwise_conv2d_native(x, w, strides=1, padding='same')
  x = tf.layers.batch_normalization(x, training=is_training)
  x = tf.nn.relu(x)
  x = tf.layers.conv2d(x, filters=dim, kernel_size=1, strides=1)
  x = tf.layers.batch_normalization(x, training=is_training)
  return tf.nn.relu(x)
</code></pre>

<p><strong>EDIT:</strong> I thought this implementation was correct, and I just needed to add a couple operations to improve the performance. Taking another look at David's comment, the depthwise+reshape+sum wasn't instead of a single depthwise operation, but instead of some other method; the code above does not compute the equivalent of the bottleneck block version 3d.</p>