<blockquote>
  <p>Depthwise  convolutions  provide  significant  performance  benefits 
  owing  to  the  reduction  in  both  parameters  and mult-adds. 
  However,  training  depthwise  convolution  layers  with GPUs  is  slow
  in  current  deep  learning  frameworks  because their  implementations
  cannot  fully  utilize  the  GPU  capacity.</p>
</blockquote>

<p><a href="https://arxiv.org/pdf/1803.09926.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1803.09926.pdf</a></p>
tf.nn.depthwise_conv2d is too slow. is it normal?
<p>I am trying out a recent arxiv work called "<a href="http://128.84.21.199/abs/1608.04337" rel="noreferrer">Factorized CNN</a>",</p>

<p>which mainly argues that spatially separated convolution (depth-wise convolution), together with channel-wise linear projection(1x1conv), can speed up the convolution operation.</p>

<p><a href="http://i.stack.imgur.com/qaSdE.png" rel="noreferrer">this is the figure for their conv layer architecture</a></p>

<p>I found out that I can implement this architecture with tf.nn.depthwise_conv2d and 1x1 convolution, or with tf.nn.separable_conv2d.</p>

<p>below is my implementation:</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-html lang-html prettyprint-override"><code>#conv filter for depthwise convolution
depthwise_filter = tf.get_variable("depth_conv_w", [3,3,64,1], initializer=tf.random_normal_initializer(stddev=np.sqrt(2.0/9/32)))
#conv filter for linear channel projection
pointwise_filter = tf.get_variable("point_conv_w", [1,1,64,64], initializer=tf.random_normal_initializer(stddev=np.sqrt(2.0/1/64)))
conv_b = tf.get_variable("conv_b", [64], initializer=tf.constant_initializer(0))
#depthwise convolution, with multiplier 1
conv_tensor = tf.nn.relu(tf.nn.depthwise_conv2d(tensor, depthwise_filter, [1,1,1,1], padding='SAME'))
#linear channel projection with 1x1 convolution
conv_tensor = tf.nn.bias_add(tf.nn.conv2d(conv_tensor, pointwise_filter, [1,1,1,1], padding='VALID'), conv_b)
#residual
tensor = tf.add(tensor, conv_tensor)</code></pre>
</div>
</div>
</p>

<p>This should be around 9 times faster than the original 3x3x64 -> 64 channel convolution.</p>

<p>However, I cannot experience any performance improvement.</p>

<p>I must assume that I am doing this wrong, or there's something wrong with tensorflow's implementation.</p>

<p>Since there is few example using depthwise_conv2d, I am leaving this question here.</p>

<p>Is this slow speed normal? or is there any mistake?</p>