<p>Although more memory efficient, depthwise 2D convolutions can indeed be slower than regular 2D convolutions.</p>
<p><a href="https://arxiv.org/pdf/1803.10615.pdf" rel="nofollow noreferrer">Gholami et al.</a> (SqueezeNext: Hardware-Aware Neural Network Design) states that:</p>
<blockquote>
<p>The reason for this is the inefficiency of depthwise-separable convolution in  terms of hardware performance, which is due to its poor arithmetic intensity (ratio of compute to memory operations).</p>
</blockquote>
Why would `DepthwiseConv2D` be slower than `Conv2D`
<p>I'm reimplementing <a href="https://arxiv.org/abs/1704.04861" rel="nofollow noreferrer">MobileNet</a>, but I find the depthwise convolution is no faster than conv2d(I haven't included the 1 by 1 pointwise convolution yet). Here's the test code run on colab: <a href="https://colab.research.google.com/drive/1nBuYrmmH5kM0jbtIZdsuiG6uJbU6mpA7?usp=sharing" rel="nofollow noreferrer">https://colab.research.google.com/drive/1nBuYrmmH5kM0jbtIZdsuiG6uJbU6mpA7?usp=sharing</a></p>
<pre class="lang-py prettyprint-override"><code>import tensorflow as tf
import time
x = tf.random.normal((2, 64, 64, 3))
conv = tf.keras.layers.Conv2D(16, 3, strides=1, padding='same')
dw = tf.keras.layers.DepthwiseConv2D(3, padding='same')
start = time.time()
conv(x)
print('conv2d:', time.time() - start)    # approximate 0.0036s
start = time.time()
dw(x)
print('dw:', time.time() - start)    # approximate 0.0034s
%timeit conv(x)    # 1000 loops, best of 3: 225 µs per loop
%timeit dw(x)    # 1000 loops, best of 3: 352 µs per loop
</code></pre>
<p>I also try it on my laptop using CPUs only, similar results are spotted. Why would <code>DepthwiseConv2D</code> be slower than <code>Conv2D</code>? Did I make any mistakes?</p>