title: How much faster is NCHW compared to NHWC in TensorFlow/cuDNN?
body: <p><a href="https://www.tensorflow.org/performance/performance_models" rel="noreferrer">The official TensorFlow performance guide</a> states:</p>
  
  <blockquote>
    <p>Most TensorFlow operations used by a CNN support both NHWC and NCHW data format. On GPU, NCHW is faster. But on CPU, NHWC is sometimes faster.</p>
  </blockquote>
  
  <p>How much faster is NCHW compared to NHWC in TensorFlow/cuDNN, for convolution? Are there any references or benchmarks for this?</p>
  
  <p>Also, why is it faster? As I understand (see <a href="https://github.com/tensorflow/tensorflow/issues/8286" rel="noreferrer">here</a>), TensorFlow for NHWC on GPU will internally always transpose to NCHW, then calls the cuDNN conv kernel for NCHW, then transpose it back. But why does it do that? The cuDNN conv kernel also works for NHWC. Maybe at some point they did the comparison and the cuDNN conv kernel for NHWC was very slow. But is that up-to-date? And how big was the difference? What are the technical reasons that NHWC is so much slower? Or is the cuDNN kernel for this case just not well optimized?</p>