<p>Your custom layers will still use the GPU and you can confirm that as explained in this <a href="https://stackoverflow.com/questions/60542475/confirm-that-tf2-is-using-my-gpu-when-training/60544417#60544417">answer</a>. </p>

<p>You are right though that the custom layers won't use cuDNN. Why does it matter? To quote after NVidia:</p>

<blockquote>
  <p>cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers</p>
</blockquote>

<p>In other words, using these optimised primitives will enhance performance of the training. Number of examples with detailed explanation is provided in the <a href="https://arxiv.org/pdf/1410.0759.pdf" rel="nofollow noreferrer">cuDNN: Efficient Primitives for Deep Learning</a> paper. Take for instance <em>spatial convolutions</em>. Non-optimised implementation would use "naive" approach, while cuDNN uses all sorts of tricks to reduce number of operations and batch them appropriately. GPU is still fast when compared to classical CPU, cuDNN just makes it faster. For more recent, independent benchmarks, check out e.g. <a href="https://ieeexplore.ieee.org/document/8721631" rel="nofollow noreferrer">this article</a>.</p>

<p>Still, if Tensorflow runs in the GPU mode, complete computational graph will be executed on the GPU (to my knowledge there's even no simple way you could take out portion of the graph, i.e. intermediate layer, and put on CPU).</p>
tensorflow 2.0 custom layers on gpu
<p>Will completely custom-made layers in TensorFlow automatically be run on GPUs? I noticed that in this document (<a href="https://www.tensorflow.org/guide/keras/rnn#rnn_layers_and_rnn_cells" rel="nofollow noreferrer">https://www.tensorflow.org/guide/keras/rnn#rnn_layers_and_rnn_cells</a>) it seems that the RNN wrappers won't be using CudNN? That means it wouldn't run on the GPU right?</p>