<p>Dropout technique is not implemented on every single layer within a neural network; it’s commonly leveraged within the neurons in the last few layers within the network.</p>
<p>The technique works by randomly reducing the number of interconnecting neurons within a neural network. At every training step, each neuron has a chance of being left out, or rather, dropped out of the collated contribution from connected neurons</p>
<p>There’s some debate as to whether the dropout should be placed before or after the activation function. As a rule of thumb, place the dropout after the activate function for all activation functions other than <code>relu</code>.</p>
<p>you can add <code>dropout</code> after every hidden layer and generally it affect only the previous layer in (your case it will effect <code>(x = layers.Dense(1024, activation='relu')(x) )</code>). In the original paper that proposed dropout layers, by <a href="https://arxiv.org/pdf/1207.0580.pdf" rel="nofollow noreferrer">Hinton (2012)</a>, dropout (with p=0.5) was used on each of the fully connected (dense) layers before the output; it was not used on the convolutional layers. This became the most commonly used configuration.</p>
<p>I am adding the resources link that might help you:</p>
<p><a href="https://towardsdatascience.com/understanding-and-implementing-dropout-in-tensorflow-and-keras-a8a3a02c1bfa" rel="nofollow noreferrer">https://towardsdatascience.com/understanding-and-implementing-dropout-in-tensorflow-and-keras-a8a3a02c1bfa</a></p>
<p><a href="https://towardsdatascience.com/dropout-on-convolutional-layers-is-weird-5c6ab14f19b2" rel="nofollow noreferrer">https://towardsdatascience.com/dropout-on-convolutional-layers-is-weird-5c6ab14f19b2</a></p>
<p><a href="https://towardsdatascience.com/machine-learning-part-20-dropout-keras-layers-explained-8c9f6dc4c9ab" rel="nofollow noreferrer">https://towardsdatascience.com/machine-learning-part-20-dropout-keras-layers-explained-8c9f6dc4c9ab</a></p>
What layers are affected by dropout layer in Tensorflow?
<p>Consider transfer learning in order to use a pretrained model in keras/tensorflow. For each old layer, <code>trained</code> parameter is set to <code>false</code> so that its weights are not updated during training whereas the last layer(s) have been substituted with new layers and these must be trained. Particularly two fully connected hidden layers with <code>512</code> and <code>1024</code> neurons and and relu activation function have been added. After these layers a Dropout layer is used with <code>rate</code> <code>0.2</code>. This means that during each epoch of training <code>20%</code> of the neurons are randomly discarded.</p>
<p>What layers does this dropout layer affect? Does it affect all the network including also the pretrained layers for which <code>layer.trainable=false</code> has been set or does it affect only the newly added layers? Or does it affect only the previous layer (i.e., the one with <code>1024</code> neurons)?</p>
<p>In other words, which layer(s) do the neurons that are turned off during each epoch by the dropout belong to?</p>
<pre><code>import os

from tensorflow.keras import layers
from tensorflow.keras import Model
  
from tensorflow.keras.applications.inception_v3 import InceptionV3

local_weights_file = 'weights.h5'

pre_trained_model = InceptionV3(input_shape = (150, 150, 3), 
                                include_top = False, 
                                weights = None)

pre_trained_model.load_weights(local_weights_file)

for layer in pre_trained_model.layers:
  layer.trainable = False
  
# pre_trained_model.summary()

last_layer = pre_trained_model.get_layer('mixed7')
last_output = last_layer.output

# Flatten the output layer to 1 dimension
x = layers.Flatten()(last_output)
# Add two fully connected layers with 512 and 1,024 hidden units and ReLU activation
x = layers.Dense(512, activation='relu')(x)
x = layers.Dense(1024, activation='relu')(x)
# Add a dropout rate of 0.2
x = layers.Dropout(0.2)(x)                  
# Add a final sigmoid layer for classification
x = layers.Dense  (1, activation='sigmoid')(x)           

model = Model( pre_trained_model.input, x) 

model.compile(optimizer = RMSprop(lr=0.0001), 
              loss = 'binary_crossentropy', 
              metrics = ['accuracy'])
</code></pre>