<p>There is a way to access pre-activation layers for pretrained Keras models using TF version 2.7.0. Here's how to access two intermediate pre-activation outputs from VGG19 in a <em>single</em> forward pass.</p>
<p>Initialize VGG19 model. We can omit top layers to avoid loading unnecessary parameters into memory.</p>
<pre class="lang-py prettyprint-override"><code>vgg19 = tf.keras.applications.VGG19(
    include_top=False,
    weights=&quot;imagenet&quot;
)
</code></pre>
<p><strong>This is the important part: Create a deepcopy of the intermediate layer form which you like to have the features, change the activation of the conv layers to linear (i.e. no activation), rename the layer (otherwise two layers in the model will have the same name which will raise errors) and finally pass the output of the <em>previous</em>  through the copied conv layer.</strong></p>
<pre class="lang-py prettyprint-override"><code># for more intermediate features wrap a loop around it to avoid copy paste
b5c4_layer = deepcopy(vgg19.get_layer(&quot;block5_conv4&quot;))
b5c4_layer.activation = tf.keras.activations.linear
b5c4_layer._name = b5c4_layer.name + str(&quot;_preact&quot;)
b5c4_preact_output = b5c4_layer(vgg19.get_layer(&quot;block5_conv3&quot;).output)

b2c2_layer = deepcopy(vgg19.get_layer(&quot;block2_conv2&quot;))
b2c2_layer.activation = tf.keras.activations.linear
b2c2_layer._name = b2c2_layer.name + str(&quot;_preact&quot;)
b2c2_preact_output = b2c2_layer(vgg19.get_layer(&quot;block2_conv1&quot;).output)
</code></pre>
<p>Finally, get the outputs and check if they equal post-activation outputs when we apply ReLU-activation.</p>
<pre class="lang-py prettyprint-override"><code>vgg19_features = Model(vgg19.input, [b2c2_preact_output, b5c4_preact_output])
vgg19_features_control = Model(vgg19.input, [vgg19.get_layer(&quot;block2_conv2&quot;).output, vgg19.get_layer(&quot;block5_conv4&quot;).output])

b2c2_preact, b5c4_preact = vgg19_features(tf.keras.applications.vgg19.preprocess_input(img))
b2c2, b5c4 = vgg19_features_control(tf.keras.applications.vgg19.preprocess_input(img))

print(np.allclose(tf.keras.activations.relu(b2c2_preact).numpy(),b2c2.numpy()))
print(np.allclose(tf.keras.activations.relu(b5c4_preact).numpy(),b5c4.numpy()))
</code></pre>
<pre><code>True
True
</code></pre>
<p>Here's a visualization similar to Fig. 6 of <a href="https://arxiv.org/pdf/1809.00219.pdf" rel="nofollow noreferrer">Wang et al.</a> to see the effect in the feature space.
<a href="https://i.stack.imgur.com/QjAyJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/QjAyJ.png" alt="VGG19-intermediate" /></a></p>
<p>Input image</p>
<p><a href="https://i.stack.imgur.com/CJ3PA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/CJ3PA.png" alt="input image" /></a></p>
Accessing intermediate tensors of a Keras Model that were not explicitly exposed as layers in TF 2.0
<p>Is it possible to access pre-activation tensors in a Keras Model? For example, given this model:</p>

<pre class="lang-python prettyprint-override"><code>import tensorflow as tf
image_ = tf.keras.Input(shape=[224, 224, 3], batch_size=1)
vgg19 = tf.keras.applications.VGG19(include_top=False, weights='imagenet', input_tensor=image_, input_shape=image_.shape[1:], pooling=None)
</code></pre>
<p>the usual way to access layers is:</p>

<pre class="lang-python prettyprint-override"><code>intermediate_layer_model = tf.keras.models.Model(inputs=image_, outputs=[vgg19.get_layer('block1_conv2').output])
intermediate_layer_model.summary()
</code></pre>
<p>This gives the ReLU outputs for a layer, while I would like the ReLU inputs. I tried doing this:</p>

<pre class="lang-python prettyprint-override"><code>graph = tf.function(vgg19, [tf.TensorSpec.from_tensor(image_)]).get_concrete_function().graph
outputs = [graph.get_tensor_by_name(tname) for tname in [
    'vgg19/block4_conv3/BiasAdd:0',
    'vgg19/block4_conv4/BiasAdd:0',
    'vgg19/block5_conv1/BiasAdd:0'
]]
intermediate_layer_model = tf.keras.models.Model(inputs=image_, outputs=outputs)
intermediate_layer_model.summary()
</code></pre>
<p>but I get the error</p>
<pre class="lang-python prettyprint-override"><code>ValueError: Unknown graph. Aborting.
</code></pre>
<p>The only workaround I've found is to edit the model file to manually expose the intermediates, turning every layer like this:</p>

<pre class="lang-python prettyprint-override"><code>x = layers.Conv2D(256, (3, 3), activation=&quot;relu&quot;, padding=&quot;same&quot;, name=&quot;block3_conv1&quot;)(x)
</code></pre>
<p>into 2 layers where the 1st one can be accessed before activations:</p>

<pre class="lang-python prettyprint-override"><code>x = layers.Conv2D(256, (3, 3), activation=None, padding=&quot;same&quot;, name=&quot;block3_conv1&quot;)(x)
x = layers.ReLU(name=&quot;block3_conv1_relu&quot;)(x)
</code></pre>
<p>Is there a way to acces pre-activation tensors in a Model without essentially editing Tensorflow 2 source code, or reverting to Tensorflow 1 which had full flexibility accessing intermediates?</p>