<p>Because it was proposed this way. Residual Connections have been investigated in the following work: <a href="https://arxiv.org/pdf/1603.05027.pdf" rel="noreferrer">https://arxiv.org/pdf/1603.05027.pdf</a> and they have found, that Skip -> BN -> RELU -> Conv -> BN -> RELU -> Conv -> Add works best. </p>

<p>However, the differences in performance are negligible and therefore the original ResNet formulation prevailed. Still, you can read the paper if you want to know what works and what does not.</p>
Why is ReLU applied after residual connection in ResNet?
<p>In the ResNet architecture, why is the ReLU activation applied after the element-wise addition with the residual in a residual block, instead of before it?</p>