<p>General answer here is to try all of them and select the one, which performs best on the validation.</p>
<p>As for EvoNorm from <a href="https://arxiv.org/abs/2004.02967" rel="nofollow noreferrer">this paper</a>, it depends on your problem. Authors tested the new layer on classification problem with limited set of models. For image synthesis results weren't as good as for classification.</p>
<p>In my opinion, batchnorm is a good starting point to construct baseline solution, because it is time tested, and then try more advanced things.</p>
What is the use of Batch Normalization Layer and Evolving normalization activation layers
<p>When to decide that we need a batch or evolving layer and how do we decide it?
I am currently using PyTorch and I want to understand how I can decide which layer to add?</p>