<p>There is a paper evaluating these choices, which can be found here: <a href="https://arxiv.org/pdf/1606.02228.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1606.02228.pdf</a>. They do get better accuracy by using PReLU, but that is very minor. I am unsure if the improvement offsets the higher workload you have to do by using PReLU instead of ReLU. The question is are you already evaluating for that last percentage point in accuracy? If not do not bother yet with choices that only have minor impact on the performance of the model.</p>
Can we improve performance by using BatchNorm with PReLU?
<p>I have a network as follows</p>

<pre><code>BN-Scale-ReLU
</code></pre>

<p>I want to replace ReLU by PReLU. Then, it will be</p>

<pre><code>BN-Scale-PReLU
</code></pre>

<p>Could I obtain any gain with the second setting? Why? As I search,The second setting is not so popular. In some paper, they replaced BN-Scale-ReLU=PReLU. Is it right? </p>