<p>I created a repo that converts Inception v3 batch-normalized weights to the de-normalized weights needed by MPSCNNConvolution.</p>

<p><a href="https://github.com/kakugawa/MetalCNNWeights" rel="noreferrer">https://github.com/kakugawa/MetalCNNWeights</a></p>

<p>In the paper 'Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift' by Sergey Ioffe and Christian Szegedy (<a href="https://arxiv.org/pdf/1502.03167v3.pdf" rel="noreferrer">https://arxiv.org/pdf/1502.03167v3.pdf</a>), we can use Algorithm 2, Output, Step 11 to derive:</p>

<pre><code>[Weight = \frac{\gamma}{\sqrt{Var[x]+0.001}} * Weight_{BN}](http://mathurl.com/z7snq3z.png)
[Bias = \beta - (\frac{\gamma}{\sqrt{Var[x]+0.001}}) * E[x]](http://mathurl.com/zo4shhf.png)
</code></pre>

<p>Note: I need 10 reputation points before I can post images or 2 links.</p>
In the MetalImageRecognition sample code provided by Apple, how is the batch normalization performed on the weights and biases parameters?
<p>The README provides the following comment:</p>

<pre><code>/* 
The weights for this particular network were batch normalized but for inference we may use : 

w =  gamma  / √(s + 0.001), b = ß - ( A * m ) 

s: variance 
m: mean 
gamma : gamma 
ß: beta 

w: weights of a feature channel 
b: bias of a feature channel 

for every feature channel separately to get the corresponding weights and  bias 
*/  
</code></pre>

<p>I have been able to export all of the trained parameters from a retrained Inception model to binary, using TensorFlow. For example, for the first convolution node, these are the available binary files:</p>

<blockquote>
  <p>conv0/BatchNorm/beta
  conv0/BatchNorm/beta/ExponentialMovingAverage
  conv0/BatchNorm/beta/RMSProp
  conv0/BatchNorm/beta/RMSProp_1
  conv0/BatchNorm/moving_mean
  conv0/BatchNorm/moving_mean/ExponentialMovingAverage
  conv0/BatchNorm/moving_variance
  conv0/BatchNorm/moving_variance/ExponentialMovingAverage
  conv0/weights
  conv0/weights/ExponentialMovingAverage
  conv0/weights/Regularizer/L2Loss/value/avg
  conv0/weights/RMSProp
  conv0/weights/RMSProp_1</p>
</blockquote>

<p>Are these files transformed or recomputed to get the corresponding <strong>conv.dat</strong> file somehow, or is there a function in TensorFlow to export each node with batch normalization?</p>

<p>Any additional direction would be extremely helpful, as there are few resources to connect the dots here.</p>

<p>Thank you.</p>