Library/ManagedCuda/CudaDNN/CudaDNNContext.cs
/// <summary>
/// This function performs the forward BatchNormalization layer computation for the training phase. 
/// This layer is based on the paper "Batch Normalization: Accelerating Deep Network Training by 
/// Reducing Internal Covariate Shift", S. Ioffe, C. Szegedy, 2015.
/// </summary>
/// <param name="mode"> Mode of operation (spatial or per-activation). </param>
/// <param name="alpha"> Pointer to scaling factors (in host memory) used to blend the layer output value with prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue. </param>
/// <param name="beta">Pointer to scaling factors (in host memory) used to blend the layer output value with prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue. </param>
/// <param name="xDesc">Tensor descriptor layer's x data.</param>
/// <param name="x">Pointer in device memory for the layer's x data.</param>
/// <param name="yDesc">Tensor descriptor the layer's y data.</param>
/// <param name="y">Pointer in device memory for the layer's y data.</param>
/// <param name="bnScaleBiasMeanVarDesc">Shared tensor descriptor desc for all the 6 tensors below in the argument list. The dimensions for this tensor descriptor are dependent on the normalization mode.</param>
/// <param name="bnScale">Pointer in device memory for the batch normalization scale parameters (in original paper scale is referred to as gamma).</param>
/// <param name="bnBias">Pointers in device memory for the batch normalization bias parameters (in original paper bias is referred to as beta). Note that bnBias parameter can replace the previous layer's bias parameter for improved efficiency. </param>
/// <param name="exponentialAverageFactor">Factor used in the moving average computation runningMean = newMean*factor + runningMean*(1-factor). Use a factor=1/(1+n) at Nth call to the function to get Cumulative Moving Average (CMA) behavior CMA[n] = (x[1]+...+x[n])/n. Since CMA[n+1] = (n*CMA[n]+x[n+1])/(n+1)= ((n+1)*CMA[n]-CMA[n])/(n+1) + x[n+1]/(n+1) = CMA[n]*(1-1/(n+1))+x[n +1]*1/(n+1)</param>
/// <param name="resultRunningMean">Running mean tensor (it has the same descriptor as the bias and scale). If this tensor is initially uninitialized, it is required that exponentialAverageFactor=1 is used for the very first call of a complete training cycle. This is necessary to properly initialize the moving average. Both resultRunningMean and resultRunningInvVariance can be NULL but only at the same time.</param>
/// <param name="resultRunningVariance">Running variance tensor (it has the same descriptor as the bias and scale). If this tensors is initially uninitialized, it is required that exponentialAverageFactor=1 is used for the very first call of a complete training cycle. This is necessary to properly initialize the moving average. Both resultRunningMean and resultRunningInvVariance can be NULL but only at the same time. The value stored in resultRunningInvVariance (or passed as an input in inference mode) is the moving average of the expression 1 / sqrt(eps+variance[x]) where variance is computed either over batch or spatial+batch dimensions depending on the mode. </param>
/// <param name="epsilon">Epsilon value used in the batch normalization formula. Minimum allowed value is currently 1e-5. Same epsilon value should be used in forward and backward functions.</param>
/// <param name="resultSaveMean">Optional cache to save intermediate results computed during the forward pass - these can then be reused to speed up the backward pass. For this to work correctly, the bottom layer data has to remain unchanged until the backward function is called. Note that both resultSaveMean and resultSaveInvVariance can be NULL but only at the same time. It is recommended to use this cache since memory overhead is relatively small because these tensors have a much lower product of dimensions than the data tensors.</param>
/// <param name="resultSaveVariance">Optional cache to save intermediate results computed during the forward pass - these can then be reused to speed up the backward pass. For this to work correctly, the bottom layer data has to remain unchanged until the backward function is called. Note that both resultSaveMean and resultSaveInvVariance can be NULL but only at the same time. It is recommended to use this cache since memory overhead is relatively small because these tensors have a much lower product of dimensions than the data tensors.</param>
This function performs the forward BatchNormalization layer computation for the training phase.
This layer is based on the paper "Batch Normalization: Accelerating Deep Network Training by
Reducing Internal Covariate Shift", S. Ioffe, C.
This function performs the forward BatchNormalization layer computation for the training phase . This layer is based on the paper " Batch Normalization : Accelerating Deep Network Training by Reducing Internal Covariate Shift " S . Ioffe C .
public void BatchNormalizationForwardTraining(
                                cudnnBatchNormMode mode,
                                float alpha, 
                                float beta,  
                                TensorDescriptor xDesc,
                                CudaDeviceVariable<float> x,     
                                TensorDescriptor yDesc,
                                CudaDeviceVariable<float> y,     
                                TensorDescriptor bnScaleBiasMeanVarDesc,
                                CudaDeviceVariable<float> bnScale,
                                CudaDeviceVariable<float> bnBias,
                                double exponentialAverageFactor,
                                CudaDeviceVariable<float> resultRunningMean,
                                CudaDeviceVariable<float> resultRunningVariance,
                                double epsilon,
                                CudaDeviceVariable<float> resultSaveMean,
                                CudaDeviceVariable<float> resultSaveVariance)
        {
            res = CudaDNNNativeMethods.cudnnBatchNormalizationForwardTraining(
                _handle, mode, ref alpha, ref beta, xDesc.Desc, x.DevicePointer, yDesc.Desc, y.DevicePointer,
                bnScaleBiasMeanVarDesc.Desc, bnScale.DevicePointer, bnBias.DevicePointer, exponentialAverageFactor,
                resultRunningMean.DevicePointer, resultRunningVariance.DevicePointer, epsilon, resultSaveMean.DevicePointer, resultSaveVariance.DevicePointer);
            Debug.WriteLine(String.Format("{0:G}, {1}: {2}", DateTime.Now, "BatchNormalizationForwardTraining", res));
            if (res != cudnnStatus.Success) throw new CudaDNNException(res);
        }
public void BatchNormalizationForwardTraining ( cudnnBatchNormMode mode , float alpha , float beta , TensorDescriptor xDesc , CudaDeviceVariable < float > x , TensorDescriptor yDesc , CudaDeviceVariable < float > y , TensorDescriptor bnScaleBiasMeanVarDesc , CudaDeviceVariable < float > bnScale , CudaDeviceVariable < float > bnBias , double exponentialAverageFactor , CudaDeviceVariable < float > resultRunningMean , CudaDeviceVariable < float > resultRunningVariance , double epsilon , CudaDeviceVariable < float > resultSaveMean , CudaDeviceVariable < float > resultSaveVariance ) { res = CudaDNNNativeMethods . cudnnBatchNormalizationForwardTraining ( _handle , mode , ref alpha , ref beta , xDesc . Desc , x . DevicePointer , yDesc . Desc , y . DevicePointer , bnScaleBiasMeanVarDesc . Desc , bnScale . DevicePointer , bnBias . DevicePointer , exponentialAverageFactor , resultRunningMean . DevicePointer , resultRunningVariance . DevicePointer , epsilon , resultSaveMean . DevicePointer , resultSaveVariance . DevicePointer ) ; Debug . WriteLine ( String . Format ( " " , DateTime . Now , " " , res ) ) ; if ( res != cudnnStatus . Success ) throw new CudaDNNException ( res ) ; }
This function performs the forward BatchNormalization layer computation for the training phase.
This function performs the forward BatchNormalization layer computation for the training phase .
// alpha[0] = result blend factor // beta[0] = dest layer blend factor // NxCxHxW // NxCxHxW /* Shared desc for the next 6 tensors in the argument list.
                                   Data type to be set as follows:
                                   type = (typeOf(x) == double) ? double : float
                                   Dimensions for this descriptor depend on normalization mode
                                   - Spatial Normalization : tensors are expected to have dims 1xCx1x1
                                    (normalization is performed across NxHxW)
                                   - Per-Activation Normalization : tensors are expected to have dims of 1xCxHxW 
                                    (normalization is performed across N) */ // 'Gamma' and 'Beta' respectively in Ioffe and Szegedy's paper's notation /* MUST use factor=1 in the very first call of a complete training cycle.
                                   Use a factor=1/(1+n) at N-th call to the function to get
                                   Cumulative Moving Average (CMA) behavior
                                   CMA[n] = (x[1]+...+x[n])/n
                                   Since CMA[n+1] = (n*CMA[n]+x[n+1])/(n+1) =
                                   ((n+1)*CMA[n]-CMA[n])/(n+1) + x[n+1]/(n+1) =
                                   CMA[n]*(1-1/(n+1)) + x[n+1]*1/(n+1) */ /* Used in Training phase only. 
                                   runningMean = newMean*factor + runningMean*(1-factor) */ /* Output in training mode, input in inference. Is the moving average
                                   of 1 / sqrt( epsilon + variance[x] ) */ /* Has to be >= CUDNN_BN_MIN_EPSILON. Should be the same in forward and backward functions. */ /* Optionally save intermediate results from the forward pass here
                                   - can be reused to speed up backward pass. NULL if unused */
mode
cudnnBatchNormMode
Mode of operation (spatial or per-activation).
Mode of operation ( spatial or per - activation ) .
alpha
float
Pointer to scaling factors (in host memory) used to blend the layer output value with prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.
Pointer to scaling factors ( in host memory ) used to blend the layer output value with prior value in the destination tensor as follows : dstValue = alpha [ 0 ] * resultValue + beta [ 0 ] * priorDstValue .
beta
float
Pointer to scaling factors (in host memory) used to blend the layer output value with prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.
Pointer to scaling factors ( in host memory ) used to blend the layer output value with prior value in the destination tensor as follows : dstValue = alpha [ 0 ] * resultValue + beta [ 0 ] * priorDstValue .
xDesc
TensorDescriptor
Tensor descriptor layer's x data.
Tensor descriptor layer ' s x data .
x
CudaDeviceVariable<float>
Pointer in device memory for the layer's x data.
Pointer in device memory for the layer ' s x data .
yDesc
TensorDescriptor
Tensor descriptor the layer's y data.
Tensor descriptor the layer ' s y data .
y
CudaDeviceVariable<float>
Pointer in device memory for the layer's y data.
Pointer in device memory for the layer ' s y data .
bnScaleBiasMeanVarDesc
TensorDescriptor
Shared tensor descriptor desc for all the 6 tensors below in the argument list. The dimensions for this tensor descriptor are dependent on the normalization mode.
Shared tensor descriptor desc for all the 6 tensors below in the argument list . The dimensions for this tensor descriptor are dependent on the normalization mode .
bnScale
CudaDeviceVariable<float>
Pointer in device memory for the batch normalization scale parameters (in original paper scale is referred to as gamma).
Pointer in device memory for the batch normalization scale parameters ( in original paper scale is referred to as gamma ) .
bnBias
CudaDeviceVariable<float>
Pointers in device memory for the batch normalization bias parameters (in original paper bias is referred to as beta). Note that bnBias parameter can replace the previous layer's bias parameter for improved efficiency.
Pointers in device memory for the batch normalization bias parameters ( in original paper bias is referred to as beta ) . Note that bnBias parameter can replace the previous layer ' s bias parameter for improved efficiency .
exponentialAverageFactor
double
Factor used in the moving average computation runningMean = newMean*factor + runningMean*(1-factor). Use a factor=1/(1+n) at Nth call to the function to get Cumulative Moving Average (CMA) behavior CMA[n] = (x[1]+...+x[n])/n.
Factor used in the moving average computation runningMean = newMean * factor + runningMean * ( 1 - factor ) . Use a factor = 1 / ( 1 + n ) at Nth call to the function to get Cumulative Moving Average ( CMA ) behavior CMA [ n ] = ( x [ 1 ] + ... + x [ n ] ) / n .
resultRunningMean
CudaDeviceVariable<float>
Running mean tensor (it has the same descriptor as the bias and scale). If this tensor is initially uninitialized, it is required that exponentialAverageFactor=1 is used for the very first call of a complete training cycle. This is necessary to properly initialize the moving average. Both resultRunningMean and resultRunningInvVariance can be NULL but only at the same time.
Running mean tensor ( it has the same descriptor as the bias and scale ) . If this tensor is initially uninitialized it is required that exponentialAverageFactor = 1 is used for the very first call of a complete training cycle . This is necessary to properly initialize the moving average . Both resultRunningMean and resultRunningInvVariance can be NULL but only at the same time .
resultRunningVariance
CudaDeviceVariable<float>
Running variance tensor (it has the same descriptor as the bias and scale). If this tensors is initially uninitialized, it is required that exponentialAverageFactor=1 is used for the very first call of a complete training cycle. This is necessary to properly initialize the moving average. Both resultRunningMean and resultRunningInvVariance can be NULL but only at the same time. The value stored in resultRunningInvVariance (or passed as an input in inference mode) is the moving average of the expression 1 / sqrt(eps+variance[x]) where variance is computed either over batch or spatial+batch dimensions depending on the mode.
Running variance tensor ( it has the same descriptor as the bias and scale ) . If this tensors is initially uninitialized it is required that exponentialAverageFactor = 1 is used for the very first call of a complete training cycle . This is necessary to properly initialize the moving average . Both resultRunningMean and resultRunningInvVariance can be NULL but only at the same time . The value stored in resultRunningInvVariance ( or passed as an input in inference mode ) is the moving average of the expression 1 / sqrt ( eps + variance [ x ] ) where variance is computed either over batch or spatial + batch dimensions depending on the mode .
epsilon
double
Epsilon value used in the batch normalization formula. Minimum allowed value is currently 1e-5. Same epsilon value should be used in forward and backward functions.
Epsilon value used in the batch normalization formula . Minimum allowed value is currently 1e - 5 . Same epsilon value should be used in forward and backward functions .
resultSaveMean
CudaDeviceVariable<float>
Optional cache to save intermediate results computed during the forward pass - these can then be reused to speed up the backward pass. For this to work correctly, the bottom layer data has to remain unchanged until the backward function is called. Note that both resultSaveMean and resultSaveInvVariance can be NULL but only at the same time. It is recommended to use this cache since memory overhead is relatively small because these tensors have a much lower product of dimensions than the data tensors.
Optional cache to save intermediate results computed during the forward pass - these can then be reused to speed up the backward pass . For this to work correctly the bottom layer data has to remain unchanged until the backward function is called . Note that both resultSaveMean and resultSaveInvVariance can be NULL but only at the same time . It is recommended to use this cache since memory overhead is relatively small because these tensors have a much lower product of dimensions than the data tensors .
resultSaveVariance
CudaDeviceVariable<float>
Optional cache to save intermediate results computed during the forward pass - these can then be reused to speed up the backward pass. For this to work correctly, the bottom layer data has to remain unchanged until the backward function is called. Note that both resultSaveMean and resultSaveInvVariance can be NULL but only at the same time. It is recommended to use this cache since memory overhead is relatively small because these tensors have a much lower product of dimensions than the data tensors.
Optional cache to save intermediate results computed during the forward pass - these can then be reused to speed up the backward pass . For this to work correctly the bottom layer data has to remain unchanged until the backward function is called . Note that both resultSaveMean and resultSaveInvVariance can be NULL but only at the same time . It is recommended to use this cache since memory overhead is relatively small because these tensors have a much lower product of dimensions than the data tensors .