Library/ManagedCuda/CudaDNN/CudaDNNContext.cs
/// <summary>
/// This function performs the forward BatchNormalization layer computation for the inference phase. 
/// This layer is based on the paper "Batch Normalization: Accelerating Deep Network 
/// Training by Reducing Internal Covariate Shift", S. Ioffe, C. Szegedy, 2015.
/// </summary>
/// <param name="mode"> Mode of operation (spatial or per-activation). </param>
/// <param name="alpha"> Pointer to scaling factors (in host memory) used to blend the layer output value with prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue. </param>
/// <param name="beta">Pointer to scaling factors (in host memory) used to blend the layer output value with prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue. </param>
/// <param name="xDesc">Tensor descriptor layer's x data.</param>
/// <param name="x">Pointer in device memory for the layer's x data.</param>
/// <param name="yDesc">Tensor descriptor the layer's y data.</param>
/// <param name="y">Pointer in device memory for the layer's y data.</param>
/// <param name="bnScaleBiasMeanVarDesc">Shared tensor descriptor desc for all the 4 tensors below in the argument list. The dimensions for this tensor descriptor are dependent on the normalization mode.</param>
/// <param name="bnScale">Pointer in device memory for the batch normalization scale parameters (in original paper scale is referred to as gamma).</param>
/// <param name="bnBias">Pointers in device memory for the batch normalization bias parameters (in original paper bias is referred to as beta). Note that bnBias parameter can replace the previous layer's bias parameter for improved efficiency. </param>
/// <param name="estimatedMean">Mean tensor (has the same descriptor as the bias and scale). It is suggested that resultRunningMean from the cudnnBatchNormalizationForwardTraining call accumulated during the training phase be passed as input here.</param>
/// <param name="estimatedVariance">Variance tensor (has the same descriptor as the bias and scale). It is suggested that resultRunningVariance from the cudnnBatchNormalizationForwardTraining call accumulated during the training phase be passed as input here.</param>
/// <param name="epsilon">Epsilon value used in the batch normalization formula. Minimum allowed value is currently 1e-5. Same epsilon value should be used in forward and backward functions.</param>
This function performs the forward BatchNormalization layer computation for the inference phase.
This layer is based on the paper "Batch Normalization: Accelerating Deep Network
Training by Reducing Internal Covariate Shift", S. Ioffe, C.
This function performs the forward BatchNormalization layer computation for the inference phase . This layer is based on the paper " Batch Normalization : Accelerating Deep Network Training by Reducing Internal Covariate Shift " S . Ioffe C .
public void BatchNormalizationForwardInference(
                                        cudnnBatchNormMode mode,
                                        double alpha, 
                                        double beta,  
                                        TensorDescriptor xDesc,
                                        CudaDeviceVariable<double> x,     
                                        TensorDescriptor yDesc,
                                        CudaDeviceVariable<double> y,     
                                        TensorDescriptor bnScaleBiasMeanVarDesc,
                                        CudaDeviceVariable<double> bnScale,
                                        CudaDeviceVariable<double> bnBias,
                                        CudaDeviceVariable<double> estimatedMean,
                                        CudaDeviceVariable<double> estimatedVariance,
                                        double epsilon)
        {
            res = CudaDNNNativeMethods.cudnnBatchNormalizationForwardInference(
                _handle, mode, ref alpha, ref beta, xDesc.Desc, x.DevicePointer, yDesc.Desc, y.DevicePointer,
                bnScaleBiasMeanVarDesc.Desc, bnScale.DevicePointer, bnBias.DevicePointer, estimatedMean.DevicePointer, estimatedVariance.DevicePointer, epsilon);
            Debug.WriteLine(String.Format("{0:G}, {1}: {2}", DateTime.Now, "cudnnBatchNormalizationForwardInference", res));
            if (res != cudnnStatus.Success) throw new CudaDNNException(res);
        }
public void BatchNormalizationForwardInference ( cudnnBatchNormMode mode , double alpha , double beta , TensorDescriptor xDesc , CudaDeviceVariable < double > x , TensorDescriptor yDesc , CudaDeviceVariable < double > y , TensorDescriptor bnScaleBiasMeanVarDesc , CudaDeviceVariable < double > bnScale , CudaDeviceVariable < double > bnBias , CudaDeviceVariable < double > estimatedMean , CudaDeviceVariable < double > estimatedVariance , double epsilon ) { res = CudaDNNNativeMethods . cudnnBatchNormalizationForwardInference ( _handle , mode , ref alpha , ref beta , xDesc . Desc , x . DevicePointer , yDesc . Desc , y . DevicePointer , bnScaleBiasMeanVarDesc . Desc , bnScale . DevicePointer , bnBias . DevicePointer , estimatedMean . DevicePointer , estimatedVariance . DevicePointer , epsilon ) ; Debug . WriteLine ( String . Format ( " " , DateTime . Now , " " , res ) ) ; if ( res != cudnnStatus . Success ) throw new CudaDNNException ( res ) ; }
This function performs the forward BatchNormalization layer computation for the inference phase.
This function performs the forward BatchNormalization layer computation for the inference phase .
// alpha[0] = result blend factor // beta[0] = dest layer blend factor // NxCxHxW // NxCxHxW
mode
cudnnBatchNormMode
Mode of operation (spatial or per-activation).
Mode of operation ( spatial or per - activation ) .
alpha
double
Pointer to scaling factors (in host memory) used to blend the layer output value with prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.
Pointer to scaling factors ( in host memory ) used to blend the layer output value with prior value in the destination tensor as follows : dstValue = alpha [ 0 ] * resultValue + beta [ 0 ] * priorDstValue .
beta
double
Pointer to scaling factors (in host memory) used to blend the layer output value with prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.
Pointer to scaling factors ( in host memory ) used to blend the layer output value with prior value in the destination tensor as follows : dstValue = alpha [ 0 ] * resultValue + beta [ 0 ] * priorDstValue .
xDesc
TensorDescriptor
Tensor descriptor layer's x data.
Tensor descriptor layer ' s x data .
x
CudaDeviceVariable<double>
Pointer in device memory for the layer's x data.
Pointer in device memory for the layer ' s x data .
yDesc
TensorDescriptor
Tensor descriptor the layer's y data.
Tensor descriptor the layer ' s y data .
y
CudaDeviceVariable<double>
Pointer in device memory for the layer's y data.
Pointer in device memory for the layer ' s y data .
bnScaleBiasMeanVarDesc
TensorDescriptor
Shared tensor descriptor desc for all the 4 tensors below in the argument list. The dimensions for this tensor descriptor are dependent on the normalization mode.
Shared tensor descriptor desc for all the 4 tensors below in the argument list . The dimensions for this tensor descriptor are dependent on the normalization mode .
bnScale
CudaDeviceVariable<double>
Pointer in device memory for the batch normalization scale parameters (in original paper scale is referred to as gamma).
Pointer in device memory for the batch normalization scale parameters ( in original paper scale is referred to as gamma ) .
bnBias
CudaDeviceVariable<double>
Pointers in device memory for the batch normalization bias parameters (in original paper bias is referred to as beta). Note that bnBias parameter can replace the previous layer's bias parameter for improved efficiency.
Pointers in device memory for the batch normalization bias parameters ( in original paper bias is referred to as beta ) . Note that bnBias parameter can replace the previous layer ' s bias parameter for improved efficiency .
estimatedMean
CudaDeviceVariable<double>
Mean tensor (has the same descriptor as the bias and scale). It is suggested that resultRunningMean from the cudnnBatchNormalizationForwardTraining call accumulated during the training phase be passed as input here.
Mean tensor ( has the same descriptor as the bias and scale ) . It is suggested that resultRunningMean from the cudnnBatchNormalizationForwardTraining call accumulated during the training phase be passed as input here .
estimatedVariance
CudaDeviceVariable<double>
Variance tensor (has the same descriptor as the bias and scale). It is suggested that resultRunningVariance from the cudnnBatchNormalizationForwardTraining call accumulated during the training phase be passed as input here.
Variance tensor ( has the same descriptor as the bias and scale ) . It is suggested that resultRunningVariance from the cudnnBatchNormalizationForwardTraining call accumulated during the training phase be passed as input here .
epsilon
double
Epsilon value used in the batch normalization formula. Minimum allowed value is currently 1e-5. Same epsilon value should be used in forward and backward functions.
Epsilon value used in the batch normalization formula . Minimum allowed value is currently 1e - 5 . Same epsilon value should be used in forward and backward functions .