Library/ManagedCuda/CudaDNN/CudaDNNContext.cs
/// <summary>
/// This function performs the backward BatchNormalization layer computation.
/// </summary>
/// <param name="mode"> Mode of operation (spatial or per-activation). </param>
/// <param name="alphaDataDiff">Pointer to scaling factors in host memory used to blend the gradient output dx with a prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.</param>
/// <param name="betaDataDiff">Pointer to scaling factors in host memory used to blend the gradient output dx with a prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.</param>
/// <param name="alphaParamDiff">Pointer to scaling factors (in host memory) used to blend the gradient outputs dBnScaleResult and dBnBiasResult with prior values in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.</param>
/// <param name="betaParamDiff">Pointer to scaling factors (in host memory) used to blend the gradient outputs dBnScaleResult and dBnBiasResult with prior values in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.</param>
/// <param name="xDesc">Tensor descriptor for the layer's x data.</param>
/// <param name="x">Pointers in device memory for the layer's x data.</param>
/// <param name="dyDesc">Tensor descriptor for the layer's backpropagated differential dy (inputs).</param>
/// <param name="dy">Pointers in device memory for the layer's backpropagated differential dy (inputs).</param>
/// <param name="dxDesc">Tensor descriptor for the layer's resulting differential with respect to x, dx (output).</param>
/// <param name="dx">Pointer in device memory for the layer's resulting differential with respect to x, dx (output).</param>
/// <param name="dBnScaleBiasDesc">Shared tensor descriptor for all the 5 tensors below in the argument list (bnScale, resultBnScaleDiff, resultBnBiasDiff, savedMean, savedInvVariance). The dimensions for this tensor descriptor are dependent on normalization mode. Note: The data type of this tensor descriptor must be 'float' for FP16 and FP32 input tensors, and 'double' for FP64 input tensors.</param>
/// <param name="bnScale">Pointers in device memory for the batch normalization scale parameter (in original paper bias is referred to as gamma). Note that bnBias parameter is not needed for this layer's computation.</param>
/// <param name="dBnScaleResult">Pointer in device memory for the resulting scale differentials computed by this routine. Note that scale and bias gradients are not backpropagated below this layer (since they are dead-end computation DAG nodes).</param>
/// <param name="dBnBiasResult">Pointer in device memory for the resulting bias differentials computed by this routine. Note that scale and bias gradients are not backpropagated below this layer (since they are dead-end computation DAG nodes).</param>
/// <param name="epsilon">Epsilon value used in the batch normalization formula. Minimum allowed value is currently 1e-5. Same epsilon value should be used in forward and backward functions.</param>
/// <param name="savedMean">Optional cache parameter saved intermediate results computed during the forward pass. For this to work correctly, the layer's x and bnScale, bnBias data has to remain unchanged until the backward function is called. Note that both savedMean and savedInvVariance parameters can be NULL but only at the same time. It is recommended to use this cache since the memory overhead is relatively small.</param>
/// <param name="savedInvVariance">Optional cache parameter saved intermediate results computed during the forward pass. For this to work correctly, the layer's x and bnScale, bnBias data has to remain unchanged until the backward function is called. Note that both savedMean and savedInvVariance parameters can be NULL but only at the same time. It is recommended to use this cache since the memory overhead is relatively small.</param>
This function performs the backward BatchNormalization layer computation.
This function performs the backward BatchNormalization layer computation .
public void BatchNormalizationBackward(
                                        cudnnBatchNormMode mode,
                                        double alphaDataDiff,
                                        double betaDataDiff,
                                        double alphaParamDiff,
                                        double betaParamDiff,
                                        TensorDescriptor xDesc, 
                                        CudaDeviceVariable<double> x,
                                        TensorDescriptor dyDesc,
                                        CudaDeviceVariable<double> dy,
                                        TensorDescriptor dxDesc,
                                        CudaDeviceVariable<double> dx,
                                        TensorDescriptor dBnScaleBiasDesc,
                                        CudaDeviceVariable<double> bnScale, 
                                        CudaDeviceVariable<double> dBnScaleResult,
                                        CudaDeviceVariable<double> dBnBiasResult,
                                        double epsilon,
                                        CudaDeviceVariable<double> savedMean,
                                        CudaDeviceVariable<double> savedInvVariance)
        {
            res = CudaDNNNativeMethods.cudnnBatchNormalizationBackward(
                _handle, mode, ref alphaDataDiff, ref betaDataDiff, ref alphaParamDiff, ref betaParamDiff,
                xDesc.Desc, x.DevicePointer, dyDesc.Desc, dy.DevicePointer, dxDesc.Desc, dx.DevicePointer,
                dBnScaleBiasDesc.Desc, bnScale.DevicePointer, dBnScaleResult.DevicePointer, dBnBiasResult.DevicePointer,
                epsilon, savedMean.DevicePointer, savedInvVariance.DevicePointer);
            Debug.WriteLine(String.Format("{0:G}, {1}: {2}", DateTime.Now, "BatchNormalizationBackward", res));
            if (res != cudnnStatus.Success) throw new CudaDNNException(res);
        }
public void BatchNormalizationBackward ( cudnnBatchNormMode mode , double alphaDataDiff , double betaDataDiff , double alphaParamDiff , double betaParamDiff , TensorDescriptor xDesc , CudaDeviceVariable < double > x , TensorDescriptor dyDesc , CudaDeviceVariable < double > dy , TensorDescriptor dxDesc , CudaDeviceVariable < double > dx , TensorDescriptor dBnScaleBiasDesc , CudaDeviceVariable < double > bnScale , CudaDeviceVariable < double > dBnScaleResult , CudaDeviceVariable < double > dBnBiasResult , double epsilon , CudaDeviceVariable < double > savedMean , CudaDeviceVariable < double > savedInvVariance ) { res = CudaDNNNativeMethods . cudnnBatchNormalizationBackward ( _handle , mode , ref alphaDataDiff , ref betaDataDiff , ref alphaParamDiff , ref betaParamDiff , xDesc . Desc , x . DevicePointer , dyDesc . Desc , dy . DevicePointer , dxDesc . Desc , dx . DevicePointer , dBnScaleBiasDesc . Desc , bnScale . DevicePointer , dBnScaleResult . DevicePointer , dBnBiasResult . DevicePointer , epsilon , savedMean . DevicePointer , savedInvVariance . DevicePointer ) ; Debug . WriteLine ( String . Format ( " " , DateTime . Now , " " , res ) ) ; if ( res != cudnnStatus . Success ) throw new CudaDNNException ( res ) ; }
This function performs the backward BatchNormalization layer computation.
This function performs the backward BatchNormalization layer computation .
// same desc for x, dx, dy /* Shared tensor desc for the 5 tensors below */ // bnBias doesn't affect backpropagation /* scale and bias diff are not backpropagated below this layer */ /* Same epsilon as forward pass */ /* Optionally cached intermediate results from
                                           forward pass */
mode
cudnnBatchNormMode
Mode of operation (spatial or per-activation).
Mode of operation ( spatial or per - activation ) .
alphaDataDiff
double
Pointer to scaling factors in host memory used to blend the gradient output dx with a prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.
Pointer to scaling factors in host memory used to blend the gradient output dx with a prior value in the destination tensor as follows : dstValue = alpha [ 0 ] * resultValue + beta [ 0 ] * priorDstValue .
betaDataDiff
double
Pointer to scaling factors in host memory used to blend the gradient output dx with a prior value in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.
Pointer to scaling factors in host memory used to blend the gradient output dx with a prior value in the destination tensor as follows : dstValue = alpha [ 0 ] * resultValue + beta [ 0 ] * priorDstValue .
alphaParamDiff
double
Pointer to scaling factors (in host memory) used to blend the gradient outputs dBnScaleResult and dBnBiasResult with prior values in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.
Pointer to scaling factors ( in host memory ) used to blend the gradient outputs dBnScaleResult and dBnBiasResult with prior values in the destination tensor as follows : dstValue = alpha [ 0 ] * resultValue + beta [ 0 ] * priorDstValue .
betaParamDiff
double
Pointer to scaling factors (in host memory) used to blend the gradient outputs dBnScaleResult and dBnBiasResult with prior values in the destination tensor as follows: dstValue = alpha[0]*resultValue + beta[0]*priorDstValue.
Pointer to scaling factors ( in host memory ) used to blend the gradient outputs dBnScaleResult and dBnBiasResult with prior values in the destination tensor as follows : dstValue = alpha [ 0 ] * resultValue + beta [ 0 ] * priorDstValue .
xDesc
TensorDescriptor
Tensor descriptor for the layer's x data.
Tensor descriptor for the layer ' s x data .
x
CudaDeviceVariable<double>
Pointers in device memory for the layer's x data.
Pointers in device memory for the layer ' s x data .
dyDesc
TensorDescriptor
Tensor descriptor for the layer's backpropagated differential dy (inputs).
Tensor descriptor for the layer ' s backpropagated differential dy ( inputs ) .
dy
CudaDeviceVariable<double>
Pointers in device memory for the layer's backpropagated differential dy (inputs).
Pointers in device memory for the layer ' s backpropagated differential dy ( inputs ) .
dxDesc
TensorDescriptor
Tensor descriptor for the layer's resulting differential with respect to x, dx (output).
Tensor descriptor for the layer ' s resulting differential with respect to x dx ( output ) .
dx
CudaDeviceVariable<double>
Pointer in device memory for the layer's resulting differential with respect to x, dx (output).
Pointer in device memory for the layer ' s resulting differential with respect to x dx ( output ) .
dBnScaleBiasDesc
TensorDescriptor
Shared tensor descriptor for all the 5 tensors below in the argument list (bnScale, resultBnScaleDiff, resultBnBiasDiff, savedMean, savedInvVariance). The dimensions for this tensor descriptor are dependent on normalization mode. Note: The data type of this tensor descriptor must be 'float' for FP16 and FP32 input tensors, and 'double' for FP64 input tensors.
Shared tensor descriptor for all the 5 tensors below in the argument list ( bnScale resultBnScaleDiff resultBnBiasDiff savedMean savedInvVariance ) . The dimensions for this tensor descriptor are dependent on normalization mode . Note : The data type of this tensor descriptor must be ' float ' for FP16 and FP32 input tensors and ' double ' for FP64 input tensors .
bnScale
CudaDeviceVariable<double>
Pointers in device memory for the batch normalization scale parameter (in original paper bias is referred to as gamma). Note that bnBias parameter is not needed for this layer's computation.
Pointers in device memory for the batch normalization scale parameter ( in original paper bias is referred to as gamma ) . Note that bnBias parameter is not needed for this layer ' s computation .
dBnScaleResult
CudaDeviceVariable<double>
Pointer in device memory for the resulting scale differentials computed by this routine. Note that scale and bias gradients are not backpropagated below this layer (since they are dead-end computation DAG nodes).
Pointer in device memory for the resulting scale differentials computed by this routine . Note that scale and bias gradients are not backpropagated below this layer ( since they are dead - end computation DAG nodes ) .
dBnBiasResult
CudaDeviceVariable<double>
Pointer in device memory for the resulting bias differentials computed by this routine. Note that scale and bias gradients are not backpropagated below this layer (since they are dead-end computation DAG nodes).
Pointer in device memory for the resulting bias differentials computed by this routine . Note that scale and bias gradients are not backpropagated below this layer ( since they are dead - end computation DAG nodes ) .
epsilon
double
Epsilon value used in the batch normalization formula. Minimum allowed value is currently 1e-5. Same epsilon value should be used in forward and backward functions.
Epsilon value used in the batch normalization formula . Minimum allowed value is currently 1e - 5 . Same epsilon value should be used in forward and backward functions .
savedMean
CudaDeviceVariable<double>
Optional cache parameter saved intermediate results computed during the forward pass. For this to work correctly, the layer's x and bnScale, bnBias data has to remain unchanged until the backward function is called. Note that both savedMean and savedInvVariance parameters can be NULL but only at the same time. It is recommended to use this cache since the memory overhead is relatively small.
Optional cache parameter saved intermediate results computed during the forward pass . For this to work correctly the layer ' s x and bnScale bnBias data has to remain unchanged until the backward function is called . Note that both savedMean and savedInvVariance parameters can be NULL but only at the same time . It is recommended to use this cache since the memory overhead is relatively small .
savedInvVariance
CudaDeviceVariable<double>
Optional cache parameter saved intermediate results computed during the forward pass. For this to work correctly, the layer's x and bnScale, bnBias data has to remain unchanged until the backward function is called. Note that both savedMean and savedInvVariance parameters can be NULL but only at the same time. It is recommended to use this cache since the memory overhead is relatively small.
Optional cache parameter saved intermediate results computed during the forward pass . For this to work correctly the layer ' s x and bnScale bnBias data has to remain unchanged until the backward function is called . Note that both savedMean and savedInvVariance parameters can be NULL but only at the same time . It is recommended to use this cache since the memory overhead is relatively small .