torch/csrc/jit/autodiff.cpp
// If we have a function y = f(x) with jacobian J, the backwards of f is dx = J^t dy.
// Note that because the backwards always implements this matrix multiply,
// we know that it maps an input vector of zeros to an output vector of zero
// regardless of what operations it choses to do inside to actually implement
// the matrix multiply (most use some optimized form and never generate J^t).
// More generally, we know that all of the backward computations are linear and
// can use this property to do more aggressive optimizations later.
// It is ok to replace any backward function with known-zero inputs with something
// that produces known-zero outputs. This function encloses each know-linear
// backward function in a 'GradOf' sub-block so that we can perform optimizations
// using this information. In particular, specializeUndef will observe if
// all the inputs to the linear block are Undef, which the autograd uses to represent
// zeros, and then propagate the undefs to the outputs of the block.
If we have a function y = f(x) with jacobian J, the backwards of f is dx = J^t dy.
Note that because the backwards always implements this matrix multiply,
we know that it maps an input vector of zeros to an output vector of zero
regardless of what operations it choses to do inside to actually implement
the matrix multiply (most use some optimized form and never generate J^t).
More generally, we know that all of the backward computations are linear and
can use this property to do more aggressive optimizations later.
It is ok to replace any backward function with known-zero inputs with something
that produces known-zero outputs. This function encloses each know-linear
backward function in a 'GradOf' sub-block so that we can perform optimizations
using this information. In particular, specializeUndef will observe if
all the inputs to the linear block are Undef, which the autograd uses to represent
zeros, and then propagate the undefs to the outputs of the block.
If we have a function y = f ( x ) with jacobian J the backwards of f is dx = J^t dy . Note that because the backwards always implements this matrix multiply we know that it maps an input vector of zeros to an output vector of zero regardless of what operations it choses to do inside to actually implement the matrix multiply ( most use some optimized form and never generate J^t ) . More generally we know that all of the backward computations are linear and can use this property to do more aggressive optimizations later . It is ok to replace any backward function with known - zero inputs with something that produces known - zero outputs . This function encloses each know - linear backward function in a ' GradOf ' sub - block so that we can perform optimizations using this information . In particular specializeUndef will observe if all the inputs to the linear block are Undef which the autograd uses to represent zeros and then propagate the undefs to the outputs of the block .
static std::vector<Value*> linearGradientForNode(Node* node, ArrayRef<Value*> grad_values) {
  auto & graph = *node->owningGraph();
  auto linear = graph.insertNode(graph.create(prim::GradOf, {grad_values}, 0));
  linear->s_(attr::name, node->kind().toDisplayString());
  auto block = linear->addBlock();
  WithInsertPoint guard(block);
  auto results = gradientForNode(node, grad_values);
  return fmap(results, [block, linear](Value *grad) -> Value* {
    if (!grad) return nullptr;
    block->registerOutput(grad);
    return linear->addOutput()->copyMetadata(grad);
  });
}
static std :: vector < Value * > linearGradientForNode ( Node * node , ArrayRef < Value * > grad_values ) { auto & graph = * node -> owningGraph ( ) ; auto linear = graph . insertNode ( graph . create ( prim :: GradOf , { grad_values } , 0 ) ) ; linear -> s_ ( attr :: name , node -> kind ( ) . toDisplayString ( ) ) ; auto block = linear -> addBlock ( ) ; WithInsertPoint guard ( block ) ; auto results = gradientForNode ( node , grad_values ) ; return fmap ( results , [ block , linear ] ( Value * grad ) -> Value * { if ( ! grad ) return nullptr ; block -> registerOutput ( grad ) ; return linear -> addOutput ( ) -> copyMetadata ( grad ) ; } ) ; }
If we have a function y = f(x) with jacobian J, the backwards of f is dx = J^t dy.
If we have a function y = f ( x ) with jacobian J the backwards of f is dx = J^t dy .
// to make reading gradient graphs easier, remember the name of the forward op