title: TensorFlow: discarding convolution gradients/parameters at test time
body: <p>Lately I've been reading up on the memory consumed by convolutional neural networks (ConvNets). During training, each convolutional layer has several parameters required for back-propagation of the gradients. <a href="http://cs231n.github.io/convolutional-networks/#comp" rel="nofollow noreferrer">These lecture notes</a> suggest that these parameters can in principle be discarded at test time. A quote from the linked notes:</p>
  
  <blockquote>
    <p>Usually, most of the activations are on the earlier layers of a ConvNet (i.e. first Conv Layers). These are kept around because they are needed for backpropagation, but a clever implementation that runs a ConvNet only at test time could in principle reduce this by a huge amount, by only storing the current activations at any layer and discarding the previous activations on layers below.</p>
  </blockquote>
  
  <p>Is there any way (using TensorFlow) to make use of this "clever implementation" for inference of large batches? Is there some flag that specifies whether or not the model is in its training phase? Or is this already handled automatically based on whether the optimiser function is called?</p>