<p>I suspect this comes from two different uses of the word <a href="https://openreview.net/pdf/25b8eee6c373d48b84e5e9c6e10e7cbbbce4ac73.pdf" rel="nofollow noreferrer">'tape'</a> in the context of automatic differentiation.</p>
<p>When people say that <a href="/questions/tagged/pytorch" class="post-tag" title="show questions tagged &#39;pytorch&#39;" rel="tag">pytorch</a> is not tape-based, they mean it uses Operator Overloading as opposed to [tape-based] Source Transformation for automatic differentiation.</p>
<blockquote>
<p>[<strong>Operator overloading</strong>] relies on a language’s ability to redefine the meaning of functions and operators. All primitives are overloaded so that they additionally perform a tracing operation: The primitive is logged onto a ‘tape’, along with its inputs to ensure that those intermediate variables are kept alive. At the end of the function’s execution, this tape contains a linear trace of all the numerical operations in the program. Derivatives can be calculated by walking this tape in reverse. [...]<br />
OO is the technique used by PyTorch, Autograd, and Chainer [37].</p>
<p>...</p>
<p><strong>Tape-based</strong> Frameworks such as ADIFOR [8] and Tapenade [20] for Fortran and C use a global stack also called a ‘tape’<sup>2</sup> to ensure that intermediate variables are kept alive. The original (primal) function is augmented so that it writes intermediate variables to the tape during the forward pass, and the adjoint program will read intermediate variables from the tape during the backward pass. More recently, tape-based ST was implemented for Python in the ML framework Tangent [38].</p>
<p>...</p>
<p><sup><sup>2</sup> The tape used in ST stores only the intermediate variables, whereas the tape in OO is a program trace that stores the executed primitives as well.</sup></p>
</blockquote>
<ul>
<li><a href="https://arxiv.org/abs/1810.11530" rel="nofollow noreferrer">Automatic differentiation in ML: Where we are and where we should be going</a></li>
</ul>
What is tape-based autograd in Pytorch?
<p>I understand <code>autograd</code> is used to imply automatic differentiation. But what exactly is <code>tape-based autograd</code> in <code>Pytorch</code> and why there are so many discussions that affirm or deny it.</p>
<p>For example:</p>
<p><a href="https://discuss.pytorch.org/t/is-pytorch-autograd-tape-based/13992" rel="noreferrer">this</a></p>
<blockquote>
<p>In pytorch, there is no traditional sense of tape</p>
</blockquote>
<p>and <a href="https://discuss.pytorch.org/t/get-the-gradient-tape/62886" rel="noreferrer">this</a></p>
<blockquote>
<p>We don’t really build gradient tapes per se. But graphs.</p>
</blockquote>
<p>but not <a href="https://pytorch.org/tutorials/beginner/former_torchies/autograd_tutorial.html" rel="noreferrer">this</a></p>
<blockquote>
<p>Autograd is now a core torch package for automatic differentiation. It
uses a tape based system for automatic differentiation.</p>
</blockquote>
<p>And for further reference, please compare it with <code>GradientTape</code> in <code>Tensorflow</code>.</p>