<p>Before explaining the distinction between tensors and variables, we should be precise about what the word "tensor" means in the context of TensorFlow:</p>

<ul>
<li><p>In the <strong>Python API</strong>, a <a href="https://www.tensorflow.org/versions/r0.12/api_docs/python/framework.html#Tensor" rel="noreferrer"><code>tf.Tensor</code></a> object represents the symbolic result of a TensorFlow operation. For example, in the expression <code>t = tf.matmul(x, y)</code>, <code>t</code> is a <code>tf.Tensor</code> object representing the result of multiplying <code>x</code> and <code>y</code> (which may themselves be symbolic results of other operations, concrete values such as NumPy arrays, or variables).</p>

<p>In this context, a "symbolic result" is more complicated than a pointer to the result of an operation. It is more analogous to a function object that, when called (i.e. passed to <code>tf.Session.run()</code>) will run the necessary computation to produce the result of that operation, and return it to you as a concrete value (e.g. a NumPy array).</p></li>
<li><p>In the <strong>C++ API</strong>, a <a href="https://github.com/tensorflow/tensorflow/blob/5657d0dee8d87f4594b3e5902ed3e3ca8d6dfc0a/tensorflow/core/framework/tensor.h" rel="noreferrer"><code>tensorflow::Tensor</code></a> object represents the concrete value of a multi-dimensional array. For example, the <code>MatMul</code> kernel takes two two-dimensional <code>tensorflow::Tensor</code> objects as inputs, and produces a single two-dimensional <code>tensorflow::Tensor</code> object as its output.</p></li>
</ul>

<p>This distinction is a little confusing, and we might choose different names if we started over (in other language APIs, we prefer the name <code>Output</code> for a symbolic result and <code>Tensor</code> for a concrete value).</p>

<p>A similar distinction exists for variables. In the Python API, a <a href="https://www.tensorflow.org/versions/r0.12/api_docs/python/state_ops.html#Variable" rel="noreferrer"><code>tf.Variable</code></a> is the symbolic representation of a variable, which has methods for creating operations that read the current value of the variable, and assign values to it. In the C++ implementation, a <a href="https://github.com/tensorflow/tensorflow/blob/5657d0dee8d87f4594b3e5902ed3e3ca8d6dfc0a/tensorflow/core/kernels/variable_ops.h#L29" rel="noreferrer"><code>tensorflow::Var</code></a> object is a wrapper around a shared, mutable <code>tensorflow::Tensor</code> object.</p>

<p>With that context out the way, we can address your specific questions:</p>

<ol>
<li><p><strong>What is the meaning of "in-memory buffers"?</strong></p>

<p>An in-memory buffer is simply a contiguous region of memory that has been allocated with a TensorFlow allocator. <code>tensorflow::Tensor</code> objects contain a pointer to an in-memory buffer, which holds the values of that tensor. The buffer could be in host memory (i.e. accessible from the CPU) or device memory (e.g. accessible only from a GPU), and TensorFlow has operations to move data between these memory spaces.</p></li>
<li><p><strong>What is the meaning of a "handle"?</strong> </p>

<p>In the explanation in <a href="https://arxiv.org/pdf/1610.01178.pdf" rel="noreferrer">the paper</a>, the word "handle" is used in a couple of different ways, which are slightly different from how TensorFlow uses the term. The paper uses "symbolic handle" to refer to a <code>tf.Tensor</code> object, and "persistent, mutable handle" to refer to a <code>tf.Variable</code> object. The TensorFlow codebase uses "handle" to refer to a name for a stateful object (like a <code>tf.FIFOQueue</code> or <code>tf.TensorArray</code>) that can be passed around without copying all of the values (i.e. <a href="https://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_reference" rel="noreferrer">call-by-reference</a>).</p></li>
<li><p><strong>Is my initial assumption about the internal of a tensor correct?</strong></p>

<p>Your assumption most closely matches the definition of a (C++) <code>tensorflow::Tensor</code> object. The (Python) <code>tf.Tensor</code> object is more complicated because it refers to a function for computing a value, rather than the value itself.</p></li>
<li><p><strong>What is the essential internal implementation difference between a tensor and a variable?</strong></p>

<p>In C++, a <code>tensorflow::Tensor</code> and <code>tensorflow::Var</code> are very similar; the only different is that <code>tensorflow::Var</code> also has a <code>mutex</code> that can be used to lock the variable when it is being updated.</p>

<p>In Python, the essential difference is that a <code>tf.Tensor</code> is implemented as a dataflow graph, and it is read-only (i.e. by calling <code>tf.Session.run()</code>). A <code>tf.Variable</code> can be both read (i.e. by evaluating its read operation) and written (e.g. by running an assign operation).</p>

<p><strong>Why are they declared differently and why is that difference essential to TensorFlow?</strong></p>

<p>Tensors and variables serve different purposes.  Tensors (<code>tf.Tensor</code> objects) can represent complex compositions of mathematical expressions, like loss functions in a neural network, or symbolic gradients. Variables represent state that is updated over time, like weight matrices and convolutional filters during training. While in principle you could represent the evolving state of a model without variables, you would end up with a very large (and repetetive) mathematical expression, so variables provide a convenient way to materialize the state of the model, and&mdash;for example&mdash;share it with other machines for parallel training.</p></li>
</ol>
Implementation difference between TensorFlow Variable and TensorFlow Tensor
<p>First of all, I am aware that a related question has been asked <a href="https://stackoverflow.com/questions/38556078/in-tensorflow-what-is-the-difference-between-a-variable-and-a-tensor">here</a>.</p>

<p>However, this question is about the implementation and internals. 
I was reading the paper "<a href="https://arxiv.org/pdf/1610.01178.pdf" rel="noreferrer">A Tour of TensorFlow</a>". The following two points are quoted from there:</p>

<p>1.</p>

<blockquote>
  <p>A tensor itself does not hold or store values in memory, but provides
  only an interface for retrieving the value referenced by the tensor.</p>
</blockquote>

<p>This suggests to me that a Tensor is an object that simply stores the pointer to a result of an operation and, on retrieving the result or value of the tensor, it simply dereferences that pointer.</p>

<p>2.</p>

<blockquote>
  <p>Variables can be described as persistent, mutable handles to in-memory buffers storing tensors. As such, variables are  characterized by a certain shape and a fixed type.</p>
</blockquote>

<p>At this I get confused because I thought, based on the previous point, that Tensors simply store a pointer. If they were simply pointers, they could be mutable as well.</p>

<p>To be precise these are my questions:</p>

<ol>
<li>What is the meaning of "in-memory buffers"?</li>
<li>What is the meaning of a "handle"?</li>
<li>Is my initial assumption about the internals of a tensor correct? </li>
<li>What is the essential internal implementation difference between a tensor and a variable? Why are they declared differently  and why is that difference essential to TensorFlow?</li>
</ol>