\draw (2.75, 2.5) node {$\times$};

      % Spacing fuck you
      \draw (0, -1);
    \end{tikzpicture}
  \end{subfigure}
  %
  \begin{subfigure}[h]{0.5\textwidth}
    \centering
    \begin{tikzpicture}
      % First operation
      \draw [very thick] (1, 1.8) circle [radius=0.4cm] node {$z$};

      % Input Nodes
      \draw [very thick] (0, 0) circle [radius=0.4cm] node {$x$};
      \draw [very thick] (2, 0) circle [radius=0.4cm] node {$y$};

      % Edges
      \draw [very thick, -stealth] (0, 0.4) -- ++(60:1.3);
      \draw [very thick, -stealth] (2, 0.4) -- ++(120:1.3);

      % Operation Label
      \draw (1, 1.2) node {$+$};

      % Square
      \draw [very thick] (1, 3.6) circle [radius=0.4cm] node {$z^2$};

      % Operation Label
      \draw (1, 3) node {$\times$};

      % Edges
      \draw [very thick, -stealth] (0.9, 2.2).. controls +(-0.4, 0.4).. +(-0.2, 1.1);
      \draw [very thick, -stealth] (1.1, 2.2).. controls +(+0.4, 0.4).. +(+0.2, 1.1);
    \end{tikzpicture}
  \end{subfigure}
  \caption{An example of how common subgraph elimination is used to transform the
    equations $z = x + y$, $z' = x + y$, $z^2 = z \cdot z'$ to just two
    equations $z = x + y$ and $z^2 = z \cdot z$. This computation could
    theoretically be optimized further to a \texttt{square} operation requiring
    only one input (thus reducing the cost of data movement), though it is not
    known if TensorFlow employs such secondary canonicalization.}
  \label{fig:subgraph-elim}
\end{figure}

\subsubsection{Scheduling}\label{sec:model-optim-schedule}

A simple yet powerful optimization is to schedule node execution as late as
possible. Ensuring that the results of operations remain in memory only for the
minimum required amount of time reduces peak memory consumption and can thus
greatly improve the overall performance of the system. The authors of
\cite{tensorflow} note that this is especially vital on devices such as GPUs,
where memory resources are scarce. Furthermore, careful scheduling also pertains
to the activation of \texttt{send} and \texttt{recv} nodes, where not only
memory but also network resources are contested.

\subsubsection{Lossy Compression}\label{sec:model-optim-lossy}

One of the primary goals of many machine learning algorithms used for
classification, recognition or other tasks is to build robust models. With
\emph{robust} we mean that an optimally trained model should ideally not change
its response if it is first fed a signal and then a noisy variation of that
signal. As such, these machine learning algorithms typically do not require high
precision arithmetic as provided by standard IEEE 754 32-bit floating point
values. Rather, 16 bits of precision in the mantissa would do just as well. For
this reason, another optimization performed by TensorFlow is the internal
addition of conversion nodes to the computational graph, which convert such
high-precision 32-bit floating-point values to truncated 16-bit representations
when communicating across devices and across machines. On the receiving end, the
truncated representation is converted back to 32 bits simply by filling in
zeros, rather than rounding \cite{tensorflow}.

\subsection{Additions to the Basic Programming Model}\label{sec:model-ext}

Having discussed the basic computational paradigms and execution model of
TensorFlow, we will now review three more advanced topics that we deem highly
relevant for anyone wishing to use TensorFlow to create machine learning
algorithms. First, we discuss how TensorFlow handles \emph{gradient
  back-propagation}, an essential concept for many deep learning
applications. Then, we study how TensorFlow graphs support \emph{control
  flow}. Lastly, we briefly touch upon the topic of \emph{checkpoints}, as they
are very useful for maintenance of large models.

\subsubsection{Back-Propagation Nodes}\label{sec:model-ext-backprop}

In a large number of deep learning and other machine learning algorithms, it is
necessary to compute the gradients of particular nodes of the computational
graph with respect to one or many other nodes. For example, in a neural network,
we may compute the cost $c$ of the model for a given example $x$ by passing that
example through a series of non-linear transformations. If the neural network
consists of two hidden layers represented by functions $f(x;w) = f_x(w)$ and
$g(x;w) = g_x(w)$ with internal weights $w$, we can express the cost for that
example as $c = (f_x \circ g_x)(w) = f_x(g_x(w))$. We would then typically
calculate the gradient $dc/dw$ of that cost with respect to the weights and use
it to update $w$. Often, this is done by means of the \emph{back-propagation}
algorithm, which traverses the graph in reverse to compute the chain rule
$[f_x(g_x(w))]' = f_x'(g_x(w)) \cdot g_x'(w)$.

In \cite{goodfellow2016}, two approaches for back-propagating gradients through
a computational graph are described. The first, which the authors refer to as
\emph{symbol-to-number differentiation}, receives a set of input values and then
computes the \emph{numerical values} of the gradients at those input values. It
does so by explicitly traversing the graph first in the forward order
(forward-propagation) to compute the cost, then in reverse order
(back-propagation) to compute the gradients via the chain rule. Another
approach, more relevant to TensorFlow, is what \cite{goodfellow2016} calls
\emph{symbol-to-symbol derivatives} and \cite{tensorflow} terms \emph{automatic
  gradient computation}. In this case, gradients are not computed by an explicit
implementation of the back-propagation algorithm. Rather, special nodes are added
to the computational graph that calculate the gradient of each operation and
thus ultimately the chain rule. To perform back-propagation, these nodes must
then simply be executed like any other nodes by the graph evaluation engine. As
such, this approach does not produce the desired derivatives as a numeric value,
but only as a \emph{symbolic handle} to compute those values.

When TensorFlow needs to compute the gradient of a particular node $\nu$ with
respect to some other tensor $\alpha$, it traverses the graph in reverse order
from $\nu$ to $\alpha$. Each operation $o$ encountered during this traversal
represents a function depending on $\alpha$ and is one of the ``links'' in the
chain $(\nu \,\circ\, \dots\, \circ\, o \,\circ\, \dots)(\alpha)$ producing the
output tensor of the graph. Therefore, TensorFlow adds a \emph{gradient node}
for each such operation $o$ that takes the gradient of the previous link (the
outer function) and multiplies it with its own gradient. At the end of the
traversal, there will be a node providing a symbolic handle to the overall
target derivative $\frac{d\nu}{d\alpha}$, which \emph{implicitly} implements the
back-propagation algorithm. It should now be clear that back-propagation in this
symbol-to-symbol approach is \emph{just another operation}, requiring no
exceptional handling. Figure \ref{fig:gradients} shows how a computational graph
may look before and after gradient nodes are added.

\begin{figure}
  \centering
  \begin{subfigure}[b]{0.2\textwidth}
    \centering
    \begin{tikzpicture}
      % Nodes
      \path [very thick] (0, 0)
            coordinate [draw, circle, text width=0.5cm] (w) node {$w$};
      \path [very thick] (0, 1.5)
             coordinate [draw, circle, text width=0.5cm] (x) node {$x$};
      \path [very thick] (0, 3)
            coordinate [draw, circle, text width=0.5cm] (y) node {$y$};
      