(red). In the autoencoder network the single data layer is rendered twice (top and bottom) and the network is cyclic.[]{data-label="fig:fig3"}](fig3.eps){width="5.5in"}

We use a uniform set of notational conventions in all the examples. Variables and parameters live on directed graphs, even in the NMF application. In our neuron-centric scheme the notion of layers arises only in the designation of the network inputs and outputs; in the autoencoder example there is also a code layer. Indices $i$ and $j$ label nodes and $i\to j$ is the label for the edge from $i$ to $j$. The index $k$ is reserved for the data item label. Inference, or feed-forward processing in the network, is always defined by the equations

\[feedforward\] $$\begin{aligned}
y[k,j]&= \sum_i x[k,i]\;w[i\to j]\\
x[k,j]&=f(y[k,j]-b[j]),\end{aligned}$$

where $w$ are the weights of the network and $b$ the bias parameters. The variables $x$ and $y$ will always be post- and pre-activation values of the nodes; the bias and activation function $f$ that relates them is absent in NMF. Note that the $x$ in the general discussion of the RRR algorithm (section \[sec:intro\] and appendix) is a search vector that includes all the variables and parameters in the optimization, not just the node values. On an 