KirchhoffNet: A Scalable Ultra Fast Analog Neural Network
In this paper, we leverage a foundational principle of analog electronic
circuitry, Kirchhoff's current and voltage laws, to introduce a distinctive
class of neural network models termed KirchhoffNet. Essentially, KirchhoffNet
is an analog circuit that can function as a neural network, utilizing its
initial node voltages as the neural network input and the node voltages at a
specific time point as the output. The evolution of node voltages within the
specified time is dictated by learnable parameters on the edges connecting
nodes. We demonstrate that KirchhoffNet is governed by a set of ordinary
differential equations (ODEs), and notably, even in the absence of traditional
layers (such as convolution layers), it attains state-of-the-art performances
across diverse and complex machine learning tasks. Most importantly,
KirchhoffNet can be potentially implemented as a low-power analog integrated
circuit, leading to an appealing property -- irrespective of the number of
parameters within a KirchhoffNet, its on-chip forward calculation can always be
completed within a short time. This characteristic makes KirchhoffNet a
promising and fundamental paradigm for implementing large-scale neural
networks, opening a new avenue in analog neural networks for AI.
Zhengqi Gao, Fan-Keng Sun, Ron Rohrer, Duane S. Boning
# KirchhoffNet: A Circuit

###### Abstract

In this paper, we exploit a fundamental principle of analog electronic circuitry, Kirchhoff's current law, to introduce a unique class of neural network models that we refer to as _KirchhoffNet_. KirchhoffNet establishes close connections with message passing neural networks and continuous-depth networks. We demonstrate that even in the absence of any traditional layers (such as convolution, pooling, or linear layers), KirchhoffNet attains 98.86% test accuracy on the MNIST dataset, comparable with state of the art (SOTA) results.

What makes KirchhoffNet more intriguing is its potential in the realm of hardware. Contemporary deep neural networks are conventionally deployed on GPUs. In contrast, KirchhoffNet can be physically realized by an analog electronic circuit. Moreover, we justify that irrespective of the number of parameters within a KirchhoffNet, its forward calculation can always be completed within \(1/f\) seconds, with \(f\) representing the hardware's clock frequency. This characteristic introduces a promising technology for implementing ultra-large-scale neural networks.

Machine Learning, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Convolutional Neural Networks, Neural Networks, Convolutional
## 2 Method

### KCL as Message Passing Criterion

KirchhoffNet or a circuit 2 is best described by a directed graph \(\mathcal{G}=(\mathcal{V},\mathcal{E})\), where \(\mathcal{V}=\{n_{0},n_{1},\cdots,n_{N}\}\) represents \((N+1)\) nodes, and \(\mathcal{E}\) specifies the connections (with directions) among nodes. The \(n\)-th node is associated with a scalar _nodal voltage_\(v_{n}\in\mathbb{R}\), where \(n=\{0,1,\cdots,N\}\). Notably, we emphasize that the node \(n_{0}\) holds a special designation and is referred to as a _ground node_, as its associated \(v_{0}\) is fixed to \(0\), following the convention of circuit theory (Desoer & Kuh, 1969; Van Valkenburg, 1974). For later simplicity, we denote \(\mathbf{v}=[v_{1},v_{2},\cdots,v_{N}]^{T}\in\mathbb{R}^{N}\). If the set \(\mathcal{E}\) contains an edge from the source node \(n_{s}\) to the destination node \(n_{d}\), then a _branch current_\(i_{sd}\in\mathbb{R}\) will be generated flowing from \(n_{s}\) to \(n_{d}\). Now, KCL states that:

Footnote 2: In this paper, we will interchangeably use these two terms.

\[\sum_{s\in\mathcal{N}^{s}(n_{j})}i_{sj}=\sum_{d\in\mathcal{N}^{d}(n_{j})}i_{jd} \tag{1}\]

where \(\mathcal{N}^{s}(n_{j})\) represents all nodes that are connected to \(n_{j}\) and are also the source nodes for the connections. Similarly, \(\mathcal{N}^{d}(n_{j})\) represents all nodes that are connected to \(n_{j}\) and are the destination. In other words, KCL states that the sum of branch currents flowing into the node \(n_{j}\), i.e., the left-hand side of Eq. (1), equals the sum of branch currents flowing out of the node \(n_{j}\), i.e., the right-hand side of Eq. (1).

Physically, the branch current \(i_{sd}\) is generated by a _device_ connecting from \(n_{s}\) to \(n_{d}\), and the value of \(i_{sd}\) is related to the device parameter \(\theta\). For instance, several straightforward current-voltage relations are:

\[\text{Source branch:}\quad i_{sd}=\theta \tag{2}\] \[\text{Conductive branch:}\quad i_{sd}=\theta(v_{s}-v_{d})\] (3) \[\text{Capacitive branch:}\quad i_{sd}=\theta(\dot{v}_{s}-\dot{v}_{ d}) \tag{4}\]

where we use \(\dot{v}=dv/dt\) to represent the derivative of a nodal voltage \(v\) with respect to the time variable \(t\). 3 Drawing inspiration from circuit theory (Desoer & Kuh, 1969; Van Valkenburg, 1974), we know that such a KirchhoffNet \(\mathcal{G}\) with device current-voltage relation shown in Eq. (3) and governed by KCL shown in Eq. (1) can be formally solved by \(\mathbf{C}\mathbf{v}+\mathbf{G}\mathbf{v}=\mathbf{0}\), where \(\{\mathbf{C}\in\mathbb{R}^{N\times N},\mathbf{G}\in\mathbb{R}^{N\times N}\}\) are determined by all parameters \(\theta\) on all edges (devices).

Footnote 3: Note that the first, the second, and the third line are respectively a current source, a conductor and a capacitance with parameter value equal to \(\theta\).

At this point, we make three significant comments: (i) such a system has long been studied in elementary circuit theory, called an _RC circuit_, and is physically realizable. (ii) It has an ordinary differential equation (ODE) form and naturally coincides with the formulation of continuous-depth neural networks (Chen et al., 2018), if we view the nodal voltage \(\mathbf{v}\) as the variable. (iii) An RC circuit alone does not have enough expressiveness power, as the function governing the ODE dynamics is linear in \(\mathbf{v}\).

Now, to improve the representation capability, we deliberately introduce non-linear current-voltage relations and design our KirchhoffNet as Figure 1 shows. Specifically, we enforce that all nodes \(\{n_{1},n_{2},\cdots,n_{N}\}\) connect to the ground node \(n_{0}\) via a capacitive device shown in Eq. (4) with parameter value \(\theta=1\). We emphasize that there will be \(N\) such capacitive devices in total, and all their parameter values \(\theta\) are not learnable and are always fixed to \(1\) in training. Then, for the rest of the connections among nodes specified by \(\mathcal{E}\), their parameters are all learnable and have the same non-linear current-voltage relation \(i_{sd}=g(v_{s},v_{d},\boldsymbol{\theta}_{sd})\), where \(\boldsymbol{\theta}_{sd}\) represents the learnable parameter, and \(g(\cdot,\cdot,\cdot)\) is a non-linear function. It can be shown, the dynamics of \(v_{j}\) (where \(j=1,2,\cdots,N\)) follows:

\[\begin{split}\theta\dot{v}_{j}=1\times\dot{v}_{j}& =\sum_{s\in\mathcal{N}^{s}(n_{j})}g(v_{s},v_{j},\boldsymbol{\theta }_{sj})\\ &\quad-\sum_{d\in\mathcal{N}^{d}(n_{j})}g(v_{j},v_{d},\boldsymbol {\theta}_{jd})\\ &=\sum_{s\in\mathcal{N}^{s}(n_{j})}i_{sj}-\sum_{d\in\mathcal{N}^ {d}(n_{j})}i_{dj}\end{split} \tag{5}\]

Our present implementation favors the function \(g\):

\[g(v_{s},v_{d},\boldsymbol{\theta}_{sd})=\theta_{sd,1}\times\text{ReLU}(v_{s}-v_ {d}-\theta_{sd,2}) \tag{6}\]

where \(\boldsymbol{\theta}_{sd}=[\theta_{sd,1},\theta_{sd,2}]^{T}\). To clarify, the voltage values at \(t=0\) and \(t=1\), denoted as \(\mathbf{v}(0)\) and \(\mathbf{v}(1)\), are respectively the KirchhoffNet input and output. Except for those \(\theta=1\) on the fixed capacitive branches, all other \(\boldsymbol{\theta}\)'s on the non-linear branches are the learnable parameters, which govern how \(\mathbf{v}(0)\) changes to \(\mathbf{v}(1)\). Importantly, multiple non-linear connections can exist between two nodes \(n_{s}\) and \(n_{d}\).

Important Remarks.In a broader context, KirchhoffNet can be seen as a fusion of continuous-depth model (Chen et al., 2018) with message passing neural networks

Figure 1: An illustration of KirchhoffNet. The dashed grey lines represent fixed capacitive branches, while the solid black lines represent learnable non-linear branches.
(MPNNs) (Gilmer et al., 2017). However, there are crucial distinctions that we would like to highlight. First, the message function in an MPNN is usually parameterized by a deep neural network with discrete layers containing many parameters, while the counterpart in our KirchhoffNet is the function \(g\), which only has two parameters. Second, a node in MPNN can have a vectorial node feature, while in our case, each node only has a scalar nodal voltage. 4 Third, KirchhoffNet does not have traditional layers such as convolution or linear layers. Fourth, it is clear that KirchhoffNet can exactly map to a real circuit, which neither of Neural ODE or MPNNs can. For detailed discussions on hardware potentials of KirchhoffNet, please refer to Section 2.2.

Footnote 4: To mitigate this limitation in KirchhoffNet, we can overcome it through a stacking approach. For instance, we can employ a KirchhoffNet with 21 nodes and utilize \([v_{2n-1},v_{2n}]^{T}\) as the node feature for the \(n\)-th node. In this setup, with \(n\) ranging from 1 to 10, we achieve a 2-dimensional nodal feature.

Now that we have defined KirchhoffNet, our focus shifts to how it can be effectively trained. In neural ordinary differential equation (Neural ODE) (Chen et al., 2018), it has been demonstrated that the adjoint method can be employed to efficiently compute the derivative \(dL/d\boldsymbol{\theta}\) of a model in the form \(\dot{\mathbf{v}}=f(\mathbf{v},t,\boldsymbol{\theta})\), where \(L\) represents a loss function, \(f\) is a deep neural network with learnable parameters \(\boldsymbol{\theta}\). Obviously, Eq. (5) satisfies this format, and thus, adjoint method will be applicable to KirchhoffNet. However, what is surprising and unknown before in our context is that we can prove that the adjoint ODE system (Chen et al., 2018) reduces to a circuit of the same topology as the original circuit, which is summarized by the following theorem.

**Theorem 2.1**.: _For a KirchhoffNet with topology \(\mathcal{G}=(\mathcal{V},\mathcal{E})\). Its adjoint circuit has exactly the same topology \(\mathcal{G}^{\text{adj}}=(\mathcal{V},\mathcal{E})\), but all the branches defined in Eq. (3) become conductive. For a specific device connecting from \(n_{s}\) to \(n_{d}\) in \(\mathcal{G}^{\text{adj}}\), its parameter \(\theta_{sd}^{\text{adj}}\in\mathbb{R}\) is defined by the nodal voltage \(\{v_{s},v_{d}\}\) and the branch current \(i_{sd}\) in the original KirchhoffNet \(\mathcal{G}\):_

\[\theta_{sd}^{\text{adj}}=\frac{di_{sd}}{d(v_{s}-v_{d})} \tag{7}\]

_Simulating \(\mathcal{G}\) forward in time and \(\mathcal{G}^{\text{adj}}\) backward in time, yields the loss gradient._

### Hardware Implementation and Potentials

The most captivating aspect of KirchhoffNet is its profound connection to physical reality. In the real world, we must account for physical units, typically represented in SI units, where time is measured in seconds (s), voltage in volts (V), current in amperes (A), and capacitance in farads (F).

Our formulation in Section 2.1 essentially affirms that a KirchhoffNet can be realized by a physical circuit. This circuit would involve \(N\) fixed capacitances, each with a value of 1 F, along with many learnable non-linear current-voltage devices. When we set the initial nodal voltage at \(t=0\) s and let the circuit evolve to \(t=1\) s, the forward calculation is completed within this time frame.

A one-second forward calculation does not sound appealing at all. But the key lies in reducing the capacitance value which can make the forward calculation much faster. To understand this mathematically, we take the simplest method for solving ODEs, the forward Euler method, as an illustration. Essentially, the forward calculation of KirchhoffNet needs to solve an ODE system shown in Eq. (5) from \(t=0\) s to \(t=1\) s. Forward Euler method accomplishes this by using numerical differentiation \(\dot{v}\approx[v(t+\Delta t)-v(t)]/\Delta t\). Substituting this expression to the left-hand side of Eq. (5) and simplifying the expression, we obtain:

\[v_{j}(t+\Delta t)=v_{j}(t)+\frac{\Delta t}{\theta}\left[\sum_{s\in\mathcal{N}^ {s}(n_{j})}i_{sj}-\sum_{d\in\mathcal{N}^{d}(n_{j})}i_{dj}\right] \tag{8}\]

where the terms inside the bracket are dependent on time \(t\). Thus, when the circuit is solved at time \(t\), Eq. (8) determines the circuit at time \((t+\Delta t)\). The step size \(\Delta t\) usually has to be very small for the solution to be accurate.

We notice that if we simultaneously reduce \(\Delta t\), \(t\), and \(\theta\) by the same scale \(a\), i.e., \(\Delta t\rightarrow\Delta t/a\), \(t\to t/a\), and \(\theta\rightarrow\theta/a\), where \(a\gg 1\), then the \(v_{j}\) at \(t\) originally will now equal the \(v_{j}\) at \(t/a\), i.e., \(v_{j}(t)\to v_{j}(t/a)\). Alternatively, this can be understood as simultaneously rescaling the units of time and capacitance. For example, when \(a=10^{9}\), we are effectively using nanofarad (nF) as the unit of capacitance, where \(1\) nF = \(10^{-9}\) F, and nanosecond (ns) as the unit of time, where \(1\) ns = \(10^{-9}\) s. Remarkably, this scenario implies that if we impose the initial nodal voltages on the circuit at \(t=0\) ns, and after letting it run to \(t=1\) ns, it finishes the forward calculation.

So far, it is evident that if we can simulate a non-unit KirchhoffNet in software to solve a task using \(\mathbf{v}(0)\) as the input and \(\mathbf{v}(1)\) as the output, this corresponds to a real physical system capable of completing the forward calculation within various timescales, such as one second (s), one millisecond (ms), one microsecond (\(\upmu\)s), one nanosecond (ns), or one picosecond (ps). The specific timescale depends on the capacitance value, whether it is one farad (F), one millifarad (mF), one microfarad (\(\upmu\)F), one nanofarad (nF), or one picofarad (pF). In theory, there is no apparent reason why this scalability cannot be pushed further. What's even more intriguing is that **this statement remains valid, regardless of the number of parameters in KirchhoffNet.** To complete the picture, it is worth noting that a one-picofarad (1 pF) capacitance is physically achievable using modern technology, while measuring nodal voltages at the picosecond (ps) timescale may be challenging. Therefore, ultimately, a physically realized KirchhoffNet will be able to complete one
forward calculation in \(1/f\) seconds, with \(f\) representing the minimum possible operation frequency (also known as the clock frequency) of a hardware system (e.g., \(f=3\) Ghz).

Lastly, a critical consideration is the physical implementability of the non-linear current-voltage relation \(g\). In our case, Eq. (6) corresponds to connecting a conductance and a current in parallel, followed by connecting them in series to a one-sided switch, as shown in Figure 2. This high-level schematic outlines a promising approach for realizing KirchhoffNet with a physical system, but we note that substantial engineering and exploration of approximate implementations (e.g., using standard MOS transistors) will be needed.

## 3 Numerical Results

We have demonstrated various intriguing properties of KirchhoffNet; however, its representational capabilities remain unverified. Naturally, this depends significantly on the topology \(\mathcal{E}\) of the circuit. Through careful topology design, KirchhoffNet can perform common tasks typically handled by traditional deep neural networks, even in the absence of classical layers like linear or convolutional layers. Take image classification on MNIST (LeCun et al., 1998) as an example. MNIST comprises gray-scale handwritten digits from 0 to 9 with a resolution of \(28\times 28\) pixels. We design a KirchhoffNet with \(835\) nodes for this task:

* Node \(n_{0}\) acts as the ground node, with its voltage \(v_{0}\) fixed to \(0\) regardless of time \(t\).
* All \(28^{2}=784\) pixels are flattened and taken as the initial nodal voltages for node \(n_{1}\) to \(n_{784}\).
* Nodes \(n_{785}\) to \(n_{824}\) are introduced to enhance the representational power of the ODE, inspired by (Dupont et al., 2019). Their nodal voltages at \(t=0\) are initialized to zeros.
* Nodes \(n_{825}\) to \(n_{834}\) have initial nodal voltages set to zeroes, and their values at time \(t=1\) are used to calculate the cross-entropy loss.

Our intuition behind this design is that we use \(n_{1}\) to \(n_{784}\) to learn a good feature map of the digit, that the last 10 nodes (\(n_{825}\sim n_{834}\)) serve as class predictions, and that the middle 40 nodes (\(n_{785}\sim n_{824}\)) are used to boost model capacity. We trained this KirchhoffNet using a learning rate of \(10^{-4}\), an AdamW optimizer, and 250 epochs, with data augmentation. The resulting model achieved an accuracy of \(98.86\)% on the test dataset, which is comparable to the state-of-the-art performance of convolution-based neural networks (LeCun et al., 1998; Chen et al., 2018; Dupont et al., 2019).

## 4 Conclusions

In this paper, we define a novel class of neural network models, termed _KirchhoffNet_, based on a fundamental principle of analog electronic circuitry, Kirchhoff's current law (KCL). KirchhoffNet is closely related to continuous-depth message passing neural networks, but it exhibits critical distinctions. Additionally, we provide a justification for deploying KirchhoffNet on an actual electronic circuit, enabling it to complete forward inference in \(1/f\) seconds, regardless of the number of model parameters. We design an example KirchoffNet architecture and achieve a 98.86% test accuracy on the MNIST dataset. Our future endeavors will involve testing it on other common datasets and tasks.

## Acknowledgements

This work is subject to change. We welcome any feedback. Zhengqi Gao would like to thank Zihui Xue (UT-Austin) for her support on computational matters.

Figure 3: Top row: An example of the input \(28\times 28\) image on the left and the learned first 784 nodal voltage values at \(t=1\) reshaped on the right. Bottom row: Evolution of the last 10 nodal voltages (after applying Softmax) over time. Except for the probability corresponding to the 2nd class, all others gradually reduce to zero.

Figure 2: Left: The schematic of a composite device made up of a conductance, a current source, and a one-sided switch, to realize the function \(g\) shown in Eq. (6). Right: The current-voltage relation is plotted. Without the switch, the current-voltage relation follows a linear pattern, as indicated by the dashed blue line. Adding the one-sided switch truncates the relation into the required red line.