The Descriptive Complexity of Graph Neural Networks
We analyse the power of graph neural networks (GNNs) in terms of Boolean
circuit complexity and descriptive complexity.
  We prove that the graph queries that can be computed by a polynomial-size
bounded-depth family of GNNs are exactly those definable in the guarded
fragment GFO+C of first-order logic with counting and with built-in relations.
This puts GNNs in the circuit complexity class (non-uniform) TC^0. Remarkably,
the GNN families may use arbitrary real weights and a wide class of activation
functions that includes the standard ReLU, logistic "sigmod", and hyperbolic
tangent functions. If the GNNs are allowed to use random initialisation and
global readout (both standard features of GNNs widely used in practice), they
can compute exactly the same queries as bounded depth Boolean circuits with
threshold gates, that is, exactly the queries in TC^0.
  Moreover, we show that queries computable by a single GNN with piecewise
linear activations and rational weights are definable in GFO+C without built-in
relations. Therefore, they are contained in uniform TC^0.
Martin Grohe
# The Descriptive Complexity of Graph Neural Networks

###### Abstract

We analyse the power of graph neural networks (GNNs) in terms of Boolean circuit complexity and descriptive complexity.

We prove that the graph queries that can be computed by a polynomial-size bounded-depth family of GNNs are exactly those definable in the guarded fragment \(\mathsf{GFO}+\mathsf{C}\) of first-order logic with counting and with built-in relations. This puts GNNs in the circuit complexity class \(\mathsf{TC}^{0}\). Remarkably, the GNN families may use arbitrary real weights and a wide class of activation functions that includes the standard ReLU, logistic "sigmoid", and hyperbolic tangent functions. If the GNNs are allowed to use random initialisation and global readout (both standard features of GNNs widely used in practice), they can compute exactly the same queries as bounded depth Boolean circuits with threshold gates, that is, exactly the queries in \(\mathsf{TC}^{0}\).

Moreover, we show that queries computable by a single GNN with piecewise linear activations and rational weights are definable in \(\mathsf{GFO}+\mathsf{C}\) without built-in relations. Therefore, they are contained in uniform \(\mathsf{TC}^{0}\).

## 1 Introduction

Graph neural networks (GNNs) [10, 28] are deep learning models for graph data that play a key role in machine learning on graphs (see, for example, [7]). A GNN describes a distributed algorithm carrying out local computations at the vertices of the input graph. At any time, each vertex has a "state", which is a vector of reals, and in each computation step it sends a message to all its neighbours. The messages are also vectors of reals, and they only depend on the current state of the sender and the receiver. Every
vertex aggregates the messages it receives and computes its new state depending on the old state and the aggregated messages. The message and state-update functions are computed by feedforward neural networks whose parameters are learned from data.

In this paper, we study the _expressiveness_ of GNNs: which functions on graphs or their vertices can be computed by GNNs? We provide answers in terms of Boolean circuits and logic, that is, computation models of classical (descriptive) complexity theory. An interesting and nontrivial aspect of this is that GNNs are "analogue" computation models operating on and with real numbers. The weights of neural networks may be arbitrary reals, and the activation functions may even be transcendental functions such as the logistic function \(x\mapsto\frac{1}{1+e^{-x}}\).

We always want functions on graphs to be _isomorphism invariant_, that is, isomorphic graphs are mapped to the same value. Similarly, we want functions on vertices to be _equivariant_, that is, if \(v\) is a vertex of a graph \(G\) and \(f\) is an isomorphism from \(G\) to a graph \(H\), then \(v\) and \(f(v)\) are mapped to the same value. Functions computed by GNNs are always invariant or equivariant, and so are functions defined in logic (a.k.a. _queries_).

In a machine learning context, it is usually assumed that the vertices of the input graph are equipped with additional features in the form of vectors over the reals; we speak of _graph signals_ in this paper. The function values are also vectors over the reals. Thus a function on the vertices of a graph is an equivariant transformation between graph signals. When comparing GNNs and logics or Boolean circuits, we focus on Boolean functions, where the input signal is Boolean, that is, it associates a \(\{0,1\}\)-vector with every vertex of the input graph, and the output is just a Boolean value \(0\) or \(1\). In the logical context, it is natural to view Boolean signals as vertex labels. Thus a Boolean signal in \(\{0,1\}^{k}\) is described as a sequence of \(k\) unary relations on the input graph. Then an invariant Boolean function becomes a _Boolean query_ on labelled graphs, and an equivariant Boolean function on the vertices becomes a _unary query_. To streamline the presentation, in this paper we focus on unary queries and equivariant functions on the vertices. All our results also have versions for Boolean queries and functions on graphs, but we only discuss these in occasional remarks. While we are mainly interested in queries (that is, Boolean functions), our results also have versions for functions with arbitrary real input and output signals. These are needed for the proofs anyway. But since the exact statements become unwieldy, we keep them out of the introduction. Before discussing further background, let us state our central result.

**Theorem 1.1**.: _Let \(\mathcal{G}\) be a unary query on labelled graphs. Then the following are equivalent._

1. \(\mathcal{G}\) _is computable by a polynomial-weight bounded-depth family of GNNs with rpl approximable activation functions._
2. \(\mathcal{G}\) _is definable in the guarded fragment_ \(\mathsf{GFO}+\mathsf{C}_{\mathrm{nu}}\) _of first-order logic with counting and with built-in relations._

The result requires more explanations. First of all, it is a _non-uniform_ result, speaking about computability by families of GNNs and a logic with built-in relation. A family
\(\mathcal{N}=(\mathfrak{N}^{(n)})_{n\in\mathbb{N}}\) of GNNs consists of GNNs \(\mathfrak{N}^{(n)}\) for input graphs of size \(n\). _Bounded depth_ refers to the number of message passing rounds, or layers, of the GNNs as well as the depth of the feed-forward neural networks they use for their message and state-update functions. We would like the GNN \(\mathfrak{N}^{(n)}\) to be of "size" polynomial in \(n\), but since we allow arbitrary real weights as parameters of the neural networks, it is not clear what this actually means. We define the _weight_ of a GNN to be the number of computation nodes of the underlying neural networks plus the absolute values of all weights. The class of _rpl (rational piecewise linear) approximable_ functions contains all functions that are commonly used as activation functions for neural networks, for example, the rectified linear unit, the logistic function, the hyperbolic tangent function, the scaled exponential linear unit (see Section 2.4 for background on neural networks and their activation functions).

On the logical side, _first-order logic with counting_ FO+C is the two-sorted extension of first-order logic over relational structures that has variables ranging over the non-negative integers, bounded arithmetic on the integer side, and counting terms that give the number of assignments satisfying a formula. In the _\(k\)-variable fragment_ FO\({}^{k}+\)C, only \(k\) variables ranging over the vertices of the input graphs are allowed (but arbitrarily many variables for the integer part). The _guarded fragment_ GFO+C is a fragment of FO\({}^{2}+\)C where quantification over vertices is restricted to the neighbours of the current vertex. _Built-in relations_ are commonly used in descriptive complexity to introduce non-uniformity to logics and compare them to non-uniform circuit complexity classes. Formally, they are just arbitrary relations on the non-negative integers that the logic can access, independently of the input structure.

It is well-known that over ordered input structures, FO+C with built-in relations captures the circuit complexity class (non-uniform) TC\({}^{0}\), consisting of Boolean functions (in our context: queries) that are computable by families of bounded-depth polynomial-size Boolean circuits with threshold gates. This implies that, as a corollary to Theorem 1.1, we get the following.

**Corollary 1.2**.: _Every unary query that is computable by a polynomial-weight bounded-depth family of GNNs with rpl approximable activation functions is in TC\({}^{0}\)._

The strength of GNNs can be increased by extending the input signals with a random component [2, 27]. In [2], it was even proved that such _GNNs with random initialisation_ can approximate all functions on graphs. The caveat of this result is that it is non-uniform and that input graphs of size \(n\) require GNNs of size exponential in \(n\) and depth linear in \(n\). We ask which queries can be computed by polynomial-weight, bounded-depth families of GNNs. Surprisingly, this gives us a converse of Corollary 1.2 and thus a characterisation of TC\({}^{0}\).

**Theorem 1.3**.: _Let \(\mathcal{G}\) be a unary query on labelled graphs. Then the following are equivalent._

1. \(\mathcal{G}\) _is computable by a polynomial-weight bounded-depth family of GNNs with random initialisation and with rpl approximable activation functions._
_._
2. \(\mathds{G}\) _is computable in_ \(\mathsf{TC}^{0}\)_._

We mention that, following [2], we allow GNNs with random initialisation to also use a feature known as _global readout_, which means that in each message-passing round of a GNN computation, the vertices not only receive messages from their neighbours, but the aggregated state of all vertices. There is also a version of Theorem 1.1 for GNNs with global readout.

### Related Work

A fundamental result on the expressiveness of GNNs [24, 32] states that two graphs are distinguishable by a GNN if and only if they are distinguishable by the 1-dimensional Weisfeiler-Leman (WL) algorithm, a simple combinatorial algorithm originally introduced as a graph isomorphism heuristics [23, 31]. This result has had considerable impact on the subsequent development of GNNs, because it provides a yardstick for the expressiveness of GNN extensions (see [25]). Its generalisation to higher-order GNNs and higher-dimensional WL algorithms [24] even gives a hierarchy of increasingly more expressive formalisms against which such extensions can be compared. However, these results relating GNNs and their extensions to the WL algorithm only consider a restricted form of expressiveness, the power to distinguish two graphs. Furthermore, the results are _non-uniform_, that is, the distinguishing GNNs depend on the input graphs or at least on their size, and the GNNs may be arbitrarily large and deep. Indeed, the GNNs from the construction in [32] may be exponentially large in the graphs they distinguish. Those of [24] are polynomial. Both have recently been improved by [1], mainly showing that the messages only need to contain logarithmically many bits.

We are not the first to study the logical expressiveness of GNNs (see [12] for a recent survey). It was proved in [3] that all unary queries definable in the guarded fragment \(\mathsf{GC}\) of the extension \(\mathsf{C}\) of first-order logic by counting quantifiers \(\exists^{\geq n}x\) ("there exist at least \(n\) vertices \(x\) satisfying some formula") are computable by a GNN. The logic \(\mathsf{GC}\) is weaker than our \(\mathsf{GFO}+\mathsf{C}\) in that it does not treat the numbers \(n\) in the quantifiers \(\exists^{\geq n}x\) as variables, but as fixed constants. What is interesting about this result, and what makes it incomparable to ours, is that it is a _uniform_ result: a query definable in \(\mathsf{GC}\) is computable by a single GNN across all graph sizes. There is a partial converse to this result, also from [3]: all unary queries that are definable in first-order logic and computable by a GNN are actually definable in \(\mathsf{GC}\). Note, however, that there are queries computable by GNNs that are not definable in first-order logic.

A different approach to capturing GNNs by logic has been taken in [9]. There, the authors introduce a new logic \(\mathsf{MPLang}\) that operates directly on the reals. The logic, also a guarded (or modal) logic, is simple and elegant and well-suited to translate GNN computations to logic. The converse translation is more problematic, though. But to be fair, it is also in our case, where it requires families of GNNs and hence non-uniformity. However, the purpose of the work in [9] is quite different from ours. It is our goal to describe GNN computations in terms of standard descriptive complexity and thus to be able to quantify the computational power of GNNs in the framework of classical
complexity. It is the goal of [9] to expand logical reasoning to real-number computations in a way that is well-suited to GNN computations. Of course, both are valid goals.

There is another line of work that is important for us. In the 1990s, researchers studied the expressiveness of feedforward neural networks (FNNs) and compared it to Boolean computation models such as Turing machines and circuits (for example, [18, 21, 22, 29]). Like GNNs, FNNs are analogue computation models operating on the reals, and this work is in the same spirit as ours. An FNN has fixed numbers \(p\) of inputs and \(q\) of outputs, and it thus computes a function from \(\mathbb{R}^{p}\) to \(\mathbb{R}^{q}\). Restricted to Boolean inputs, we can use FNNs with \(p\) inputs to decide subsets of \(\{0,1\}^{p}\), and we can use families of FNNs to decide languages. It was proved in [21] that a language is decidable by a family of bounded-depth polynomial-weight FNNs using piecewise-polynomial activation functions if and only if it is in \(\mathsf{TC}^{0}\). It may seem that our Corollary 1.2, at least for GNNs with piecewise-linear (or even piecewise-polynomial) activations, follows easily from this result. But this is not the case, because when processing graphs, the inputs to the FNNs computing the message and update functions of the GNN may become large through the aggregation ranging over all neighbours. Also, the arguments of [21] do not extend to rpl-approximable activation functions like the logistic function. There has been related work [18] that extends to a wider class of activation functions including the logistic function, using arguments based on o-minimality. But the results go in a different direction; they bound the VC dimension of FNN architectures and do not relate them to circuit complexity.

### Techniques

The first step in proving the difficult implication (1)\(\Rightarrow\)(2) of Theorem 1.1 is to prove a uniform result for a simpler class of GNNs; this may be of independent interest.

**Theorem 1.4**.: _Let \(\mathcal{G}\) be a unary query computable by a GNN with rational weights and piecewise linear activations. Then \(\mathcal{G}\) is definable in \(\mathsf{GFO}+\mathsf{C}\)._

Compare this with the result of [3]: every query definable in the (weaker) logic \(\mathsf{GC}\) is computable by a GNN and in fact a GNN with rational weights and piecewise linear activations. Thus we may write \(\mathsf{GC}\subseteq\mathsf{GNN}\subseteq\mathsf{GFO}+\mathsf{C}.\) It is not hard to show that both inclusions are strict.

To prove Theorem 1.4, we need to show that the rational arithmetic involved in GNN computations, including unbounded linear combinations, can be simulated, at least approximately, in the logic \(\mathsf{GFO}+\mathsf{C}\). Establishing this is a substantial part of this paper, and it may be of independent interest.

So how do we prove the forward implication of Theorem 1.1 from Theorem 1.4? It was our first idea to look at the results for FNNs. In principle, we could use the linear-programming arguments of [21]. This would probably work, but would be limited to piecewise linear or piecewise polynomial activations. We could then use o-minimality to extend our results to wider classes of activation functions. After all, o-minimality was also applied successfully in the somewhat related setting of constraint databases [19].
Again, this might work, but our analytical approach seems simpler and more straightforward. Essentially, we use the Lipschitz continuity of the functions computed by FNNs to show that we can approximate arbitrary GNNs with rpl-approximable activations by GNNs with rational weights and piecewise linear activations, and then we apply Theorem 1.3.

Let us close the introduction with a few remarks on Theorem 1.3. The reader may have noted that assertion (1) of the theorem involves randomness in the computation model, whereas (2) does not. To prove the implication (1)\(\Rightarrow\)(2) we use the well-known "Adleman Trick" that allows us to trade randomness for non-uniformity. To prove the converse implication, the main insight is that with high probability, random node initialisation gives us a linear order on the vertices of the input graph. Then we can use the known fact that FO+C with built-in relations captures \(\mathsf{TC}^{0}\) on ordered structures.

### Structure of this Paper

After collecting preliminaries from different areas in Section 2, in Section 3 we develop a machinery for carrying out the required rational arithmetic in first-order logic with counting and its guarded fragment. This is a significant part of this paper. We then introduce GNNs (Section 4) and prove the uniform Theorem 1.4 (Section 5). We prove the forward direction of Theorem 1.1 in Section 6 and the backward direction in Section 7. Finally, we prove Theorem 1.3 in Section 8.

## 2 Preliminaries

By \(\mathbb{Z},\mathbb{N},\mathbb{N}_{>0},\mathbb{Q},\mathbb{R}\) we denote the sets of integers, nonnegative integers, positive integers, rational numbers, and real numbers, respectively. Instead or arbitrary rationals, we will often work with _dyadic rationals_, that is, rationals whose denominator is a power of two. These are precisely the numbers that have a presentation as finite precision binary floating point numbers. We denote the set of dyadic rationals by \(\mathbb{Z}\!\left[\frac{1}{2}\right]\).

We denote the binary representation of \(n\in\mathbb{N}\) by \(\operatorname{bin}(n)\). The _bitsize_ of \(n\) is length of the binary representation, that is,

\[\operatorname{bsize}(n)\coloneqq|\operatorname{bin}(n)|=\begin{cases}1&\text{ if }n=0,\\ \left[\log(n+1)\right]&\text{ if }n>0,\end{cases}\]

where \(\log\) denotes the binary logarithm. We denote the \(i\)th bit of the binary representation of \(n\in\mathbb{N}\) by \(\operatorname{Bit}(i,n)\), where we count bits starting from \(0\) with the lowest significant bit. It will be convenient to let \(\operatorname{Bit}(i,n)\coloneqq 0\) for all \(i\geq\operatorname{bsize}(n)\). So

\[n=\sum_{i=0}^{\operatorname{bsize}(n)-1}\operatorname{Bit}(i,n)\cdot 2^{i}= \sum_{i\in\mathbb{N}}\operatorname{Bit}(i,n)\cdot 2^{i}.\]

The _bitsize_ of an integer \(n\in\mathbb{Z}\) is \(\operatorname{bsize}(n)\coloneqq 1+\operatorname{bsize}(|n|)\), and the _bitsize_ of a dyadic rational \(q=\frac{n}{2^{\ell}}\in\mathbb{Z}\!\left[\frac{1}{2}\right]\) in reduced form is \(\operatorname{bsize}(q)\coloneqq\operatorname{bsize}(n)+\ell+1\).
We denote tuples (of numbers, variables, vertices, et cetera) using boldface letters. Usually, a \(k\)-tuple \(\mathbf{t}\) has entries \(t_{1},\ldots,t_{k}\). The empty tuple is denoted \(\varnothing\) (just like the emptyset, this should never lead to any confusion), and for every set \(S\) we have \(S^{0}=\{\varnothing\}\). For tuples \(\mathbf{t}=(t_{1},\ldots,t_{k})\) and \(\mathbf{u}=(u_{1},\ldots,u_{\ell})\), we let \(\mathbf{tu}=(t_{1},\ldots,t_{k},u_{1},\ldots,u_{\ell})\). To improve readability, we often write \((\mathbf{t},\mathbf{u})\) instead of \(\mathbf{tu}\). This does not lead to any confusion, because we never consider nested tuples.

For a vector \(\mathbf{x}=(x_{1},\ldots,x_{k})\in\mathbb{R}^{k}\), the \(\ell_{1}\)_-norm_ (a.k.a Manhattan norm) is \(\|\mathbf{x}\|_{1}\coloneqq\sum_{i=1}^{k}|x_{i}|\), the \(\ell_{2}\)_-norm_ (a.k.a Euclidean norm) is \(\|\mathbf{x}\|_{2}\coloneqq\sqrt{\sum_{i=1}^{k}x_{i}^{2}}\), and the \(\ell_{\infty}\)_-norm_ (a.k.a. maximum norm) \(\|\mathbf{x}\|_{\infty}\coloneqq\max_{i\in[k]}|x_{i}|\). As \(\frac{1}{k}\|\mathbf{x}\|_{1}\leq\|\mathbf{x}\|_{\infty}\leq\|\mathbf{x}\|_{2}\leq\|\mathbf{x }\|_{1}\), it does not make much of a difference which norm we use; most often, it will be convenient for us to use the \(\ell_{\infty}\)-norm.

### Functions and Approximations

A function \(f:\mathbb{R}^{p}\to\mathbb{R}^{q}\) is _Lipschitz continuous_ if there is some constant \(\lambda\), called a _Lipschitz constant_ for \(f\), such that for all \(\mathbf{x},\mathbf{y}\in\mathbb{R}^{p}\) it holds that \(\|f(\mathbf{x})-f(\mathbf{y})\|_{\infty}\leq\lambda\,\|\mathbf{x}-\mathbf{y}\|_{\infty}\).

A function \(L:\mathbb{R}\to\mathbb{R}\) is _piecewise linear_ if there are \(n\in\mathbb{N}\), \(a_{0},\ldots,a_{n}\), \(b_{0},\ldots,b_{n}\), \(t_{1},\ldots,t_{n}\in\mathbb{R}\) such that \(t_{1}<t_{2}<\ldots<t_{n}\) and

\[L(x)=\begin{cases}a_{0}x+b_{0}&\text{if $x<t_{1}$,}\\ a_{i}x+b_{i}&\text{if $t_{i}\leq x<t_{i+1}$ for some $i<n$,}\\ a_{n}x+b_{n}&\text{if $x\geq t_{n}$}\end{cases}\]

if \(n\geq 1\), or \(L(x)=a_{0}x+b_{0}\) for all \(x\) if \(n=0\). Note that there is a unique _minimal representation_ of \(L\) with minimal number \(n\) of pieces. We call \(t_{1},\ldots,t_{n}\) in the minimal representation of \(L\) the _thresholds_ of \(L\); these are precisely the points where \(L\) is non-linear. \(L\) is _rational_ if all its parameters \(a_{i},b_{i},t_{i}\) in the minimal representation are dyadic rationals.1 If \(L\) is rational, then its _bitsize_ of \(\operatorname{bsize}(L)\) is the sum of the bitsizes of all the parameters \(a_{i},b_{i},t_{i}\) of the minimal representation. Observe that if \(L\) is continuous then it is Lipschitz continuous with Lipschitz constant \(\max_{0\leq i\leq n}a_{i}\).

Footnote 1: Throughout this paper we work with dyadic rationals. For this reason, we are a little sloppy in our terminology. For example, we call a function “rational piecewise linear” when the more precise term would be “dyadic-rational piecewise linear”.

**Example 2.1**.: The most important example of a rational piecewise linear function for us is the _rectified linear unit_\(\operatorname{relu}:\mathbb{R}\to\mathbb{R}\) defined by \(\operatorname{relu}(x)\coloneqq\max\{0,x\}\).

In fact, it is not hard to see that every piecewise linear function can be written as a linear combination of \(\operatorname{relu}\)-terms. For example, the _identity_ function \(\operatorname{id}(x)=x\) can be written as \(\operatorname{relu}(x)-\operatorname{relu}(-x)\), and the _linearised sigmoid_ function \(\operatorname{lsig}:\mathbb{R}\to\mathbb{R}\) can be defined by \(\operatorname{lsig}(x)=0\) if \(x<0\), \(\operatorname{lsig}(x)=x\) if \(0\leq x<1\), an \(\operatorname{lsig}(x)=1\) if \(x\geq 1\), can be written as \(\operatorname{relu}(x)-\operatorname{relu}(x-1)\).
We need a notion of approximation between functions on the reals. Let \(f,g:\mathbb{R}\to\mathbb{R}\) and \(\varepsilon\in\mathbb{R}_{>0}\). Then \(g\) is an _\(\varepsilon\)-approximation_ of \(f\) if for all \(x\in\mathbb{R}\) it holds that

\[\big{|}f(x)-g(x)\big{|}\leq\varepsilon|f(x)|+\varepsilon.\]

Note that we allow for both an additive and a multiplicative approximation error. This notion is not symmetric, but if \(g\)\(\varepsilon\)-approximates \(f\) for some \(\varepsilon<1\) then \(f\)\(\frac{\varepsilon}{1-\varepsilon}\)-approximates \(g\).

We call a function \(f:\mathbb{R}\to\mathbb{R}\)_rpl-approximable_ if for every \(\varepsilon>0\) there is a continuous rational piecewise linear function \(L\) of bitsize polynomial in \(\varepsilon^{-1}\) that \(\varepsilon\)-approximates \(f\).

**Example 2.2**.: The _logistic function_\(\operatorname{sig}(x)=\frac{1}{1+e^{-x}}\) and the _hyperbolic tangent_\(\tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}\) are rpl-approximable. Examples of unbounded rpl-approximable functions are the _soft plus function_\(\ln(1+e^{x})\) and the _exponential linear units_ defined by \(\operatorname{elu}_{\alpha}(c)=x\) if \(x>0\) and \(\alpha(e^{x}-1)\) if \(x\leq 0\), where \(\alpha>0\) is a constant. We omit the straightforward proofs based on simple calculus.

### Graphs and Signals

Graphs play two different roles in this paper: they are the basic data structures on which logics and graph neural networks operate, and they form the skeleton of Boolean circuits and neural networks. In the first role, which is the default, we assume graphs to be undirected, in the second role they are directed acyclic graphs (dags).

We always denote the vertex set of a graph or dag \(G\) by \(V(G)\) and the edge set by \(E(G)\). We denote edges by \(vw\) (without parentheses). We assume the vertex set of all graphs in this paper to be finite and nonempty. The _order_ of a graph \(G\) is \(|G|\coloneqq|V(G)|\), and the _bitsize_\(\operatorname{bsize}(G)\) of \(G\) is the size of a representation of \(G\). (For simplicity, we can just take adjacency matrices, then \(\operatorname{bsize}(G)=|G|^{2}\).) The class of all (undirected) graphs is denoted by \(\mathscr{F}\).

For a vertex \(v\) in an (undirected) graph, we let \(N_{G}(v)\coloneqq\{w\in V(G)\mid vw\in E(G)\}\) be the _neighbourhood_ of \(v\) in \(G\), and we let \(N_{G}[v]\coloneqq\{v\}\cup N_{G}(v)\) be the _closed neighbourhood_. Furthermore, we let \(\deg_{G}(v)\coloneqq|N_{G}(v)|\) be the _degree_ of \(v\) in \(G\). For a vertex \(v\) in a directed graph \(G\), we let \(N_{G}^{+}(v)\coloneqq\{w\in V(G)\mid vw\in E(G)\}\) be the _out-neighbourhood_ of \(v\) and \(N_{G}^{-}(v)\coloneqq\{u\in V(G)\mid uv\in E(G)\}\) the _in-neighbourhood_, and we let \(\deg_{G}^{+}(v)\coloneqq|N_{G}^{+}(v)|\) and \(\deg_{G}^{-}(v)\coloneqq|N_{G}^{-}(v)|\) be the _out-degree_ and _in-degree_. In these and similar notations we omit the index \({}_{G}\) if the graph is clear from the context. We call nodes of in-degree \(0\)_sources_ and nodes of out-degree \(0\)_sinks_. The _depth_\(\operatorname{dp}_{G}(v)\) of node \(v\) in a dag \(G\) is the length of the longest path from a source to \(v\). The _depth_\(\operatorname{dp}(G)\) of a dag \(G\) is the maximum depth of a sink of \(G\).

When serving as data for graph neural networks, the vertices of graphs usually have real-valued features, which we call _graph signals_. An _\(\ell\)-dimensional signal_ on a graph \(G\) is a function \(\mathbf{x}:V(G)\to\mathbb{R}^{\ell}\). We denote the class of all \(\ell\)-dimensional signals on \(G\) by \(\mathscr{S}_{\ell}(G)\) and the class of all pairs \((G,\mathbf{x})\), where \(\mathbf{x}\) is an \(\ell\)-dimensional signal on \(G\), by \(\mathscr{S}\mathscr{S}_{\ell}\). An \(\ell\)-dimensional signal is _Boolean_ if its range is contained in \(\{0,1\}^{\ell}\). By \(\mathscr{S}_{\ell}^{\mathrm{bool}}(G)\) and \(\mathscr{S}\mathscr{S}_{\ell}^{\mathrm{bool}}\) we denote the restrictions of the two classes to Boolean signals.
Isomorphisms between pairs \((G,\mathbf{x})\in\mathcal{GF}_{\ell}\) are required to preserve the signals. We call a mapping \(f:\mathcal{GF}_{\ell}\to\mathcal{GF}_{m}\) a _signal transformation_ if for all \((G,\mathbf{x})\in\mathcal{GF}_{\ell}\) we have \(f(G,\mathbf{x})=(G,\mathbf{x}^{\prime})\) for some \(\mathbf{x}^{\prime}\in\mathcal{G}_{m}(G)\). Such a signal transformation \(f\) is _equivariant_ if for all isomorphic \((G,\mathbf{x}),(H,\mathbf{y})\in\mathcal{GF}_{\ell}\), every isomorphisms \(h\) from \((G,\mathbf{x})\) to \((H,\mathbf{y})\) is also an isomorphism from \(f(G,\mathbf{x})\) to \(f(H,\mathbf{y})\).

We can view signals \(\mathbf{x}\in\mathcal{S}_{\ell}(G)\) as matrices in the space \(\mathbb{R}^{V(G)\times\ell}\). Flattening them to vectors of length \(|G|\ell\), we can apply the usual vector norms to graph signals. In particular, we have \(\left\|\mathbf{x}\right\|_{\infty}=\max\big{\{}\left\|\mathbf{x}(v)\right\|_{\infty} \big{|}\,v\in V(G)\big{\}}\). Sometimes, we need to restrict a signal to a subsets of \(W\in V(G)\). We denote this restriction by \(\mathbf{x}|_{W}\), which is a matrix in \(\mathbb{R}^{W\times\ell}\).

### Boolean Circuits

A _Boolean circuit_\(\mathfrak{C}\) is a directed acyclic graph where all nodes except for the sources are labelled as _negation_, _disjunction_, or _conjunction_ nodes. Negation nodes must have in-degree \(1\). Sources (that is, node of in-degree \(0\)) are _input nodes_, and we always denote them by \(X_{1},\ldots,X_{p}\). Similarly, sinks (that is, nodes of out-degree \(0\)) are _output nodes_, and we denote them by \(Y_{1},\ldots,Y_{q}\). The number \(p\) of input nodes is the _input dimension_ of \(\mathfrak{C}\), and the number \(q\) of output nodes the _output dimension_. Most of the time, we consider circuits that also have _threshold nodes_ of arbitrary positive in-degree, where a \(\geq t\)-threshold node evaluates to \(1\) if at least \(t\) of its in-neighbours evaluate to \(1\). To distinguish them from the Boolean circuits over the standard basis we refer to such circuits as _threshold circuits_. The _depth_\(\operatorname{dp}(\mathfrak{C})\) of a circuit \(\mathfrak{C}\) is the maximum length of a path from an input node to an output node. The _order_\(|\mathfrak{C}|\) of \(\mathfrak{C}\) is the number of nodes, and the _size_ is the number of nodes plus the number of edges.

A circuit \(\mathfrak{C}\) of input dimension \(p\) and output dimension \(q\) computes a function \(f_{\mathfrak{C}}:\{0,1\}^{p}\to\{0,1\}^{q}\) defined in the natural way. To simplify the notation, we simply denote this function by \(\mathfrak{C}\), that is, we write \(\mathfrak{C}(\mathbf{x})\) instead of \(f_{\mathfrak{C}}(\mathbf{x})\) to denote the output of \(\mathfrak{C}\) on input \(\mathbf{x}\in\{0,1\}^{p}\).

In complexity theory, we mostly study which languages \(L\subseteq\{0,1\}^{*}\) or functions \(F:\{0,1\}^{*}\to\{0,1\}^{*}\) can be computed by families \(\mathcal{G}=(\mathfrak{C}_{n})_{n\in\mathbb{N}_{>0}}\) of circuits, where \(\mathfrak{C}_{n}\) is a circuit of input dimension \(n\). Such a family \(\mathcal{G}\)_computes_\(F\) if for all \(n\in\mathbb{N}_{>0}\), \(\mathfrak{C}_{n}\) computes the restriction \(F_{n}\) of \(F\) to \(\{0,1\}^{n}\). We say that \(\mathcal{G}\)_decides_\(L\) if it computes its characteristic function. _Non-uniform_\(\mathsf{TC}^{0}\) is the class of all languages that are decided by a family \(\mathcal{G}=(\mathfrak{C}_{n})_{n\in\mathbb{N}_{>0}}\) of threshold circuits of _bounded depth_ and _polynomial size_. There is also a class _(dlogtime) uniform_\(\mathsf{TC}^{0}\) where the family \(\mathcal{G}\) itself is required to be easily computable; we refer the reader to [4]. We will never work directly with uniform circuit families, but instead use a logical characterisation in terms of first-order logic with counting (Theorem 3.3)

An important fact that we shall use is that standard arithmetic functions on the bit representations of natural numbers can be computed by bounded-depth polynomial-size threshold circuits. We define \(\operatorname{ADD}_{2n}:\{0,1\}^{2n}\to\{0,1\}^{n+1}\) to be the bitwise addition of two \(n\)-bit numbers (whose result may be an \((n+1)\)-bit number). We let \(\operatorname{ADD}:\{0,1\}^{*}\to\{0,1\}^{*}\) be the function that coincides with \(\operatorname{ADD}_{2n}\) on inputs of even size and
maps all inputs of odd size to \(0\). Similarly, we define \(\mathrm{SUB}_{2n}:\{0,1\}^{2n}\to\{0,1\}^{2n}\) and \(\mathrm{SUB}:\{0,1\}^{*}\to\{0,1\}^{*}\) for the truncated subtraction \(m\div n\coloneqq\max\{0,m-n\}\), \(\mathrm{MUL}_{2n}:\{0,1\}^{2n}\to\{0,1\}^{2n}\) and \(\mathrm{MUL}:\{0,1\}^{*}\to\{0,1\}^{*}\) for multiplication. We also introduce a binary integer division function \(\mathrm{DIV}_{2n}:\{0,1\}^{2n}\to\{0,1\}^{n}\) mapping \(n\)-bit numbers \(k,\ell\) to \(\lfloor k/\ell\rfloor\) (with some default value, say \(0\), if \(\ell=0\)). The _iterated addition_ function \(\mathrm{ITADD}_{n^{2}}:\{0,1\}^{n^{2}}\to\{0,1\}^{2n}\) and the derived and \(\mathrm{ITADD}:\{0,1\}^{*}\to\{0,1\}^{*}\) add \(n\) numbers of \(n\)-bits each. Finally, we need the less-than-or-equal-to predicate \(\mathrm{LEQ}_{2n}:\{0,1\}^{2n}\to\{0,1\}\) and \(\mathrm{LEQ}:\{0,1\}^{*}\to\{0,1\}\).

**Lemma 2.3** ([8],[14]).: ADD_,_ MUL_,_ DIV_,_ ITADD_, and_ LEQ _are computable by dlogtime uniform families of bounded-depth polynomial-size threshold circuits._

The fact that ADD, MUL, ITADD, and LEQ are computable by families of bounded-depth polynomial-size threshold circuits goes back to [8] (also see [30]). The arguments given there are non-uniform, but it is not hard to see that they can be "uniformised". The situation for DIV is more complicated. It was known since the mid 1980s that DIV is computable by a non-uniform (or polynomial-time uniform) family of bounded-depth polynomial-size threshold circuits, but the uniformity was only established 15 years later in [14, 15].

### Feedforward Neural Networks

It will be convenient for us to formalise feedforward neural networks in a similar way as Boolean circuits. A more standard presentation of multilayer perceptrons can easily be seen as a special case. A _feedforward neural network architecture_\(\mathfrak{A}\) is a triple \(\big{(}V,E,(\mathfrak{a}_{v})_{v\in V}\big{)}\), where \((V,E)\) is a directed acyclic graph that we call the _skeleton_ of \(\mathfrak{A}\) and for every vertex \(v\in V\), \(\mathfrak{a}_{v}:\mathbb{R}\to\mathbb{R}\) is a continuous function that we call the _activation function_ at \(v\). A _feedforward neural network (FNN)_ is a tuple \(\mathfrak{F}=(V,E,(\mathfrak{a}_{v})_{v\in V},\boldsymbol{w},\boldsymbol{v})\), where \(\big{(}V,E,(\mathfrak{a}_{v})_{v\in V}\big{)}\) is an FNN architecture, \(\boldsymbol{w}=(w_{e})_{e\in E}\in\mathbb{R}^{E}\) associates a _weight_\(w_{e}\) with every edge \(e\in E\), and \(\boldsymbol{b}=(b_{v})_{v\in V}\in\mathbb{R}^{V}\) associates a _bias_\(b_{v}\) with every node \(v\in V\). As for circuits, the sources of the dag are _input nodes_, and we denote them by \(X_{1},\ldots,X_{p}\). sinks are _output nodes_, and we denote them by \(Y_{1},\ldots,Y_{q}\). We define the _order_\(|\mathfrak{F}|\), the _depth_\(\mathrm{dp}(\mathfrak{F})\), the _input dimension_, and the _output dimension_ of \(\mathfrak{F}\) in the same way as we did for circuits.

To define the semantics, let \(\mathfrak{A}=\big{(}V,E,(\mathfrak{a}_{v})_{v\in V}\big{)}\). For each node \(v\in V\), we define a function \(f_{\mathfrak{A},v}:\mathbb{R}^{p}\times\mathbb{R}^{E}\times\mathbb{R}^{V}\to \mathbb{R}\) inductively as follows. Let \(\boldsymbol{x}=(x_{1},\ldots,x_{p})\in\mathbb{R}^{n}\), \(\boldsymbol{b}=(b_{v})_{v\in V}\in\mathbb{R}^{V}\), and \(\boldsymbol{w}=(w_{e})_{e\in E}\in\mathbb{R}^{E}\). Then

\[f_{\mathfrak{A},v}(\boldsymbol{x},\boldsymbol{w},\boldsymbol{b})\coloneqq \begin{cases}x_{i}&\text{if $v$ is the input node $X_{i}$},\\ \mathfrak{a}_{v}\big{(}b_{v}+\sum_{v^{\prime}\in N^{-}(v)}f_{\mathfrak{A},v^{ \prime}}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})\cdot w_{v^{\prime}v} \big{)}&\text{if $v$ is not an input node}.\end{cases}\]

We define \(f_{\mathfrak{A}}:\mathbb{R}^{p}\times\mathbb{R}^{E}\times\mathbb{R}^{V}\to \mathbb{R}^{q}\) by

\[f_{\mathfrak{A}}(\boldsymbol{x},\boldsymbol{w},\boldsymbol{b})\coloneqq\big{(} f_{\mathfrak{A},Y_{1}}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w}),\ldots,f_{ \mathfrak{A},Y_{q}}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})\big{)}.\]
For an FNN \(\mathfrak{F}=\big{(}V,E,(\mathfrak{a}_{v})_{v\in V},\mathbf{w},\mathbf{b}\big{)}\) with architecture \(\mathfrak{A}=\big{(}V,E,(\mathfrak{a}_{v})_{v\in V}\big{)}\) we define functions \(f_{\mathfrak{F},v}:\mathbb{R}^{p}\to\mathbb{R}\) for \(v\in V\) and \(f_{\mathfrak{F}}:\mathbb{R}^{p}\to\mathbb{R}^{q}\) by

\[f_{\mathfrak{F},v}(\mathbf{x}) =f_{\mathfrak{A},v}(\mathbf{x},\mathbf{w},\mathbf{b}),\] \[f_{\mathfrak{F}}(\mathbf{x}) =f_{\mathfrak{A}}(\mathbf{x},\mathbf{w},\mathbf{b}).\]

As for circuits, to simplify the notation we usually denote the functions \(f_{\mathfrak{A}}\) and \(f_{\mathfrak{F}}\) by \(\mathfrak{A}\) and \(\mathfrak{F}\), respectively.

_Remark 2.4_.: The reader may have noticed that we never use the activation function \(\mathfrak{a}_{v}\) or the bias \(b_{v}\) for input nodes \(v=X_{i}\). We only introduce them for notational convenience. _We may always assume that \(\mathfrak{a}_{v}\equiv 0\) and \(b_{v}=0\) for all input nodes \(v\)._

Typically, the weights \(w_{e}\) and biases \(b_{e}\) are learned from data. We are not concerned with the learning process here, but only with the functions computed by pre-trained models.

_Throughout this paper, we assume the activation function in neural networks to be Lipschitz continuous._ Our theorems can also be proved with weaker assumptions on the activation functions, but assuming Lipschitz continuity simplifies the proofs, and since all activation functions typically used in practice are Lipschitz continuous, there is no harm in making this assumption. Since linear functions are Lipschitz continuous and the concatenation of Lipschitz continuous functions is Lipschitz continuous as well, it follows that for all FNNs \(\mathfrak{F}\) the function \(f_{\mathfrak{F}}\) is Lipschitz continuous. A consequence of the Lipschitz continuity is that the output of an FNN can be linearly bounded in the input. For later reference, we state these facts as a lemma.

**Lemma 2.5**.: _Let \(\mathfrak{F}\) be an FNN of input dimension \(p\)._

1. _There is a Lipschitz constant_ \(\lambda=\lambda(\mathfrak{F})\in\mathbb{N}_{>0}\) _for_ \(\mathfrak{F}\) _such that for all_ \(\mathbf{x},\mathbf{x}^{\prime}\in\mathbb{R}^{p}\)_,_ \[\big{\|}\mathfrak{F}(\mathbf{x})-\mathfrak{F}(\mathbf{x}^{\prime})\big{\|}_{\infty} \leq\lambda\,\big{\|}\mathbf{x}-\mathbf{x}^{\prime}\big{\|}_{\infty}\,.\]
2. _There is a_ \(\gamma=\gamma(\mathfrak{F})\in\mathbb{N}_{>0}\) _such that for all_ \(\mathbf{x}\in\mathbb{R}^{p}\)_,_ \[\|\mathfrak{F}(\mathbf{x})\|_{\infty}\leq\gamma\cdot\big{(}\,\big{\|}\mathbf{x}\big{\|} _{\infty}+1\big{)}.\]

Proof.: Assertion (1) is simply a consequence of the fact that concatenation of Lipschitz continuous functions is Lipschitz continuous. For (2), note that by (1) we have

\[\big{\|}\mathfrak{F}(\mathbf{x})\big{\|}_{\infty}\leq\lambda(\mathfrak{F})\,\big{\|} \mathbf{x}\big{\|}_{\infty}+\big{\|}\mathfrak{F}(\mathbf{0})\big{\|}_{\infty}\,.\]

We let \(\gamma\coloneqq\max\big{\{}\lambda(\mathfrak{F}),\big{\|}\mathfrak{F}(\mathbf{0}) \big{\|}_{\infty}\,\big{\}}\). 

We often make further restrictions on the FNNs we consider. An FNN is _piecewise linear_ if all its activation functions are piecewise linear. An FNN is _rational piecewise linear_ if all weights and biases are dyadic rationals and all activation functions are rational piecewise linear. The relu function and the linearised sigmoid function (see
Example 2.1) are typical examples of rational piecewise linear activation functions. An FNN is _rpl approximable_ if all its activation functions are rpl approximable. The logistic function and the hyperbolic tangent function (see Example 2.2) are typical examples of rpl approximable activation functions.

It is a well known fact that FNNs can simulate threshold circuits.

**Lemma 2.6**.: _For every threshold circuit \(\mathfrak{C}\) of input dimension \(p\) there is an FNN \(\mathfrak{F}=(V,E,(\mathfrak{a}_{v})_{v\in V},(w_{e})_{e\in E},(b_{v})_{v\in V})\) of input dimension \(p\) such that \(|\mathfrak{F}|=O(|\mathfrak{C}|)\), \(\mathfrak{a}_{v}=\operatorname{relu}\) for all \(v\), \(w_{e}\in\{1,-1\}\) for all \(e\), \(b_{v}\in\mathbb{N}\) is bounded by the maximum threshold in \(\mathfrak{C}\) for all \(v\), and \(\mathfrak{C}(\boldsymbol{x})=\mathfrak{F}(\boldsymbol{x})\) for all \(\boldsymbol{x}\in\{0,1\}^{p}\)._

Proof.: We we simulate \(\mathfrak{C}\) gatewise, noting that a Boolean \(\neg x\) negation can be expressed as \(\operatorname{relu}(1-x)\) and a threshold \(\sum x_{i}\geq t\) can be expressed as \(\operatorname{relu}(\sum x_{i}-t+1)-\operatorname{relu}(\sum x_{i}-t)\) for Boolean inputs \(x,x_{i}\). 

### Relational Structures

A _vocabulary_ is a finite set \(\tau\) of relation symbols. Each relation symbol \(R\in\tau\) has an _arity_\(\operatorname{ar}(R)\in\mathbb{N}\). A _\(\tau\)-structure_\(A\) consists of a finite set \(V(A)\), the _universe_ or _vertex set_, and a relation \(R(A)\subseteq V(A)^{k}\) for every relation symbol \(R\in\tau\) of arity \(\operatorname{ar}(R)=k\). For a \(\tau\)-structure \(A\) and a subset \(\tau^{\prime}\subseteq\tau\), the _restriction of \(A\) to \(\tau^{\prime}\)_ is the \(\tau^{\prime}\)-structure \(A|_{\tau^{\prime}}\) with \(V(A|_{\tau^{\prime}})\coloneqq V(A)\) and \(R(A|_{\tau^{\prime}})\coloneqq R(A)\) for all \(R\in\tau^{\prime}\). The _order_ of a structure \(A\) is \(|A|\coloneqq|V(A)|\).

For example, a graph may be viewed as an \(\{E\}\)-structure \(G\), where \(E\) is a binary symbol, such that \(E(G)\) is symmetric and irreflexive. A pair \((G,\ell)\in\mathscr{G}_{\ell}^{\operatorname{bool}}\), that is, a graph with a Boolean signal \(\ell:V(G)\to\{0,1\}^{\ell}\), may be viewed as an \(\{E,P_{1},\ldots,P_{\ell}\}\)-structure \(G_{\ell}\) with \(V(G_{\ell})=V(G)\), \(E(G_{\ell})=E(G)\), and \(P_{i}(G_{\ell})=\{v\in V(G)\mid\ell(v)_{i}=1\}\). We may think of the \(P_{i}\) as labels and hence refer to \(\{E,P_{1},\ldots,P_{\ell}\}\)-structures whose \(\{E\}\)-restriction is an undirected graph as _\(\ell\)-labeled graphs_. In the following, we do not distinguish between graphs with Boolean signals and the corresponding labeled graphs.

A _\(k\)-ary query_ on a class \(\mathscr{C}\) of structures is an equivariant mapping \(\mathscr{G}\) that associates with each structure \(A\in\mathscr{C}\) a mapping \(\mathscr{G}(A):V(A)^{k}\to\{0,1\}\). In this paper, we are mainly interested in \(0\)-ary (or _Boolean_) and unary queries on (labelled) graphs. We observe that a Boolean query on \(\ell\)-labelled graphs is an invariant mapping from \(\mathscr{G}_{\ell}^{\operatorname{bool}}\) to \(\{0,1\}\) and a unary query is an equivariant signal transformations from \(\mathscr{G}_{\ell}^{\operatorname{bool}}\) to \(\mathscr{G}_{1}^{\operatorname{bool}}\).

## 3 First-Order Logic with Counting

Throughout this section, we fix a vocabulary \(\tau\). We introduce two types of variables, _vertex variables_ ranging over the vertex set of a structure, and _number variables_ ranging over \(\mathbb{N}\). We typically denote vertex variables by \(x\) and variants like \(x^{\prime},x_{1}\), number variables by \(y\) and variants, and we use \(z\) and variants to refer to either vertex or number variables.
We define the sets of \(\mathsf{FO}+\mathsf{C}\)_-formulas_ and \(\mathsf{FO}+\mathsf{C}\)_-terms_ of vocabulary \(\tau\) inductively as follows:

* All number variables and \(\mathsf{0},\mathsf{1}\) are \(\mathsf{FO}+\mathsf{C}\)-terms.
* For all \(\mathsf{FO}+\mathsf{C}\)-terms \(\theta,\theta^{\prime}\) the expressions \(\theta+\theta^{\prime}\) and \(\theta\cdot\theta^{\prime}\) are \(\mathsf{FO}+\mathsf{C}\)-terms.
* For all \(\mathsf{FO}+\mathsf{C}\)-terms \(\theta,\theta^{\prime}\) the expression \(\theta\leq\theta^{\prime}\) is an \(\mathsf{FO}+\mathsf{C}\)-formula.
* For all vertex variables \(x_{1},\ldots,x_{k}\) and all \(k\)-ary \(R\in\tau\) the expressions \(x_{1}=x_{2}\) and \(R(x_{1},\ldots,x_{k})\) are \(\mathsf{FO}+\mathsf{C}\)-formulas (_relational atoms_).
* For all \(\mathsf{FO}+\mathsf{C}\)-formulas \(\varphi,\psi\) the expressions \(\neg\varphi\) and \(\varphi\wedge\psi\) are \(\mathsf{FO}+\mathsf{C}\)-formulas.
* For all \(\mathsf{FO}+\mathsf{C}\)-formulas \(\varphi\), all \(k,\ell\in\mathbb{N}\) with \(k+\ell\geq 1\), all vertex variables \(x_{1},\ldots,x_{k}\), all number variables \(y_{1},\ldots,y_{\ell}\), and all \(\mathsf{FO}+\mathsf{C}\)-terms \(\theta_{1},\ldots,\theta_{\ell}\), \[\#(x_{1},\ldots,x_{k},y_{1}<\theta_{1},\ldots,y_{\ell}<\theta_{\ell}).\varphi\] is an \(\mathsf{FO}+\mathsf{C}\)-term (a _counting term_).

A _\(\tau\)-interpretation_ is a pair \((A,\alpha)\), where \(A\) is a \(\tau\)-structure and \(\alpha\) is an _assignment over \(A\)_, that is, a mapping from the set of all variables to \(V(A)\cup\mathbb{N}\) such that \(\alpha(x)\in V(A)\) for every vertex variable \(x\) and \(\alpha(y)\in\mathbb{N}\) for every number variable \(y\). For a tuple \(\boldsymbol{z}=(z_{1},\ldots,z_{k})\) of distinct variables, and a tuple \(\boldsymbol{c}=(c_{1},\ldots,c_{k})\in(V(A)\cup\mathbb{N})^{k}\) such that \(c_{i}\in V(A)\) if \(z_{i}\) is a vertex variable and \(c_{i}\in\mathbb{N}\) if \(z_{i}\) is a number variable, we let \(\alpha\frac{\boldsymbol{c}}{\boldsymbol{z}}\) be the interpretation with \(\alpha\frac{\boldsymbol{c}}{\boldsymbol{z}}(z_{i})=c_{i}\) and \(\alpha\frac{\boldsymbol{c}}{\boldsymbol{z}}(z)=\alpha(z)\) for all \(z\not\in\{z_{1},\ldots,z_{k}\}\). We inductively define a value \([\![\theta]\!]^{(A,a)}\in\mathbb{N}\) for each \(\mathsf{FO}+\mathsf{C}\)-term \(\theta\) and a Boolean value \([\![\varphi]\!]^{(A,a)}\in\{0,1\}\) for each \(\mathsf{FO}+\mathsf{C}\)-formula \(\varphi\).

* We let \([\![y]\!]^{(A,a)}\coloneqq\alpha(y)\) and \([\![0]\!]^{(A,a)}\coloneqq 0\), \([\![1]\!]^{(A,a)}\coloneqq 1\).
* We let \([\![\theta+\theta^{\prime}]\!]^{(A,a)}\coloneqq[\![\theta]\!]^{(A,a)}+[\![ \theta^{\prime}]\!]^{(A,a)}\) and \([\![\theta\cdot\theta^{\prime}]\!]^{(A,a)}\coloneqq[\![\theta]\!]^{(A,a)}\cdot[ \![\theta^{\prime}]\!]^{(A,a)}\).
* We let \([\![\theta\leq\theta^{\prime}]\!]^{(A,a)}=1\) if and only if \([\![\theta]\!]^{(A,a)}\leq[\![\theta^{\prime}]\!]^{(A,a)}\).
* We let \([\![x_{1}=x_{2}]\!]^{(A,a)}=1\) if and only if \(\alpha(x_{1})=\alpha(x_{2})\) and \([\![R(x_{1},\ldots,x_{k})]\!]^{(A,a)}=1\) if and only if \(\alpha(x_{1}),\ldots,\alpha(x_{k})\big{)}\in R(A)\).
* We let \([\![\neg\varphi]\!]^{(A,a)}\coloneqq 1-[\![\varphi]\!]^{(A,a)}\) and \([\![\varphi\wedge\psi]\!]^{(A,a)}\coloneqq[\![\varphi]\!]^{(A,a)}\cdot[\![ \varphi]\!]^{(A,a)}\).
* We let \[[\![\#(x_{1},\ldots,x_{k},y_{1}<\theta_{1},\ldots,y_{\ell}<\theta_{\ell}). \varphi]\!]^{(A,a)}\] be the number of tuples \((a_{1},\ldots,a_{k},b_{1},\ldots,b_{\ell})\in V(A)^{k}\times\mathbb{N}^{\ell}\) such that
* \(b_{i}<[\![\theta_{i}]\!]^{(A,a\frac{(a_{1},\ldots,a_{k},b_{1},\ldots,b_{i-1})} {(x_{1},\ldots,x_{k},y_{1},\ldots,y_{i-1})})}\) for all \(i\in[\ell]\);
* \([\![\varphi]\!]^{(A,a\frac{(a_{1},\ldots,a_{k},b_{1},\ldots,b_{\ell})}{(x_{1}, \ldots,x_{k},y_{1},\ldots,y_{\ell})})}=1\).
Note that we allow _adaptive bounds_: the bound on the variable \(y_{i}\) may depend on the values for all previous variables. While it can be shown that this does not increase the expressive power of the plain logic, the adaptive bounds do add power to an extension of the logic with function variables (see Section 3.4).

For \(\mathsf{FO}+\mathsf{C}\)-formulas \(\varphi\), instead of \(\llbracket\varphi\rrbracket^{(A,a)}=1\) we also write \((A,\mathpzc{e})\vDash\varphi\).

An \(\mathsf{FO}+\mathsf{C}\)_-expression_ is either an \(\mathsf{FO}+\mathsf{C}\)-term or an \(\mathsf{FO}+\mathsf{C}\)-formula. The set \(\operatorname{free}(\xi)\) of _free variables_ of an \(\mathsf{FO}+\mathsf{C}\)-expression \(\xi\) is defined inductively in the obvious way, where for a counting term we let

\[\operatorname{free}\big{(}\#(x_{1},\ldots,x_{k},y_{1}<\theta_{1},\ldots,y_{\ell}<\theta_{\ell}).\varphi\big{)}\coloneqq\\ \big{(}\operatorname{free}(\varphi)\setminus\{x_{1},\ldots,x_{k},y_{1},\ldots,y_{\ell}\}\big{)}\cup\bigcup_{i=1}^{\ell}\operatorname{free} \big{(}\theta_{i}\setminus\{x_{1},\ldots,x_{k},y_{1},\ldots,y_{i-1}\}\big{)}.\]

A _closed term_ is a term without free variables, and a _sentence_ is a formula without free variables.

For an expression \(\xi\), the notation \(\xi(z_{1},\ldots,z_{k})\) stipulates that \(\operatorname{free}(\xi)\subseteq\{z_{1},\ldots,z_{k}\}\). It is easy to see that the value \(\llbracket\xi\rrbracket^{(A,a)}\) only depends on the interpretations \(c_{i}\coloneqq\mathpzc{e}(z_{i})\) of the free variables. Thus we may avoid explicit reference to the assignment \(\mathpzc{e}\) and write \(\llbracket\xi\rrbracket^{A}\,(c_{1},\ldots,c_{k})\) instead of \(\llbracket\xi\rrbracket^{(A,a)}\). If \(\xi\) is a closed term or sentence, we just write \(\llbracket\xi\rrbracket^{A}\). For formulas \(\varphi(z_{1},\ldots,z_{k})\), we also write \(A\vDash\varphi(c_{1},\ldots,c_{k})\) instead of \(\llbracket\varphi\rrbracket^{A}\,(c_{1},\ldots,c_{k})=1\)..

Observe that every \(\mathsf{FO}+\mathsf{C}\)-formula \(\varphi(x_{1},\ldots,x_{k})\) of vocabulary \(\tau\) defines a \(k\)-ary query on the class of \(\tau\)-structures, mapping a structure \(A\) to the set of all \((a_{1},\ldots,a_{k})\in A^{k}\) such that \(A\vDash\varphi(a_{1},\ldots,a_{k})\).

We defined the logic \(\mathsf{FO}+\mathsf{C}\) with a minimal syntax, avoiding unnecessary operators. However, we can use other standard arithmetical and logical operators as abbreviations:

* For \(n\geq 2\), we can use \(n\) as an abbreviation for the corresponding sum of \(1\)s.
* We use \(\mathsf{ord}\) as an abbreviation for the term \(\#x.x=x\). Then \(\llbracket\mathsf{ord}\rrbracket^{A}=|A|\) for all structures \(A\), that is, \(\mathsf{ord}\) defines the order of a structure.
* We can express the relations \(\coloneqq,\geq,<,>\) on \(\mathbb{N}\) using Boolean combinations and \(\leq\).
* We can express Boolean connectives like \(\vee\) or \(\to\) using \(\neg\) and \(\wedge\).
* For vertex variables \(x\), we can express existential quantification \(\exists x.\varphi\) as \(1\leq\#x.\varphi\). Then we can express universal quantification \(\forall x.\varphi\) using \(\exists x\) and \(\neg\) in the usual way. In particular, this means that we can view first-order logic \(\mathsf{FO}\) as a fragment of \(\mathsf{FO}+\mathsf{C}\).
* For number variables \(y\), we can similarly express bounded quantification \(\exists y<\theta.\varphi\) and \(\forall y<\theta.\varphi\).
* In counting terms, we do not have to use strict inequalities to bound number variables. For example, we write \(\#(x,y\leq\theta).\varphi\) to abbreviate \(\#(x,y<\theta+1).\varphi\).
* We can express truncated subtraction \(\tilde{\preceq}\), minimum and maximum of two numbers, and integer division: \[y\tilde{\preceq}y^{\prime} \text{ abbreviates }\#(y^{\prime\prime}<y).(y^{\prime}\leq y^{\prime \prime}),\] \[\mathsf{min}(y,y^{\prime}) \text{ abbreviates }\#(y^{\prime\prime}<y).y^{\prime\prime}<y^{\prime},\] \[\mathsf{max}(y,y^{\prime}) \text{ abbreviates }\#(y^{\prime\prime}<y+y^{\prime}).(y^{\prime \prime}<y\lor y^{\prime\prime}<y^{\prime}),\] \[\mathsf{div}(y,y^{\prime}) \text{ abbreviates }\#(y^{\prime\prime}\leq y).y^{\prime}\cdot y^{ \prime\prime}<y.\] Note that for all structures \(A\) and \(b,b^{\prime}\in\mathbb{N}\) we have \[\left[\mathsf{div}\right]^{A}(b,b^{\prime})=\begin{cases}\left\lfloor\frac{ b}{b^{\prime}}\right\rfloor&\text{if }b^{\prime}\neq 0,\\ b+1&\text{if }b^{\prime}=0\text{ and }b\neq 0,\\ 0&\text{if }b^{\prime}=b=0.\end{cases}\]

_Remark 3.1_.: There are quite a few different versions of first-order logic with counting in the literature. The logics that only involve counting quantifiers \(\exists^{\geq n}\) for constant \(n\) are strictly weaker than our \(\mathsf{FO+C}\), and so are logics with modular counting quantifiers.

Counting logics with quantification over numbers have first been suggested, quite informally, by Immerman [16]. The 2-sorted framework was later formalised by Gradel and Otto [11]. Essentially, our \(\mathsf{FO+C}\) corresponds to Kuske and Schweikardt's [20]\(\mathsf{FCON}(\{\mathbb{P}_{\leq}\})\), with one important difference: we allow counting terms also over number variables. This makes no difference over ordered structures, but it makes the logic stronger over unordered structures (at least we conjecture that it does; this is something that with current techniques one can probably only prove modulo some complexity theoretic assumptions). Other differences, such as that [20] use the integers for the numerical part, whereas we use the non-negative integers, are inessential. Importantly, the two logics and other first-order logics with counting, such as first-order logic with a majority quantifier, are equivalent over ordered arithmetic structures and thus all capture the complexity class uniform \(\mathsf{TC}^{0}\), as will be discussed in the next section.

A simple lemma that we will frequently use states that all \(\mathsf{FO+C}\)-terms are polynomially bounded. At this point, the reader may safely ignore the reference to function variables in the assertion of the lemma; we will only introduce them in Section 3.3. We just mention them here to avoid confusion in later applications of the lemma.

**Lemma 3.2**.: _For every \(\mathsf{FO+C}\)-term \(\theta(x_{1},\ldots,x_{k},y_{1},\ldots,y_{k})\) without function variables there is a polynomial \(\pi(X,Y)\) such that for all structures \(A\), all \(a_{1},\ldots,a_{k}\in V(A)\), and all \(b_{1},\ldots,b_{\ell}\in\mathbb{N}\) it holds that_

\[\theta^{A}(a_{1},\ldots,a_{k},b_{1},\ldots,b_{k})\leq\pi\Big{(}|A|,\max\big{\{} b_{i}\bigm{|}i\in[\ell]\big{\}}\Big{)}.\]

Proof.: A straightforward induction on \(\theta\).
### Descriptive Complexity

We review some results relating the logic \(\mathsf{FO}+\mathsf{C}\) to the complexity class \(\mathsf{TC}^{0}\). In the descriptive complexity theory of "small" complexity classes (say, within \(\mathsf{PTIME}\)), we need to expand structures by a linear order of the vertex set (and possibly additional arithmetical relations). We introduce a distinguished binary relation symbol \(\leqslant\), which we assume to be not contained in the usual vocabularies \(\tau\). Note that \(\leqslant\) is distinct from \(\leq\), which we use for the standard linear order on \(\mathbb{N}\). We denote the interpretation of \(\leqslant\) in a structure \(A\) by \(\leqslant^{A}\) instead of \(\leqslant(A)\), and we use the symbol in infix notation.

An _ordered \(\tau\)-structure_ is a \(\tau\cup\{\leqslant\}\)-structure \(A\) where \(\leqslant^{A}\) is a linear order of the vertex set \(V(A)\). It will be convenient to have the following notation for ordered structures \(A\). For \(0\leq i<n\coloneqq|A|\), we \(\langle i\rangle_{A}\) be the \((i+1)\)st element of the linear order "\(\leqslant^{A}\), that is, we have \(V(A)=\{\langle i\rangle_{A}\mid 0\leq i<n\}\) with \(\langle 0\rangle_{A}\leqslant^{A}\langle 1\rangle_{A}\leqslant^{A}... \leqslant^{A}\left(n-1\right)_{A}\). We omit the subscript \({}_{A}\) if \(A\) is clear from the context. The reason that ordered structures are important in descriptive complexity is that they have simple canonical representations as bitstrings. Let \(s(A)\in\{0,1\}^{*}\) denote the string representing an ordered structure \(A\). Then for every class \(\mathscr{C}\) of ordered \(\tau\)-structures, we let \(L(\mathscr{C})=\{s(A)\mid A\in\mathscr{C}\}\).

**Theorem 3.3** (Barrington, Immerman, and Straubing [4]).: _Let \(\mathscr{C}\) be a class of ordered \(\tau\)-structures. Then \(L(\mathscr{C})\) is in uniform \(\mathsf{TC}^{0}\) if and only if there is an \(\mathsf{FO}+\mathsf{C}\)-sentence \(\psi\) of vocabulary \(\tau\cup\{\leqslant\}\) such that for all ordered \(\tau\)-structures \(A\) it holds that \(A\in\mathscr{C}\iff A\models\psi\)._

We need to rephrase this theorem for queries. While the theorem is really about Boolean queries, we state a version for unary queries. For a class \(\mathscr{C}\) of \(\tau\)-structures, we let \(\mathscr{C}_{\leqslant}\) be the class of all ordered \(\tau\) structures \(A\) with \(A|_{\tau}\in\mathscr{C}\), and we let \(L_{\leqslant}(\mathscr{C})\coloneqq L(\mathscr{C}_{\leqslant})\). Extending this to unary queries, for a query \(\mathscr{C}\) on a class \(\mathscr{C}\) of \(\tau\)-structures, we let

\[L_{\leqslant}(\mathscr{C})\coloneqq\left\{s(A)\#\operatorname{bin}(i)\mathrel {\big{|}}A\in\mathscr{C}_{\leqslant},\mathscr{C}(A)(\langle i\rangle)=1\right\}\]

We say that a formula \(\varphi(x)\) of vocabulary \(\tau\cup\{\leqslant\}\) is _order invariant_ if for all ordered \(\tau\)-structures \(A,A^{\prime}\) with \(A|\tau=A^{\prime}|\tau\) and all \(a\in V(A)\) it holds that \(A\models\varphi(a)\iff A^{\prime}\models\varphi(a)\). We say that a unary query \(\mathscr{C}\) on a class of \(\tau\)-structures is definable in _order-invariant_\(\mathsf{FO}+\mathsf{C}\) if there is an order invariant \(\mathsf{FO}+\mathsf{C}\)-formula \(\varphi(x)\) of vocabulary \(\tau\cup\{\leqslant\}\) such that for all \(A\in\mathscr{C}_{\leqslant}\) and all \(a\in V(A)\) it holds that \(A\models\varphi(a)\iff\mathscr{C}(A|_{\tau})(a)=1\).

**Corollary 3.4**.: _Let \(\mathscr{C}\) be a unary query. Then \(L_{\leqslant}(\mathscr{C})\) is in uniform \(\mathsf{TC}^{0}\) if and only if \(\mathscr{C}\) is definable in order-invariant \(\mathsf{FO}+\mathsf{C}\)._

### Non-Uniformity and Built-in Relations

The usual way to capture non-uniformity in descriptive complexity is by adding _built-in relations_. The classical way is to only consider structures with universe \(\{0,\ldots,n-1\}\), for some \(n\in\mathbb{N}\), and then add relation symbols \(S\) to the language that have a fixed interpretation \(S^{(n)}\subseteq\{0,\ldots,n-1\}^{k}\) in all structures with universe \(\{0,\ldots,n-1\}\). Slightly more abstractly, we can consider ordered structures and transfer the definition of \(S^{(n)}\) to all linearly ordered structures of order \(n\) via the natural mapping \(i\mapsto\langle i\rangle\).
We take a slightly different approach to built-in relations here, which allows us to also use them over structures that are not necessarily ordered. A _built-in numerical relation_ is simply a relation over \(\mathbb{N}\), that is, a subset \(N\subseteq\mathbb{N}^{k}\) for some \(k\geq 0\), the _arity_ of \(N\). We use the same letter \(N\) to denote both the relation \(N\subseteq\mathbb{N}^{k}\) and a \(k\)-ary relation symbol representing it in the logic. In other words, the relation symbol \(N\) will be interpreted by the same relation \(N\subseteq\mathbb{N}^{k}\) in all structures. We extend the logic \(\mathsf{FO}\!+\!\mathsf{C}\) by new atomic formulas \(N(y_{1},\ldots,y_{k})\) for all \(k\)-ary numerical relations \(N\) and number variables \(y_{1},\ldots,y_{k}\), with the obvious semantics. By \(\mathsf{FO}\!+\!\mathsf{C}_{\mathrm{nu}}\) we denote the extension of \(\mathsf{FO}\!+\!\mathsf{C}\) to formulas using arbitrary built-in numerical relations.2

Footnote 2: Think of the index ‘nu’ as an abbreviation of either ‘numerical’ or ‘non-uniform’

Then it follows easily from Theorem 3.3 that \(\mathsf{FO}\!+\!\mathsf{C}_{\mathrm{nu}}\) captures (non-uniform) \(\mathsf{TC}^{0}\). (Or it can be proved directly, in fact, it is much easier to prove than Theorem 3.3.)

**Corollary 3.5**.: _Let \(\mathscr{C}\) be a class of ordered \(\tau\)-structures. Then \(L(\mathscr{C})\) is in \(\mathsf{TC}^{0}\) if and only if there is an \(\mathsf{FO}\!+\!\mathsf{C}_{\mathrm{nu}}\)-sentence \(\psi\) of vocabulary \(\tau\cup\{\leqslant\}\) such that for all ordered \(\tau\)-structures \(A\) it holds that \(A\in\mathscr{C}\iff A\vDash\psi\)._

We also state the version of this result for unary queries.

**Corollary 3.6**.: _Let \(\mathscr{G}\) be a unary query. Then \(L_{\xi}(\mathscr{G})\) is in \(\mathsf{TC}^{0}\) if and only if \(\mathscr{G}\) is definable in order-invariant \(\mathsf{FO}\!+\!\mathsf{C}_{\mathrm{nu}}\)._

### Types and Second-Order Variables

The counting extension of first-order logic refers to a 2-sorted extension of relational structures and adheres to a strict type discipline. For the extension we are going to introduce next, we need to make this formal. We assign a _type_ to each variable: a vertex variable has type \(\mathtt{v}\), and number variable has type \(\mathtt{n}\). A \(k\)-tuple \((z_{1},\ldots,z_{k})\) of variables has a type \((t_{1},\ldots,t_{k})\in\{\mathtt{v},\mathtt{n}\}^{k}\), where \(t_{i}\) is the type of \(z_{i}\). We denote the type of a tuple \(\boldsymbol{z}\) by \(\mathrm{tp}(\boldsymbol{z})\). For a structure \(A\) and a type \(\boldsymbol{t}=(t_{1},\ldots,t_{k})\in\{\mathtt{v},\mathtt{n}\}^{k}\) we let \(A^{\boldsymbol{t}}\) be the set of all \((c_{1},\ldots,c_{k})\in(V(A)\cup\mathbb{N})^{k}\) such that \(c_{i}\in V(A)\) if \(t_{i}=\mathtt{v}\) and \(c_{i}\in\mathbb{N}\) if \(t_{i}=\mathtt{n}\).

Now we extend our logic by relation variables (denoted by uppercase letters \(X,Y\) and function variables (denoted by \(U,V\)). Each relation variable \(X\) has a type \(\mathrm{tp}(X)\) of the form \(\{\boldsymbol{t}\}\), and each function variable \(U\) has a type \(\mathrm{tp}(U)\) of the form \(\boldsymbol{t}\to\mathtt{n}\), for some \(\boldsymbol{t}\in\{\mathtt{v},\mathtt{n}\}^{k}\). We extend the logic \(\mathsf{FO}\!+\!\mathsf{C}\) by allowing additional atomic formulas \(X(\xi_{1},\ldots,\xi_{k})\) and terms \(U(\xi,\ldots,\xi_{k})\), where \(X\) is a relation variable of type \(\{(t_{1},\ldots,t_{k})\}\) for some tuple \((t_{1},\ldots,t_{k})\in\{\mathtt{v},\mathtt{n}\}^{k}\), \(U\) a function variable of type \((t_{1},\ldots,t_{k})\to\mathtt{n}\), and for all \(i\in[k]\), if \(t_{i}=\mathtt{v}\) then \(\xi_{i}\) is a vertex variable and if \(t_{i}=\mathtt{n}\) then \(\xi_{i}\) is a term.

To define the semantics, let \(A\) be a structure and \(\omega\) an assignment over \(A\). Then \(\omega\) maps each relation variable \(X\) of type \(\{\boldsymbol{t}\}\) to a subset \(\omega(X)\subseteq A^{\boldsymbol{t}}\) and each function variable \(U\) of type \(\boldsymbol{t}\to\mathtt{n}\) to a function \(\omega(U):A^{\boldsymbol{t}}\to\mathbb{N}\). Moreover, for a tuple \(\boldsymbol{\xi}=(\xi_{1},\ldots,\xi_{k})\) of vertex variables and terms we let \([\boldsymbol{\xi}]^{A,e})=(c_{1},\ldots,c_{k})\) where \(c_{i}=\omega(\xi_{i})\) if \(\xi_{i}\) is a vertex variables and \(c_{i}=[\xi_{i}]^{(A,e)}\) if \(c_{i}\) is a term. We let

\[[\![X(\boldsymbol{\xi})]^{(A,\omega)}\coloneqq\begin{cases}1&\text{if }[ \boldsymbol{\xi}]^{A,a})\in\omega(X),\\ 0&\text{otherwise},\end{cases}\]
\[\llbracket U(\mathbf{\xi})\rrbracket^{(A,a)}\coloneqq a(U)\big{(}\,\llbracket\mathbf{\xi} \rrbracket^{A,a}\,\big{)}.\]

Observe that a function variable \(U\) of type \(\varnothing\to\texttt{n}\) is essentially just a number variable, if we identify a \(0\)-ary function with the value it takes on the empty tuple \(\varnothing\). It is still useful sometimes to use \(0\)-ary function variables. We usually write \(\omega(U)\) instead of \(\omega(U)(\varnothing)\) to denote their value. We call a relation variable _purely numerical_ if it is of type \(\{\texttt{n}^{k}\}\), for some \(k\geq 0\). Similarly, we call a function variable _purely numerical_ if it is of type \(\texttt{n}^{k}\to\texttt{n}\). To distinguish them from the "second-order" relation and function variables, we refer to our original "first-order" vertex variables and number variables as _individual variables_.

When we list variables of an expression in parentheses, as in \(\xi(\mathbf{z})\), we only list the free individual variables, but not the free relation or function variables. Thus \(\xi(\mathbf{z})\) stipulates that all free individual variables of \(\xi\) occur in \(\mathbf{z}\). However, \(\xi\) may have free relation variables and free function variables that are not listed in \(\mathbf{z}\). For a structure \(A\), an assignment \(\omega\), and a tuple \(\mathbf{c}\in A^{\text{tp}(\mathbf{z})}\), we write \(\llbracket\xi\rrbracket^{(A,a)}\,(\mathbf{c})\) instead of \(\llbracket\xi\rrbracket^{(A,a,\frac{\mathbf{c}}{\mathbf{z}})}\). If \(\xi\) is a formula, we may aslo write \((A,\omega)\models\xi(\mathbf{c})\).

The role of relation variables and function variables is twofold. First, we will use them to specify "inputs" for formulas, in particular formulas defining numerical functions. (In the next sections we will see how to use relation and function variables to specify natural and rational numbers.) And second, we may just use relation and function variables as placeholders for formulas and terms that we may later substitute for them.

Note that Lemma 3.2 no longer holds if the term \(\theta\) contains function variables, because these variables may be interpreted by functions of super-polynomial growth. Nevertheless, even in the presence of relation and function variables, our logic remains "first-order", because we do not allow quantification over theses variables.

### Arithmetic in FO+C

In this section, we will show that arithmetic on bitwise representations of integers is expressible in FO+C. Almost none of the formulas we shall define make any reference to a structure \(A\); they receive their input in the form of purely numerical relation variables and function variables and only refer to the numerical part, which is the same for all structures. We call an FO+C-expression \(\xi\)_arithmetical_ if it contains no vertex variables. Note that if \(\xi\) is an arithmetical expression, then for all structures \(A,A^{\prime}\) and all assignments \(\omega,\omega^{\prime}\) over \(A,A^{\prime}\), respectively, such that \(\omega(y)=\omega^{\prime}(y)\) for all number variables \(y\) and \(\omega(Z)=\omega^{\prime}(Z)\) for all purely numerical relation or function variables \(Z\), we have \(\llbracket\xi\rrbracket^{(A,a)}=\llbracket\xi\rrbracket^{(A^{\prime},\omega^{ \prime})}\). Thus there is no need to mention \(A\) at all; we may write \(\llbracket\xi\rrbracket^{a}\) instead of \(\llbracket\xi\rrbracket^{(A,a)}\). In fact, we can even use the notation \(\llbracket\xi\rrbracket^{a}\) if \(\omega\) is only a partial assignment that assigns values only to number variables and purely numerical relation and function variables. We call such a partial assignment a _numerical_ assignment. As usual, if \(\xi=\xi(y_{1},\ldots,y_{k})\) has all free individual variables among \(y_{1},\ldots,y_{k}\), for \(b_{1},\ldots,b_{k}\in\mathbb{N}\) we may write \(\llbracket\xi\rrbracket^{a}\,(b_{1},\ldots,b_{k})\) instead of \(\llbracket\xi\rrbracket^{a}\frac{b_{1}\ldots b_{k}}{y_{1}\ldots,y_{k}}\). If, in addition, \(\xi\) has no free relation or function variables, we may just write \(\llbracket\xi\rrbracket\,(b_{1},\ldots,b_{k})\).
It is worth mentioning that arithmetical \(\mathsf{FO}+\mathsf{C}\)-formulas are formulas of _bounded arithmetic_ (see, for example, [13]) augmented by bounded counting terms.

The following well-known lemma is the foundation for expressing bitwise arithmetic in \(\mathsf{FO}+\mathsf{C}\). The proof for the logic with counting is easy. There is also a (significantly deeper) version of the lemma for first-order logic without counting, which goes back to Bennett [5] (see [17, Section 1.2.1] for a proof).

**Lemma 3.7**.: _There is an arithmetical \(\mathsf{FO}+\mathsf{C}\)-formula \(\mathsf{bit}(y,y^{\prime})\) such that for all \(i,n\in\mathbb{N}\),_

\[\llbracket\mathsf{bit}\rrbracket\,(i,n)=\mathrm{Bit}(i,n).\]

Proof.: Clearly, there is a formula \(\mathsf{pow2}(y)\) expressing that \(y\) is a power of \(2\); it simply states that all divisors of \(y\) are divisible by \(2\). Then the formula

\[\mathsf{exp2}(y,y^{\prime})\coloneqq\mathsf{pow2}(y^{\prime})\wedge y=\#y^{ \prime\prime}<y^{\prime}.\mathsf{pow2}(y^{\prime\prime})\]

expresses that \(y^{\prime}=2^{y}\).

To express the bit predicate, we define the auxiliary formula

\[\mathsf{pow2bit}(y,y^{\prime})\coloneqq\mathsf{pow2}(y)\wedge\exists y_{1}< y^{\prime}.\exists y_{2}<y.y^{\prime}=2y_{1}y+y+y_{2},\]

expressing that \(y\) is \(2^{i}\) for some \(i\) and the \(i\)th bit of \(y^{\prime}\) is \(1\). We let

\[\mathsf{bit}(y,y^{\prime})\coloneqq\exists y^{\prime\prime}<y^{\prime}. \big{(}\mathsf{exp2}(y,y^{\prime\prime})\wedge\mathsf{pow2bit}(y^{\prime\prime },y^{\prime})\big{)}.\qed\]

**Corollary 3.8**.: _There is an arithmetical \(\mathsf{FO}+\mathsf{C}\)-term \(\mathsf{len}(y)\) such that for all \(n\in\mathbb{N}\),_

\[\llbracket\mathsf{len}\rrbracket\,(n)=|\,\mathrm{bin}(n)|.\]

Proof.: Observe that for \(n\geq 1\),

\[|\,\mathrm{bin}(n)|=1+\max\big{\{}i\,\big{|}\,\mathrm{Bit}(i,n)=1\big{\}}.\]

Noting that \(|\,\mathrm{bin}(n)|\leq n\) for all \(n\geq 1\), the following term defines the length for all \(n\geq 1\):

\[1+\#z<y.\exists y^{\prime}\leq y.\Big{(}z<y^{\prime}\wedge\mathsf{bit}(y^{ \prime},y)\wedge\forall y^{\prime\prime}\leq y\big{(}y^{\prime}<y^{\prime \prime}\to\neg\mathsf{bit}(y^{\prime\prime},y)\big{)}\Big{)}.\]

Note that that this term also gives us the correct result for \(n=0\), simply because the formula \(\exists y^{\prime}\leq y.\Big{(}z<y^{\prime}\wedge\ldots\big{)}\) is false for all \(z\) if \(y=0\). 

In the proof of the previous lemma we used a trick that is worthwhile to be made explicit. Suppose we have a formula \(\varphi(y,\boldsymbol{z})\) that defines a function \(\boldsymbol{z}\mapsto y\), that is, for all structures \(A\) and \(\boldsymbol{c}\in A^{\mathrm{tp}(\boldsymbol{z})}\) there is a unique \(b=f_{A}(\boldsymbol{c})\in\mathbb{N}\) such that \(A\vDash\varphi(b,\boldsymbol{c})\). Often, we then want a term expressing the same function. In general, there is no such term, because the function may grow too fast. (Recall that all terms are polynomially bounded by Lemma 3.2, but the function \(f_{A}\) may grow exponentially fast.) Suppose, however, that we have a term \(\theta(\boldsymbol{z})\) that yields an upper bound for this function, that
is, \(f_{A}(\mathbf{c})<\left[\theta\right]^{A}(\mathbf{c})\) for all \(A\) and \(\mathbf{c}\in A^{\operatorname{tp}(\mathbf{z})}\). Then we obtain a term \(\eta(\mathbf{z})\) such that \(\left[\eta\right]^{A}(\mathbf{c})=f_{A}(\mathbf{c})\) as follows: we let

\[\eta(\mathbf{z})\coloneqq\#y^{\prime}<\theta(\mathbf{z}).\exists y<\theta(\mathbf{z}). \big{(}y^{\prime}<y\wedge\varphi(y,\mathbf{z})\big{)}.\]

It is our goal for the rest of this section to express bitwise arithmetic in \(\mathsf{FO}+\mathsf{C}\). We will use relation variables to encode binary representations of natural numbers. Let \(Y\) be a relation variable of type \(\mathsf{n}\), and let \(\omega\) be a numerical interpretation. We think of \(Y\) as representing the number whose \(i\)th bit is \(1\) if and only if \(i\in\omega(Y)\). But as \(\omega(Y)\) may be infinite, this representation is not yet well defined. We also need to specify a bound on the number of bits we consider, which we can specify by a \(0\)-ary function variable \(U\). Then the pair \((Y,U)\) represents the number

\[\left\langle\!\left\langle Y,U\right\rangle\!\right\rangle^{\omega}\coloneqq \sum_{i\in\omega(Y),i<\omega(U)}2^{i} \tag{3.A}\]

We can also specify numbers by formulas and terms. We let \(\widehat{y}\) be a distinguished number variable (that we fix for the rest of this paper). Let \(\chi\) be a formula and \(\theta\) a term. We usually assume that \(\widehat{y}\) occurs freely in \(\chi\) and does not occur in \(\theta\), but neither is necessary. Let \(A\) be a structure and \(\omega\) an assignment over \(A\). Recall that \(\omega\frac{i}{\widehat{y}}\) denotes the assignment that maps \(\widehat{y}\) to \(i\) and coincides with \(\omega\) on all other variables. We let

\[\left\langle\!\left\langle\chi,\theta\right\rangle\!\right\rangle^{(A,a)} \coloneqq\sum_{\begin{subarray}{c}(A,e,\frac{1}{\widehat{y}})\succeq\chi,\\ i<\left[\theta\right]^{(A,a)}\end{subarray}}2^{i}. \tag{3.B}\]

If \(\chi\) and \(\theta\) are arithmetical, we may write \(\left\langle\!\left\langle\chi,\theta\right\rangle\!\right\rangle^{\omega}\) instead of \(\left\langle\!\left\langle\chi,\theta\right\rangle\!\right\rangle^{(A,a)}\).

The following Lemmas 3.9, 3.10, and 3.14 follow easily from the facts that the arithmetic operations are in uniform \(\mathsf{TC}^{0}\) (Lemma 2.3) and \(\mathsf{FO}+\mathsf{C}\) captures uniform \(\mathsf{TC}^{0}\) (Theorem 3.3). However, we find it helpful to sketch at least some of the proofs, in particular the proof of Lemma 3.10 for iterated addition. While researchers in the circuit-complexity community seem to be well-aware of the fact that iterated addition is in _uniform_\(\mathsf{TC}^{0}\), all references we know only show that it is in non-uniform \(\mathsf{TC}^{0}\), so we think it is worthwhile to give a proof of this lemma. Our proof is purely logical, circumventing circuit complexity altogether, so it may be of independent interest.

**Lemma 3.9**.: _Let \(Y_{1},Y_{2}\) be relation variables of type \(\{\mathsf{n}\}\), and let \(U_{1},U_{2}\) be function variables of type \(\varnothing\to\mathsf{n}\)._

1. _There are arithmetical_ \(\mathsf{FO}+\mathsf{C}\)_-formulas_ \(\mathsf{add}\)_,_ \(\mathsf{sub}\) _and arithmetical_ \(\mathsf{FO}+\mathsf{C}\)_-terms_ \(\mathsf{bd}\)_-_\(\mathsf{add}\)_,_ \(\mathsf{bd}\)_-_\(\mathsf{sub}\) _such that for all structures_ \(A\) _and assignments_ \(\omega\) _over_ \(A\)_,_ \[\left\langle\!\left\langle\mathsf{add},\mathsf{bd}\text{-}\mathsf{ add}\right\rangle\!\right\rangle^{\omega} =\left\langle\!\left\langle Y_{1},U_{1}\right\rangle\!\right\rangle^{ \omega}+\left\langle\!\left\langle Y_{2},U_{2}\right\rangle\!\right\rangle^{ \omega},\] \[\left\langle\!\left\langle\mathsf{sub},\mathsf{bd}\text{-}\mathsf{ sub}\right\rangle\!\right\rangle^{\omega} =\left\langle\!\left\langle Y_{1},U_{1}\right\rangle\!\right\rangle^{ \omega}\doteq\left\langle\!\left\langle Y_{2},U_{2}\right\rangle\!\right\rangle^{ \omega}.\]
2. _There is an arithmetical_ \(\mathsf{FO}+\mathsf{C}\)_-formula_ \(\mathsf{leq}\) _such that for all structures_ \(A\) _and assignments_ \(\omega\) _over_ \(A\)_,_ \[\left[\mathsf{leq}\right]^{a}=1\iff\left\langle\!\left\langle Y_{1},U_{1} \right\rangle\!\right\rangle^{a}\leq\left\langle\!\left\langle Y_{2},U_{2} \right\rangle\!\right\rangle^{\omega}.\]
Proof.: The key observation is that we can easily define the carry bits. Suppose that we want to add numbers \(m,n\). Then for \(i\geq 0\), the \(i\)th carry is \(1\) if any only if there is a \(j\leq i\) such that \(\operatorname{Bit}(j,m)=\operatorname{Bit}(j,n)=1\), and for all \(k,j<k\leq i\), either \(\operatorname{Bit}(k,m)=1\) or \(\operatorname{Bit}(k,n)=1\).

We can use a similar observation for subtraction.

Less-than-or-equal-to can easily be expressed directly. 

To define families of numbers, we can use relation and function variables of higher arity, treating the additional entries as parameters. For a type \(\boldsymbol{t}\in\{\mathtt{v},\mathtt{n}\}^{k}\), let \(Y\) be a relation variable of type \(\{\boldsymbol{nt}\}\), and let \(U\) be a number variable of type \(\boldsymbol{t}\to\mathtt{n}\). Then for every structure \(A\), assignment \(\alpha\), and tuple \(\boldsymbol{c}\in A^{\boldsymbol{t}}\) we let

\[\left\langle\!\left\langle Y,U\right\rangle\!\right\rangle^{(A,\alpha)}( \boldsymbol{c})=\sum_{\begin{subarray}{c}(j,\boldsymbol{c})\in\alpha(Y),\\ j<\alpha(U)(\boldsymbol{c})\end{subarray}}2^{j}. \tag{3.C}\]

We can slightly extend this definition to a setting where \(U\) is a function variable of type \(\boldsymbol{t}^{\prime}\to\mathtt{n}\) for some subtuple \(\boldsymbol{t}^{\prime}\) of \(\boldsymbol{t}\). For example, in the following lemma we have \(\boldsymbol{t}=\mathtt{n}\) and \(\boldsymbol{t}^{\prime}=\varnothing\).

**Lemma 3.10**.: _Let \(Y\) be a relation variable of type \(\{\mathtt{(n,n)}\}\), and let \(U\) be a function variable of type \(\varnothing\to\mathtt{n}\). Then there is an arithmetical \(\mathsf{FO}+\mathsf{C}\)-formula \(\mathsf{s}\)-\(\mathsf{itadd}\) and an arithmetical \(\mathsf{FO}+\mathsf{C}\)-term \(\mathsf{bd}\mathsf{s}\)-\(\mathsf{itadd}\)3 such that for all numerical assignments \(\alpha\) we have_

Footnote 3: The ’s’ in \(\mathsf{s}\)-\(\mathsf{itadd}\) indicates that this is a simple version of iterated addition.

\[\left\langle\!\left\langle\mathsf{s}\text{-}\mathsf{itadd},\mathsf{bd}\mathsf{ s}\text{-}\mathsf{itadd}\right\rangle\!\right\rangle^{\alpha}=\sum_{i<\alpha(U)} \left\langle\!\left\langle Y,U\right\rangle\!\right\rangle^{\alpha}(i).\]

The proof of this lemma requires some preparation. Our first step will be to fix an encoding of sequences by natural numbers. We first encode a sequence \(\boldsymbol{i}=(i_{0},\ldots,i_{k-1})\in\mathbb{N}^{*}\) by the string

\[s_{1}(\boldsymbol{i})\coloneqq\#\operatorname{bin}(i_{1})\#\operatorname{ bin}(i_{2})\#\ldots\#\operatorname{bin}(i_{k})\]

over the alphabet \(\{0,1,\#\}\). Then we replace every \(0\) in \(s_{1}(\boldsymbol{i})\) by \(01\), every \(1\) by \(10\), and every \(\#\) by \(11\) to obtain a string \(s_{2}(\boldsymbol{i})=s_{\ell-1}\ldots s_{0}\) over the alphabet \(\{0,1\}\). We read \(s_{2}(\boldsymbol{i})\) as a binary number and let

\[\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{ \operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{ \operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname \,}}}}}}}}}}}}}}} \operatorname{\left\{\left\langle \mathtt{bin}^{-1}\left(s_{2}(i)\right)=\sum_{i=0}^{\ell-1}s_{i}2^{i} \text{if }\boldsymbol{i}\neq\varnothing,\right.\] \[\left.0\text{if }\boldsymbol{i}=\varnothing.\right.\]

**Example 3.11**.: Consider the sequence \(\boldsymbol{i}=(2,5,0)\). We have \(s_{1}(\boldsymbol{i})=\#10\#101\#0\) and \(s_{2}(\boldsymbol{i})=111001111001101101\). This yields

\[\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{ \operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{ \operatorname{\operatorname{\operatorname}}}}}}}}}}}}}} \operatorname{\left\langle\mathtt{i}\operatorname{\operatorname{\operatorname{ \operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\,{\,{\,{\,}}}}}}}}} }}}\operatorname{\left\langle\mathtt{i}\operatorname{\operatorname{ \operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\,{\,{\,{\,}}}}}}} }}}}\operatorname{\left\langle\mathtt{i}\operatorname{\operatorname{\operatorname{ \operatorname{\operatorname{\operatorname{\operatorname{\operatorname{\,{\operatorname{\,{\,{\,\,}}}}}} }}}}\operatorname{\left\langle\mathtt{i}\operatorname{\operatorname{\operatorname{ \operatorname{\operatorname{\operatorname{\operatorname{\,{\,{\,{\,{\,{\,}}}}}}}}}} }\operatorname{\left\langle\mathtt{i}\operatorname{\operatorname{\operatorname{ \operatorname{\operatorname{\operatorname{\,{\,{\,{\,{\,{\,{\,{\,{\,{\,\,{\,{\,{\,\,{\,\,{\,\,\,{\,\,\,\,}}}}}}}}}}}} }\operatorname{\left\langle\mathtt{i}\operatorname{\operatorname{\operatorname{ \operatorname{\operatorname{\operatorname{\,{\,{\,{\,{\,{\,{\,{\,{\,{\,\,{\,\,{\,\,\,{\,\
It is easy to see that the mapping \(\overset{r}{\iota}\colon\mathbb{N}^{*}\to\mathbb{N}\) is injective, but not bijective. Observe that for \(\boldsymbol{i}=(i_{0},\ldots,i_{k-1})\in\mathbb{N}^{*}\) we have

\[\text{bsize}\left(\overset{r}{\iota}\overset{\ast}{\iota}\right)=\sum_{i=0}^{k -1}2(\text{bsize}(i_{j})+1)\leq 4\sum_{i=0}^{k-1}\text{bsize}(i_{j}) \tag{3.D}\]

and thus

\[\overset{r}{\iota}\overset{\ast}{\iota}\overset{\ast}{\iota}\overset{\ast}{ \iota}\overset{\ast}{\iota}\overset{\ast}{\iota}<2^{4\sum_{i=0}^{k-1}\text{bsize }(i_{j})}. \tag{3.E}\]

**Lemma 3.12**.:
1. _There is an arithmetical_ \(\mathsf{FO}+\mathsf{C}\)_-formula_ \(\mathsf{seq}(y)\) _such that for all_ \(n\in\mathbb{N}\) _we have_ \[\llbracket\mathsf{seq}\rrbracket\,(n)=1\iff n=\overset{r}{\iota}\overset{\ast} {\iota}\overset{\ast}{\iota}\overset{\ast}{\iota}\] _for some_ \(\boldsymbol{i}\in\mathbb{N}\)_._
2. _There is an arithmetical_ \(\mathsf{FO}+\mathsf{C}\)_-term_ \(\mathsf{seqlen}(y)\) _such that for all_ \(n\in\mathbb{N}\) _we have_ \[\llbracket\mathsf{seqlen}\rrbracket\,(n)=\begin{cases}k&\text{if }n=\overset{r}{ \iota}(i_{0},\ldots,i_{k-1})\overset{\ast}{\iota}\text{ for some }k,i_{0},\ldots,i_{k-1}\in\mathbb{N},\\ 0&\text{otherwise}.\end{cases}\]
3. _There is an arithmetical_ \(\mathsf{FO}+\mathsf{C}\)_-term_ \(\mathsf{entry}(y,y^{\prime})\) _such that for all_ \(j,n\in\mathbb{N}\) _we have_ \[\llbracket\mathsf{entry}\rrbracket\,(j,n)=\begin{cases}i&\text{if }n=\overset{r}{ \iota}(i_{0},\ldots,i_{k-1})\overset{\ast}{\iota},j<k,i=i_{j}\text{ for some }k,i_{0},\ldots,i_{k-1}\in\mathbb{N},\\ n&\text{otherwise}.\end{cases}\]

Proof.: Let \(S=\{s_{2}(\boldsymbol{i})\mid\boldsymbol{i}\in\mathbb{N}^{*}\}\). Let \(\text{bin}(n)\eqeqcolon\boldsymbol{s}=s_{\ell-1}\ldots s_{0}\). We want to detect if \(\boldsymbol{s}=S\). As a special case, we note that if \(\ell=0\) and thus \(\boldsymbol{s}=\varnothing\), we have \(\boldsymbol{s}=s_{2}(\varnothing)\). In the following, we assume that \(\ell>0\). Then if \(\ell\) is odd, we have \(\boldsymbol{s}\not\in S\). Furthermore, if there is a \(p<\frac{\ell}{2}\) such that \(s_{2p}=0\) and \(s_{2p+1}=0\), again we have \(\boldsymbol{s}\not\in S\), because \(\boldsymbol{s}\) is not obtained from a string \(\boldsymbol{s}^{\prime}\in\{0,1,\#\}^{*}\) by replacing \(0\)s by \(01\), \(1\)s by \(10\), and \(\#\)s by \(11\). Otherwise, we let \(\boldsymbol{s}^{\prime}=s_{\frac{\ell}{2}-1}^{\prime}\ldots s_{0}^{\prime}\in \{0,1,\#\}^{*}\) be the corresponding string. Then \(\boldsymbol{s}^{\prime}=s_{1}(\boldsymbol{i})\) for some \(\boldsymbol{i}\in\mathbb{N}^{*}\) if and only if \(\boldsymbol{s}^{\prime}\) satisfies the following conditions:

* \(s_{0}^{\prime}\neq\#\) and \(s_{\ell/2-1}^{\prime}=\#\);
* for all \(p<\frac{\ell}{2}\), if \(s_{p}^{\prime}=\#\) and \(s_{p-1}^{\prime}=0\) then either \(p=1\) or \(s_{p-2}=\#\).

We can easily translate the conditions to conditions on the string \(\boldsymbol{s}\) and, using the bit predicate, to conditions on \(n\). As the bit predicate is definable in \(\mathsf{FO}+\mathsf{C}\) (by Lemma 3.7), we can express these conditions by an arithmetical \(\mathsf{FO}+\mathsf{C}\)-formula \(\mathsf{seq}(y)\).

To prove (2), we observe that if \(n=\overset{r}{\iota}\overset{\ast}{\iota}\overset{\ast}{\iota}\) for some \(\boldsymbol{i}\in\mathbb{N}^{*}\), then the length of the sequence \(\boldsymbol{i}\) is the number of \(\#\)s in the string \(s_{1}(\boldsymbol{i})\), or equivalently, the number of \(p<\frac{\ell}{2}\) such that \(\text{Bit}(2p,n)=1\) and \(\text{Bit}(2p+1,n)=1\) Using the formula \(\mathsf{seq}(y)\) and the bit predicate, we can easily express this by a term \(\mathsf{seqlen}(y)\).

To prove (3), we first write an arithmetical formula \(\mathsf{isEntry}(y,y^{\prime},y^{\prime\prime})\) such that

\[\llbracket\mathsf{isEntry}\rrbracket\,(j,n,i)=1\iff n=\overset{r}{\iota}(i_{0}, \ldots,i_{k-1})\overset{\ast}{\iota},j<k,i=i_{j}\text{ for some }k,i_{0},\ldots,i_{k-1}\in\mathbb{N}.\]
Once we have this formula, we let

\[\mathsf{entry}(y,y^{\prime})=\mathsf{min}\Big{(}y^{\prime},\#z<y^{\prime}.\exists y ^{\prime\prime}<y^{\prime}.\big{(}\mathsf{isEntry}(y,y^{\prime},y^{\prime\prime} )\wedge z<y^{\prime\prime}\big{)}\Big{)}.\]

To define \(\mathsf{isEntry}(y,y^{\prime},y^{\prime\prime})\), observe that for \(\mathbf{i}=(i_{0},\dots,i_{k-1})\in\mathbb{N}^{*}\), the \(j\)th entry \(i_{j}\) is located between the \(j\)th and \((j+1)\)st '\(\#\)' in the string \(s_{1}(\mathbf{i})\) and thus between the \(j\)th and \((j+1)\)st occurrence of '\(11\)' at positions \(2p,2+1\) in the string \(s_{2}(\mathbf{i})=\mathrm{bin}\big{(}\overset{\raisebox{-0.5pt}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7 }{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7 }{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7 }}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0.7}{\scalebox{0.7 }{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0. }}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0.7 }{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0. 7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0.7 }{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0.7}{\scalebox{0. 7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0. 7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\, \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0} \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0}}{\scalebox{0.7}{\scalebox{0.7 }{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0}}{\scalebox{0.7}{\, \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{\scalebox}}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0.7}{\scalebox{0}}\scalebox{0.7pt}{\scalebox{0.7}{\scalebox{0.7}{\, \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{\scalebox{0}}}{\scalebox{0.7}{\scalebox{ \scalebox{0}}{\scalebox{0.7}{\scalebox{0}}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{ \scalebox{0}}{\scalebox{0}}{\scalebox{0.7}{\scalebox{0}}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0}{\scalebox{0}}{\scalebox{0.7}{\scalebox{0}}{\scalebox{0.7}{\scalebox{0.7}{ \scalebox{0.7}{\scalebox{0}{\scalebox{\scalebox}}{0.7}{\scalebox{0.7}{\scalebox{0}{\scalebox{0}}{\scalebox{ \scalebox{0}}{\scalebox{0.7}{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0}}{\scalebox{0.7}{ \scalebox{0}{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0}}{\scalebox{0.7}{0.7}{\scalebox{0.7}{ \scalebox{0.7}{0.7pt}{\scalebox{0.7}{\scalebox{0.7}{0.7pt}{\scalebox{0.7}{\scalebox{0.7}{0.7pt}{\, \scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0.7}}{\scalebox{0.7}{\scalebox{0. }}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0.7}{\, \raisebox{0.7}{0.7pt{0.7}{\scalebox{\,}{0.7pt{\scalebox{\mul}}}{\scalebox{0.7}{\scalebox{0. \,\raisebox{\mul}{0.7}{0.7pt{\scalebox \mul}{0.7}{\scalebox{0}}{\scalebox{0.7}{\scalebox{0.7}{0.7}{\scalebox{ \mul}}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0}}{\scalebox{0.7}{\, \raisebox{0.7}{0.7}{\scalebox{0.7}{\scalebox{0.7}}{\scalebox{0.7}{0.7pt{0.7}{ \scalebox{0.7}}{\scalebox{0.7}{\scalebox{0.7}{\scalebox{0.7}{0.7}{\scalebox{0. \raisebox{\mul}}{0.7}{\scalebox{0.7}{\scalebox{0.7}{0.7}{\scalebox{0. \raisebox{\mul}}{0.7}{\scalebox{0.7}{0.7pt}{\scalebox{0. \raisebox{0}{0.7}{\raisebox{0}}{\scalebox{0.7}{\raisebox{0.7}{0.7}{\scalebox{0. \raisebox}{0.7}{\raisebox{0.7}{0.7pt}{\raisebox{0.7}{0.7pt{0.7}{ \raisebox{0.7}{0.7pt{0.7pt}{\raisebox{0.7}{0.7pt}{0.7pt}{\scalebox{0. \raisebox{\raisebox}{0.7}{0.7pt}{0.7pt{0.7pt}{0.7pt}{\scalebox{0. \raisebox{0.7}{0.7pt{0.7}{0.7pt}{\scalebox{0.0.7}{0.7pt}{\raisebox{0..7pt}{0.7pt{0.7pt}{\raisebox{0.0.7pt}{0.7pt}{\raisebox{0.0.7pt{0.7}{ \raisebox{0.0.7pt{\raisebox}{0.}}{\raisebox{0.0.7}{0.7pt{0.7pt}{\raisebox{0..0.7pt{0.7pt{\raisebox{\raisebox}}}{0.0.7pt{0.7pt}{\raisebox{0.0.7}{0.7pt{0..7pt}{\raisebox{0.0.7pt{0.7.0.7pt{\raisebox{\raisebox}{\raisebox}}{0.0.7}{\raisebox{0..0.7}{\raisebox{0.0.7}{\raisebox{0.0.7}{\raisebox{0.}{0.0.7pt{0.7}{\raisebox{0..0.7}{\raisebox{0.0.7}{\raisebox{0.}{0.0.7pt{0.7}{\raisebox{0.0.7}{\raisebox{0..0.7}{\raisebox{0.0.7}{\raisebox{0.}{0.0.7pt{0.7}{\raisebox{0..0.7}{\raisebox{0.0.7}{\raisebox{0.0.7}{0.0.7pt{0.0.7}{\
For \(j<\ell^{(0)}\), let \(n^{(0)}_{i,j}\coloneqq\operatorname{Bit}(j,n^{(0)}_{i})\) and

\[s^{(0)}_{j}\coloneqq\sum_{i=0}^{m^{(0)}-1}n^{(0)}_{i,j}=\big{|}\big{\{}i\; \big{|}\;n^{(0)}_{i,j}=1\big{\}}\big{|}. \tag{3.G}\]

Then

\[\sum_{i=0}^{m^{(0)}-1}n^{(0)}_{i}=\sum_{i=0}^{m^{(0)}-1}\sum_{j=0}^{\ell^{(0)}- 1}2^{j}n^{(0)}_{i,j}=\sum_{j=0}^{\ell^{(0)}-1}2^{j}\sum_{i=0}^{m^{(0)}-1}n^{(0) }_{i,j}=\sum_{j=0}^{\ell^{(0)}-1}2^{j}s^{(0)}_{j}.\]

Let \(m^{(1)}\coloneqq\ell^{(0)}\) and \(n^{(1)}_{i}=2^{i}s^{(0)}_{i}\) for \(i<\ell^{(0)}\). Then

\[\sum_{i=0}^{m-1}n_{i}=\sum_{i=0}^{m^{(0)}-1}n^{(0)}_{i}=\sum_{i=0}^{m^{(1)}-1} n^{(1)}_{i}.\]

Moreover,

\[n^{(1)}_{i}\leq 2^{\ell^{(0)}-1}s^{(0)}_{i}=2^{\ell^{(0)}-1+\log s^{(0)}_{i} }<2^{\ell^{(0)}+\big{|}\log s^{(0)}_{i}\big{|}}.\]

Let \(p^{(1)}\coloneqq\big{|}\log m^{(0)}\big{|}\) and \(\ell^{(1)}\coloneqq\ell^{(0)}+p^{(1)}\). Noting that \(s^{(0)}_{i}\leq m^{(0)}\) for all \(i\), we thus have

\[n^{(1)}_{i}<2^{\ell^{(1)}}\]

for all \(i\). Finally, note that

\[\operatorname{Bit}(j,n^{(1)}_{i})\neq 0\implies i\leq j\leq i+p^{(1)}\]

This completes the base step of the construction. For the inductive step, suppose that we have defined \(m^{(k)},\ell^{(k)},p^{(k)}\in\mathbb{N}_{>0}\) and \(n^{(k)}_{i}\in\mathbb{N}\) for \(i\in\{0,\dots,m^{(k)}-1\}\) such that

\[\sum_{i=0}^{m-1}n_{i}=\sum_{i=0}^{m^{(k)}-1}n^{(k)}_{i}\]

and for all \(i\):

* \(n^{(k)}_{i}<2^{\ell^{(k)}}\) for all \(i\);
* \(\operatorname{Bit}(j,n^{(k)}_{i})\neq 0\implies i\leq j\leq i+p^{(k)}\).

For \(j<\ell^{(k)}\), let \(n^{(k)}_{i,j}\coloneqq\operatorname{Bit}(j,n^{(k)}_{i})\) and

\[s^{(k)}_{j}\coloneqq\sum_{i=0}^{m^{(k)}-1}n^{(k)}_{i,j}=\sum_{i=\max\{0,j-p^{ (k)}\}}^{j}n^{(k)}_{i,j}.\]

Then

\[\sum_{i=0}^{m^{(k)}-1}n^{(k)}_{i}=\sum_{j=0}^{\ell^{(k)}-1}2^{j}s^{(k)}_{j}.\]
Let \(m^{(k+1)}\coloneqq\ell^{(k)}\) and \(n_{i}^{(k+1)}\coloneqq 2^{i}s_{i}^{(k)}\) for \(i<\ell^{(k)}\). Then

\[\sum_{i=0}^{m-1}n_{i}=\sum_{i=0}^{m^{(k)}-1}n_{i}^{(k)}=\sum_{i=0}^{m^{(k+1)}-1}n _{i}^{(k+1)}.\]

Moreover,

\[n_{i}^{(k+1)}\leq 2^{\ell^{(k)}-1}s_{i}^{(k)}=2^{\ell^{(k)}-1+\log s_{i}^{(k)}}< 2^{\ell^{(k)}+\left\lfloor\log s_{i}^{(k)}\right\rfloor}.\]

Note that \(s_{i}^{(k)}\leq p^{(k)}+1\). Let \(p^{(k+1)}\coloneqq\left\lfloor\log(p^{(k)}+1)\right\rfloor\) and \(\ell^{(k+1)}\coloneqq\ell^{(k)}+p^{(k+1)}\). Then

\[n_{i}^{(k+1)}<2^{\ell^{(k+1)}}\]

and for all \(j\),

\[\operatorname{Bit}(j,n_{i}^{(k+1)})\neq 0\implies i\leq j\leq i+p^{(k+1)}.\]

Observe that if \(p^{(k)}=1\) then \(p^{(k^{\prime})}=1\) for all \(k^{\prime}>k\). Let \(k^{*}\) be the least \(k\) such that \(p^{(k)}=1\). It is easy to see that

\[\begin{array}{rccc}&p^{(k^{*}-1)}&=&2,\\ 3&\leq&p^{(k^{*}-2)}&\leq&6,\\ 7&\leq&p^{(k^{*}-3)}&\leq&126,\\ 127&\leq&p^{(k)}&&\text{for $k<k^{*}-3$}.\end{array} \tag{3.H}\]

_Claim 1._

* \(\sum_{i=2}^{k^{*}}ip^{(i)}\leq 5\log\log m\);
* \(\sum_{i=1}^{k^{*}}\operatorname{bsize}(p^{(i)})\leq 5\log\log m\).

Proof.: By induction on \(k=k^{*},k^{*}-1,\ldots,2\) we prove

\[\sum_{i=k}^{k^{*}}i\cdot p^{(i)}\leq 2kp^{(k)} \tag{3.I}\]

As base case, we need to check (3.I) for \(k\in\{k^{*},k^{*}-1,k^{*}-2,k^{*}-3\}\). Using (3.H), this is straightforward. For example, if \(k=k^{*}-2\) and \(p^{(k)}=3\) we have

\[\sum_{i=k}^{k^{*}}ip^{(i)}=3(k^{*}-2)+2(k^{*}-1)+k^{*}=6k^{*}-8\leq 6k^{*}-4=2kp ^{(k)}.\]

Or if \(k=k^{*}-3\) and \(p^{(k)}=15\) we have \(p^{(k+1)}=\left\lfloor\log(p^{(k)}+1)\right\rfloor=4\) and thus

\[\sum_{i=k}^{k^{*}}ip^{(i)}=15(k^{*}-3)+4(k^{*}-2)+2(k^{*}-1)+k^{*}=22k^{*}-55 \leq 30k^{*}-45=2kp^{(k)}.\]
The inequality \(22k^{*}-55\leq 30k^{*}-45\) holds because if \(2\leq k\leq k^{*}-3\) then \(k^{*}\geq 5\), which implies \(8k^{*}\geq 10\).

For the inductive step \(k+1\mapsto k\), where \(2\leq k<k^{*}-3\), we argue as follows:

\[\sum_{i=k}^{k^{*}}i\cdot p^{(i)} =k\cdot p^{(k)}+\sum_{i=k+1}^{k^{*}}i\cdot p^{(i)}\] \[\leq k\cdot p^{(k)}+2(k+1)p^{(k+1)}\] induction hypothesis \[\leq k\cdot p^{(k)}+2(k+1)\log(p^{(k)}+1)\] \[\leq k\cdot p^{(k)}+3k\log(p^{(k)}+1)\] because

\[k\geq 2\]

Since by (3.H) we have \(p^{(k)}\geq 127\) for \(k<k^{*}-3\), we have \(p^{(k)}\geq 3\log(p^{(k)}+1)\). Inequality (3.I) follows.

For \(k=2\), this yields

\[\sum_{i=2}^{k^{*}}i\cdot p^{(i)}\leq 4p^{(2)}\leq 4\log(p^{(1)}+1)\leq 4\log( \log m+1)\leq 5\log\log m.\]

To prove (2), note that \(\text{bsize}(p^{(k^{*})})=\text{bsize}(1)=1\) and \(\text{bsize}(p^{(k)})=\left\lceil\log(p^{(k)}+1)\right\rceil\leq p^{(k+1)}+1\) for \(k<k^{*}\). Thus (2) holds if \(k^{*}=1\). If \(k^{*}\geq 2\) we have

\[\sum_{i=1}^{k^{*}}\text{bsize}(p^{(i)})=1+\sum_{i=2}^{k^{*}}(p^{(i)}+1)=k^{*}+ \sum_{i=2}^{k^{*}}p^{(i)}\leq k^{*}+1+\frac{1}{2}\sum_{i=2}^{k^{*}-1}ip^{(i)} \leq\sum_{i=2}^{k^{*}}ip^{(i)},\]

and (2) follows from (1).

_Claim 2._ There is an arithmetical \(\mathsf{FO}+\mathsf{C}\)-term \(\mathsf{pseq}(y)\) such that

\[\left\lceil\mathsf{pseq}\right\rceil^{\alpha}(m)={}^{r}_{\alpha}(p^{(1)}, \ldots,p^{(k^{*})}{}_{\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!
defines \(\sideset{{}^{\prime}_{i}(p^{(1)},\ldots,p^{(k^{*})})}{}_{\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!
To compute \(s_{j}^{(k^{*})}\), we need to know \(s_{j}^{(k^{*}-1)},\ldots,s_{j-p^{(k^{*})}}^{(k^{*}-1)}\). To compute these numbers, we need to know \(s_{j}^{(k^{*}-2)},\ldots,s_{j-p^{(k^{*}-2)}}^{(k^{*}-2)}\), et cetera. Thus if we want to compute \(s_{j}^{(k^{*})}\) starting from values \(s_{j^{\prime}}^{(k)}\), we need to know \(s_{j^{\prime}}^{(k)}\) for

\[j-\sum_{i=k+1}^{k^{*}}p^{(i)}\leq j^{\prime}\leq j.\]

For \(2\leq k<k^{*}\), we let

\[\boldsymbol{s}_{jk}\coloneqq(s_{j-\sum_{i=k+1}^{k^{*}}p^{(i)}}^{(k)},\ldots,s _{j}^{(k)}),\]

and we let \(\boldsymbol{s}_{jk^{*}}\coloneqq(s_{j}^{(k^{*})})\). Concatenating these sequences, we let

\[\boldsymbol{s}_{j}\coloneqq\boldsymbol{s}_{j2}\boldsymbol{s}_{j3}\ldots \boldsymbol{s}_{jk^{*}}.\]

_Claim 5._ There is an arithmetical \(\mathsf{FO}+\mathsf{C}\)-formula \(\mathsf{all}\mathsf{-s}(y,z)\) such that for all \(j\in\{0,\ldots,\ell^{(k^{*})}-1\}\) and \(t\in\mathbb{N}\) we have

\[\llbracket\mathsf{all}\mathsf{-s}\rrbracket^{a}\,(j,t)=1\iff t={}_{\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!
again assuming that \(m\) is sufficiently large. This proves (3.K).

We let

\[\mathsf{skstar}(y):=\exists z<U.\Big{(}\mathsf{all\text{-}s}(y,z)\wedge y= \mathsf{entry}\big{(}\mathsf{seqlen}(z)-1,z\big{)}\Big{)},\]

where the formula \(\mathsf{all\text{-}s}(y,z)\) is from Claim 5 and the terms \(\mathsf{entry}\) and \(\mathsf{seqlen}\) are from Lemma 3.12.

It remains to compute \(\sum_{i=0}^{m^{(k^{*})}-1}n_{i}^{(k^{*}+1)}\), where \(n_{i}^{(k^{*}+1)}=2^{i}s_{i}^{(k^{*})}\). Since \(s_{j}^{(k^{*})}\leq 2\), we have \(\operatorname{Bit}(j,n_{i}^{(k^{*})})=0\) unless \(j\in\{i-1,i\}\). We split the sum into the even and the odd entries:

\[\sum_{i=0}^{m^{(k^{*})}-1}n_{i}^{(k^{*}+1)}=\underbrace{\Big{[}m^{(k^{*})/2} \Big{]}^{-1}n_{2i}^{(k^{*})}}_{=n_{1}^{*}}+\underbrace{\Big{[}m^{(k^{*})}/2 \Big{]}^{-1}n_{2i+1}^{(k^{*})}}_{=n_{2}^{*}}.\]

Since the entries in the two partial sums have no non-zero bits in common, it is easy to define the two partial sums (of course, using Claim 6). Then we can apply Lemma 3.9 to define \(n_{1}^{*}+n_{2}^{*}\). 

**Lemma 3.14**.: _Let \(Y_{1},Y_{2}\) be relation variables of type \(\{\mathsf{n}\}\), and let \(U_{1},U_{2}\) be function variables of type \(\varnothing\to\mathsf{n}\). Then there are arithmetical \(\mathsf{FO+C}\)-formulas \(\mathsf{mul}\), \(\mathsf{div}\), and \(\mathsf{FO+C}\)-terms \(\mathsf{bd\text{-}mul}\), \(\mathsf{bd\text{-}div}\) such that for all numerical assignments \(\omega\),_

\[\langle\!\langle\mathsf{mul},\mathsf{bd\text{-}mul}\rangle\! \rangle^{\omega} =\langle\!\langle Y_{1},U_{1}\rangle\!\rangle^{\omega}\cdot\langle\! \langle Y_{2},U_{2}\rangle\!\rangle^{\omega}\,,\] \[\langle\!\langle\mathsf{div},\mathsf{bd\text{-}div}\rangle\! \rangle^{\omega} =\left|\frac{\langle\!\langle Y_{1},U_{1}\rangle\!\rangle^{\omega} }{\langle\!\langle Y_{2},U_{2}\rangle\!\rangle^{\omega}}\right|\] if \[\langle\!\langle Y_{2},U_{2}\rangle\!\rangle^{\omega}\neq 0.\]

Proof.: The expressibility multiplication follows easily from the expressibility of iterated addition (Lemma 3.10). Division is significantly more difficult, we refer the reader to [15]. 

We need be an extension of Lemma 3.10 where the family of numbers is no longer indexed by numbers, but by arbitrary tuples of vertices and numbers, and where the bounds on the bitsite of the numbers in the family are not uniform (in Lemma 3.10 we use the \(0\)-ary function variable \(U\) to provide a bound on the bitsize of all numbers in our family). Before presenting this extension, we consider a version of iterated addition as well as taking the maximum and minimum of a family of numbers given directly as values of a function instead of the binary representation that we considered in the previous lemmas.

**Lemma 3.15**.: _Let \(X\) be a relation variable of type \(\{\mathsf{v}^{k}\mathsf{n}^{\ell}\}\), and let \(U,V\) be a function variables of types \(\mathsf{v}^{k}\mathsf{n}^{\ell}\to\mathsf{n}\), \(\mathsf{v}^{k}\to\mathsf{n}\), respectively._

1. _There is a_ \(\mathsf{FO+C}\)_-term_ \(\mathsf{u}\mathsf{\text{-}tadd}^{\ell}\) _such that for all structures_ \(A\) _and assignments_ \(\omega\) _over_ \(A\)_,_ \[\llbracket\mathsf{u}\mathsf{\text{-}tadd}\rrbracket^{(A,a)}=\sum_{(\mathbf{a},\mathbf{b })}\omega(U)(\mathbf{a},\mathbf{b}),\]
_where the sum ranges over all \((\mathbf{a},\mathbf{b})\in\omega(X)\) such that \(\mathbf{b}=(b_{1},\ldots,b_{\ell})\in\mathbb{N}^{\ell}\) with \(b_{i}<\omega(V)(\mathbf{a})\) for all \(i\in[\ell]\)._
2. _There are_ \(\mathsf{FO}+\mathsf{C}\)_-terms_ \(\mathsf{u}\mathsf{-max}\) _and_ \(\mathsf{u}\mathsf{-min}\) _such that for all structures_ \(A\) _and assignments_ \(\omega\) _over_ \(A\)_,_ \[\llbracket\mathsf{u}\mathsf{-max}\rrbracket^{(A,a)} =\max_{(\mathbf{a},\mathbf{b})}\omega(U)(\mathbf{a},\mathbf{b}),\] \[\llbracket\mathsf{u}\mathsf{-min}\rrbracket^{(A,a)} =\min_{(\mathbf{a},\mathbf{b})}\omega(U)(\mathbf{a},\mathbf{b}),\] _where_ \(\max\) _and_ \(\min\) _range over all_ \((\mathbf{a},\mathbf{b})\in\omega(X)\) _such that_ \(\mathbf{b}=(b_{1},\ldots,b_{\ell})\in\mathbb{N}^{\ell}\) _with_ \(b_{i}<\omega(V)(\mathbf{a})\) _for all_ \(i\in[\ell]\)_._

_Furthermore, if \(k=0\) then the terms \(\mathsf{u}\mathsf{-add}\), \(\mathsf{u}\mathsf{-max}\), and \(\mathsf{u}\mathsf{-min}\) are arithmetical._

Proof.: To simplify the notation, in the proof we assume \(\ell=1\). The generalisation to arbitrary \(\ell\) is straightforward.

The trick is to use adaptive bounds in our counting terms. Let \(A\) be a structure and \(\omega\) an assignment over \(A\). Let \(\mathscr{I}\subseteq V(A)^{k}\times\mathbb{N}\) be the set of all all \((\mathbf{a},b)\in\omega(X)\) with \(b<\omega(V)(\mathbf{a})\). We exploit that for all \((\mathbf{a},b)\in\mathscr{I}\), \(\omega(U)(\mathbf{a},b)\) is the number of \(c\in\mathbb{N}\) such that \(c<\omega(U)(\mathbf{a},b)\). This implies

\[\sum_{(\mathbf{a},b)}\omega(U)(\mathbf{a},b) =\Big{|}\Big{\{}(\mathbf{a},b,c)\Bigm{|}(\mathbf{a},b)\in\mathscr{I} \text{ and }c<\omega(U)(\mathbf{a},b)\Big{\}}\Big{|}\] \[=\Big{|}\Big{\{}(\mathbf{a},b,c)\Bigm{|}(\mathbf{a},b)\in\omega(X)\text{ with }b<\omega(V)(\mathbf{a})\text{ and }c<\omega(U)(\mathbf{a},b)\Big{\}}\Big{|}.\]

Thus

\[\mathsf{u}\mathsf{-itald}\coloneqq\#\big{(}\mathbf{x},y<V(\mathbf{x}),z<U(\mathbf{x},y) \big{)}.X(\mathbf{x},y).\]

satisfies assertion (1).

Once we have this, assertion (2) is easy because maximum and minimum are bounded from above by the sum. For example, we let

\[\mathsf{u}\mathsf{-max}\coloneqq\#z<\mathsf{u}\mathsf{-itald}.\exists\mathbf{x}. \exists y<V(\mathbf{x}).\big{(}X(\mathbf{x})\wedge z<U(\mathbf{x},y)\big{)}.\qed\]

The following lemma is the desired generalisation of Lemma 3.10.

**Lemma 3.16**.: _Let \(X,Y\) be relation variables of type \(\{\mathsf{v}^{k}\mathtt{n}^{\ell}\}\) and \(\{\mathsf{nv}^{k}\mathtt{n}^{\ell}\}\), respectively, and let \(U,V\) be function variables of type \(\mathsf{v}^{k}\mathtt{n}^{\ell}\to\mathtt{n}\) and \(\mathsf{v}^{k}\to\mathtt{n}\), respectively. Then there is an \(\mathsf{FO}+\mathsf{C}\)-formula \(\mathsf{itadd}\) and an \(\mathsf{FO}+\mathsf{C}\)-term \(\mathsf{bd}\mathsf{-itald}\) such that for all structures \(A\) and assignments \(\omega\) over \(A\),_

\[\langle\!\langle\mathsf{itadd},\mathsf{bd}\mathsf{-itald}\rangle\!\rangle^{(A, a)}=\sum_{(\mathbf{a},\mathbf{b})}\langle\!\langle Y,U\rangle\!\rangle^{(A,a)}\,(\mathbf{a},\mathbf{b}),\]

_where the sum ranges over all \((\mathbf{a},\mathbf{b})\in\omega(X)\) such that \(\mathbf{b}=(b_{1},\ldots,b_{\ell})\in\mathbb{N}^{\ell}\) with \(b_{i}<\omega(V)(\mathbf{a})\) for all \(i\in[\ell]\)._

_If \(k=0\), then the formula \(\mathsf{itadd}\) and the term \(\mathsf{bd}\mathsf{-itald}\) are arithmetical._
Proof.: Let \(A\) be a structure and \(\omega\) an assignment over \(A\). Let \(\mathscr{I}\subseteq V(A)^{k}\times\mathbb{N}^{\ell}\) be the set of all all \((\mathbf{a},\mathbf{b})\in\omega(X)\) such that \(\mathbf{b}=(b_{1},\ldots,b_{\ell})\) with \(b_{i}<\omega(V)\mathbf{(a)}\). Let \(m\coloneqq|\mathscr{I}|\) and \(p\coloneqq\max\big{\{}\omega(U)\mathbf{(a,b)}\bigm{|}(\mathbf{a},\mathbf{b})\in\mathscr{I} \big{\}}\). Note that \(p=[\mathfrak{u}\text{-}\mathsf{max}]^{(A,e)}\) for the term \(\mathfrak{u}\text{-}\mathsf{max}\) of Lemma 3.15. We have to add the family of \(m\) numbers \(n_{\mathbf{c}}\coloneqq\langle\!\langle Y,U\rangle\!\rangle^{(A,e)}\,\mathbf{(c)}\) for \(\mathbf{c}\in\mathscr{I}\). We think of these numbers as \(p\)-bit numbers, padding them with zeroes if necessary. Note that we cannot directly apply Lemma 3.10 to add these numbers, because the family, being indexed by vertices, is not ordered, and Lemma 3.10 only applies to ordered families indexed by numbers. But there is a simple trick to circumvent this difficulty. (We applied the same trick in the proof of Lemma 3.10).

For all \(i<p\), we let \(n_{\mathbf{c},i}\coloneqq\operatorname{Bit}(i,n_{\mathbf{c}})\) be the \(i\)th bit of \(n_{\mathbf{c}}\), and we let

\[s_{i}\coloneqq\sum_{\mathbf{c}\in\mathscr{I}}n_{\mathbf{c},i}.\]

We have

\[\sum_{\mathbf{c}\in\mathscr{I}}n_{\mathbf{c}}=\sum_{\mathbf{c}\in\mathscr{I}}\sum_{i=0}^{ p-1}n_{\mathbf{c},i}\cdot 2^{i}=\sum_{i=0}^{p-1}s_{i}\cdot 2^{i}.\]

This reduces the problem of adding the _unordered_ family of \(m\)\(p\)-bit numbers \(n_{\mathbf{c}}\) to adding the _ordered_ family of \(p\)\(p+\log m\)-bit numbers \(n^{\prime}_{i}\coloneqq s_{i}\cdot 2^{i}\).

The partial sums \(s_{i}\) are definable by a counting term in \(\mathsf{FO}+\mathsf{C}\), because \(s_{i}\) is the number of \(\mathbf{c}\in\mathscr{I}\) such that \(n_{\mathbf{c},i}=1\). As the bit predicate is definable in \(\mathsf{FO}+\mathsf{C}\), we can obtain the bit-representation of these numbers and then shift \(s_{i}\) by \(i\) to obtain the bit representation of \(n^{\prime}_{i}=s_{i}\cdot 2^{i}\). Then we can apply Lemma 3.10 to compute the sum. 

**Lemma 3.17**.: _Let \(X,Y\) be relation variables of type \(\{\mathsf{v}^{k}\mathsf{n}^{\ell}\}\) and \(\{\mathsf{nv}^{k}\mathsf{n}^{\ell}\}\), respectively, and let \(U,V\) be function variables of type \(\mathsf{v}^{k}\mathsf{n}^{\ell}\to\mathfrak{n}\) and \(\mathsf{v}^{k}\to\mathfrak{n}\), respectively. Then there are \(\mathsf{FO}+\mathsf{C}\)-formulas \(\mathsf{itmax}\), \(\mathsf{itmin}\) and \(\mathsf{FO}+\mathsf{C}\)-terms \(\mathsf{bd}\text{-}\mathsf{itmax}\), \(\mathsf{bd}\text{-}\mathsf{itmin}\) such that for all structures \(A\) and assignments \(\omega\) over \(A\),_

\[\langle\!\langle\mathsf{itmax},\mathsf{bd}\text{-}\mathsf{itmax} \rangle\!\rangle^{(A,e)} =\max_{(\mathbf{a},\mathbf{b})}\langle\!\langle Y,U\rangle\!\rangle^{(A,e)} \,(\mathbf{a},\mathbf{b}),\] \[\langle\!\langle\mathsf{itmin},\mathsf{bd}\text{-}\mathsf{itmin} \rangle\!\rangle^{(A,e)} =\min_{(\mathbf{a},\mathbf{b})}\langle\!\langle Y,U\rangle\!\rangle^{(A,e)} \,(\mathbf{a},\mathbf{b}),\]

_where max and min range over all \((\mathbf{a},\mathbf{b})\in\omega(X)\) such that \(\mathbf{b}=(b_{1},\ldots,b_{\ell})\in\mathbb{N}^{\ell}\) with \(b_{i}<\omega(V)\mathbf{(a)}\) for all \(i\in[\ell]\)._

_If \(k=0\), then the formula \(\mathsf{itmax}\), \(\mathsf{itmin}\) and the term \(\mathsf{bd}\text{-}\mathsf{itmin}\), \(\mathsf{bd}\text{-}\mathsf{itmin}\) are arithmetical._

Proof.: Again, to reduce the notational overhead we assume \(\ell=1\). We only give the proof for the maximum; the proof for the minimum is completely analogous.

The following formula says that \((\mathbf{x},y)\) is the index of the maximum number in the family:

\[\mathsf{maxind}(\mathbf{x},y)\coloneqq X(\mathbf{x},y)\wedge y<V(\mathbf{x})\wedge\forall \mathbf{x}^{\prime}\forall y^{\prime}<V(\mathbf{x}^{\prime}).\big{(}X(\mathbf{x}^{\prime},y) \to\mathsf{leq}^{\prime}(\mathbf{x}^{\prime},y^{\prime},\mathbf{x},y)\big{)},\]
where \(\mathsf{leq}^{\prime}(\mathbf{x}^{\prime},y^{\prime},\mathbf{x},\mathbf{y})\) is the formula obtained from the formula \(\mathsf{leq}\) of Lemma 3.9(2) by substituting \(Y_{1}(z)\) with \(Y(z,\mathbf{x}^{\prime},y^{\prime})\), \(U_{1}()\) with \(U(\mathbf{x}^{\prime},y^{\prime})\), \(Y_{2}(z)\) with \(Y(z,\mathbf{x},y)\), \(U_{2}()\) with \(U(\mathbf{x},y)\).

Then we let

\[\mathsf{itmax}(\widehat{y}) \coloneqq\exists\mathbf{x}\exists y<V(\mathbf{x}).\big{(}\mathsf{maxind}( \mathbf{x},y)\wedge Y(\widehat{y},\mathbf{x},y)\big{)},\] \[\mathsf{bd}\mbox{-}\mathsf{itmax} \coloneqq\#z<\mathsf{u}\mbox{-}\mathsf{max}.\forall\mathbf{x}\forall y <V(\mathbf{x})\big{(}\mathsf{maxind}(\mathbf{x},y)\to z<U(\mathbf{x},y)\big{)},\]

where \(\mathsf{u}\mbox{-}\mathsf{max}\) is the formula of Lemma 3.15. 

### Rational Arithmetic

We need to lift the results of the previous section to arithmetic on rational numbers. However, we will run into a problem with iterated addition, because the denominator of the sum can get too large. To avoid this problem, we will work with arithmetic on dyadic rationals. Then we have a problem with division, because the dyadic rationals are not closed under division, but division is not as important for us as iterated addition.

Our representation system for dyadic rationals by relation and function variables, or by formulas and terms, is based on a representations of dyadic rationals by tuples \((r,I,s,t)\in\{0,1\}\times 2^{\mathbb{N}}\times\mathbb{N}\times\mathbb{N}\): such a tuple represents the number

\[\langle\!\langle r,I,s,t\rangle\!\rangle\coloneqq(-1)^{r}\cdot 2^{-s}\cdot \sum_{i\in I,i<t}2^{i}.\]

This representation is not unique: there are distinct \((r,I,s,t)\) and \((r^{\prime},I^{\prime},s^{\prime},t^{\prime})\) such that \(\langle\!\langle r,I,s,t\rangle\!\rangle=\langle\!\langle r^{\prime},I^{ \prime},s^{\prime},t^{\prime}\rangle\!\rangle\). For example, \(\langle\!\langle r,I,s,t\rangle\!\rangle=\langle\!\langle r,I^{\prime},s+1,t+1\rangle\!\rangle\), where \(I^{\prime}=\{i+1\mid i\in I,i<t\}\). However, each dyadic rational \(q\) has a unique representation \(\operatorname{crep}(q)=(r,I,s,t)\) satisfying the following conditions:

1. \(i<t\) for all \(i\in I\);
2. \(s=0\) or \(0\in I\) (that is, the fraction \(\frac{\sum_{i\in I,i\in T}2^{i}}{2^{s}}\) is reduced);
3. if \(I=\varnothing\) (and hence \(\langle\!\langle s,I,s,t\rangle\!\rangle=0\)) then \(r=s=t=0\).

We call \(\operatorname{crep}(q)\) the _canonical representation_ of \(q\).

To represent a dyadic rational in our logical framework, we thus need four variables. As this tends to get a bit unwieldy, we introduce some shortcuts. An _r-schema_ of type \(\mathbf{t}\to\mathbf{r}\) for some \(\mathbf{t}\in\{\mathsf{v},\mathsf{n}\}^{k}\) is a tuple \(\mathbf{Z}=(Z_{\mathrm{sg}},Z_{\mathrm{Ind}},Z_{\mathrm{dn}},Z_{\mathrm{bd}})\), where \(Z_{\mathrm{sg}}\) is a relation variable of type \(\{\mathbf{t}\}\), \(Z_{\mathrm{Ind}}\) is a relation variable of type \(\{\mathbf{n}\mathbf{t}\}\), and \(Z_{\mathrm{dn}},Z_{\mathrm{bd}}\) are function variables of type \(\mathbf{t}\to\mathsf{n}\). For a structure \(A\), an interpretation \(\omega\) over \(A\), and a tuple \(\mathbf{c}\in A^{\mathbf{t}}\) we let

\[\langle\!\langle\mathbf{Z}\rangle\!\rangle^{(A,a)}\,(\mathbf{c})\coloneqq(-1)^{r}\cdot 2 ^{-\omega(Z_{\mathrm{dn}})(\mathbf{c})}\cdot\sum_{\begin{subarray}{c}(i,\mathbf{c}) \in\omega(Z_{\mathrm{Ind}}),\\ i<a(Z_{\mathrm{bd}})(\mathbf{c})\end{subarray}}2^{i}, \tag{3.1}\]

where \(r=1\) if \(\mathbf{c}\in\omega(Z_{\mathrm{sg}})\) and \(r=0\) otherwise. Note that with \(I=\{i\in\mathbb{N}\mid(i,\mathbf{c})\in\omega(Z_{\mathrm{Ind}})\}\), \(s=\omega(Z_{\mathrm{dn}})(\mathbf{c})\), and \(t=\omega(Z_{\mathrm{bd}})(\mathbf{c})\) we have \(\langle\!\langle\mathbf{Z}\rangle\!\rangle^{(A,a)}\,(\mathbf{c})=\langle\!\langle r,I,s, t\rangle\!\rangle\).
An _r-expression_ is a tuple \(\mathbf{\rho}(\mathbf{z})=\left(\mathbf{\rho}_{\mathrm{sg}}(\mathbf{z}),\mathbf{\rho}_{\mathrm{Ind}}( \widehat{y},\mathbf{z}),\mathbf{\rho}_{\mathrm{dn}}(\mathbf{z}),\mathbf{\rho}_{\mathrm{bd}}(\mathbf{ z})\right)\) where \(\mathbf{z}\) is a tuple of individual variables, \(\mathbf{\rho}_{\mathrm{sg}}(\mathbf{z}),\mathbf{\rho}_{\mathrm{Ind}}(\widehat{y},\mathbf{z})\) are \(\mathsf{FO}+\mathsf{C}\)-formulas, and \(\mathbf{\rho}_{\mathrm{dn}}(\mathbf{z}),\mathbf{\rho}_{\mathrm{bd}}(\mathbf{z})\) are \(\mathsf{FO}+\mathsf{C}\)-terms. For a structure \(A\), an interpretation \(\mathpzc{\omega}\) over \(A\), and a tuple \(\mathbf{c}\in A^{\mathrm{tp}(\mathbf{z})}\) we let

\[\left\langle\!\left\langle\mathbf{\rho}\right\rangle\!\right\rangle^{(A,\mathpzc {\omega})}(\mathbf{c})\coloneqq\left\langle\!\left\langle r,I,s,t\right\rangle\! \right\rangle, \tag{3.M}\]

where \(r=1\) if \((A,\mathpzc{\omega})\coloneqq\mathbf{\rho}_{\mathrm{sg}}(\mathbf{c})\) and \(r=0\) otherwise, \(I\) is the set of all \(i\in\mathbb{N}\) such that \(A\coloneqq\mathbf{\rho}_{\mathrm{Ind}}(i,\mathbf{c})\), \(s=\left[\mathbf{\rho}_{\mathrm{dn}}\right]^{(A,\mathpzc{\omega})}(\mathbf{c})\), and \(t=\left[\mathbf{\rho}_{\mathrm{bd}}\right]^{(A,\mathpzc{\omega})}(\mathbf{c})\). We sometimes say that \(\mathbf{\rho}\)_defines_ the representation \((r,I,s,t)\) of the dyadic rational \(\left\langle\!\left\langle r,I,s,t\right\rangle\!\right\rangle\)_in_\((A,\mathpzc{\omega})\).

For a fragment \(\mathsf{L}\) of \(\mathsf{FO}+\mathsf{C}\), such as those introduced in Section 3.7, an _r-expression in \(\mathsf{L}\)_ is an r-expression consisting of formulas and terms from \(\mathsf{L}\). An _arithmetical r-expression_ is an r-expression consisting of arithmetical formulas and terms.

For an r-schema \(\mathbf{Z}\) of type \(\mathsf{n}^{k}\to\mathtt{r}\), for some \(k\geq 0\), and a numerical assignment \(\mathpzc{\omega}\) we may just write \(\left\langle\!\left\langle\mathbf{Z}\right\rangle\!\right\rangle^{\mathpzc{\omega}} (\mathbf{c})\) without referring to a structure. Similarly, for an arithmetical r-expression \(\mathbf{\rho}(\mathbf{z})\) we may write \(\left\langle\!\left\langle\mathbf{\rho}\right\rangle\!\right\rangle^{\mathpzc{ \omega}}(\mathbf{c})\). We use a similar notation for other objects, in particular the L,F-schemas and L,F-expressions that will be introduced in Section 3.6.

**Lemma 3.18**.: _Let \(\mathbf{Z}\) be an r-schema of type \(\varnothing\to\mathtt{r}\). Then there is an arithmetical r-expression \(\mathsf{crep}\) such that for all structures \(A\) and assignments \(\mathpzc{\omega}\) over \(A\), \(\mathsf{crep}\) defines the canonical representation of \(\left\langle\!\left\langle\mathbf{Z}\right\rangle\!\right\rangle^{(A,\mathpzc{ \omega})}\) in \((A,\mathpzc{\omega})\)._

Proof.: Straightforward. 

Using this lemma, in the following we can always assume that the formulas and terms defining arithmetical operations, as for example, in Lemma 3.19, 3.20, et cetera, return their results in canonical representation.

**Lemma 3.19**.: _Let \(\mathbf{Z}_{1},\mathbf{Z}_{2}\) be r-schemas of type \(\varnothing\to\mathtt{r}\)._

1. _There are arithmetical r-expressions_ add_,_ sub_, and_ mul _such that for all structures_ \(A\) _and assignments_ \(\mathpzc{\omega}\) _over_ \(A\)_,_ \[\left\langle\!\left\langle\mathsf{add}\right\rangle\!\right\rangle^{(A, \mathpzc{\omega})} =\left\langle\!\left\langle\mathbf{Z}_{1}\right\rangle\!\right\rangle^{(A, \mathpzc{\omega})}+\left\langle\!\left\langle\mathbf{Z}_{2}\right\rangle\!\right\rangle ^{(A,\mathpzc{\omega})},\] \[\left\langle\!\left\langle\mathsf{sub}\right\rangle\!\right\rangle^{(A, \mathpzc{\omega})} =\left\langle\!\left\langle\mathbf{Z}_{1}\right\rangle\!\right\rangle^{(A, \mathpzc{\omega})}-\left\langle\!\left\langle\mathbf{Z}_{2}\right\rangle\!\right\rangle ^{(A,\mathpzc{\omega})},\] \[\left\langle\!\left\langle\mathsf{mul}\right\rangle\!\right\rangle^{(A, \mathpzc{\omega})} =\left\langle\!\left\langle\mathbf{Z}_{1}\right\rangle\!\right\rangle^{(A, \mathpzc{\omega})}\cdot\left\langle\!\left\langle\mathbf{Z}_{2}\right\rangle\! \right\rangle^{(A,\mathpzc{\omega})}.\]
2. _There is an arithmetical_ \(\mathsf{FO}+\mathsf{C}\)_-sentence_ \(\mathsf{leq}\) _such that for all structures_ \(A\) _and assignments_ \(\mathpzc{\omega}\) _over_ \(A\)_,_ \[(A,\mathpzc{\omega})\coloneqq\mathsf{leq}\,\Longleftrightarrow\,\left\langle\! \left\langle\mathbf{Z}_{1}\right\rangle\!\right\rangle^{(A,\mathpzc{\omega})} \leq\left\langle\!\left\langle\mathbf{Z}_{2}\right\rangle\!\right\rangle^{(A, \mathpzc{\omega})}.\]

Proof.: These are straightforward consequences of Lemmas 3.9 and 3.14. 

**Lemma 3.20**.: _Let \(\mathbf{Z}\) be an r-schema of type \(\mathsf{v}^{k}\mathsf{n}^{\ell}\to\mathtt{r}\). Furthermore, let \(X\) be a relation variable of type \(\{\mathsf{v}^{k}\mathsf{n}^{\ell}\}\), and let \(V\) be function variable of type \(\mathsf{v}^{k}\to\mathtt{n}\)._
1. _There is an r-expression_ it-add _such that for all structures_ \(A\) _and assignments_ \(\boldsymbol{a}\) _over_ \(A\)_,_ \[\left\langle\!\left\langle\mathsf{tadd}\right\rangle\!\right\rangle^{(A,\, \boldsymbol{a})}=\sum_{(\boldsymbol{a},\boldsymbol{b})}\left\langle\!\left\langle \boldsymbol{Z}\right\rangle\!\right\rangle^{(A,\,\boldsymbol{a})}( \boldsymbol{a},\boldsymbol{b}),\] _where the sum ranges over all_ \((\boldsymbol{a},\boldsymbol{b})\in\boldsymbol{a}(X)\) _such that_ \(\boldsymbol{b}=(b_{1},\ldots,b_{\ell})\in\mathbb{N}^{\ell}\) _with_ \(b_{i}<\boldsymbol{a}(V)(\boldsymbol{a})\) _for all_ \(i\in[\ell]\)_._
2. _There are r-expressions_ \(\mathsf{max}\) _and_ \(\mathsf{min}\) _such that for all structures_ \(A\) _and assignments_ \(\boldsymbol{a}\) _over_ \(A\)_,_ \[\left\langle\!\left\langle\mathsf{max}\right\rangle\!\right\rangle^{(A,\, \boldsymbol{a})} =\max_{(\boldsymbol{a},\boldsymbol{b})}\left\langle\!\left\langle \boldsymbol{Z}\right\rangle\!\right\rangle^{(A,\,\boldsymbol{a})}( \boldsymbol{a},\boldsymbol{b}),\] _where_ \(\max\) _and_ \(\min\) _range over all_ \((\boldsymbol{a},\boldsymbol{b})\in\boldsymbol{a}(X)\) _such that_ \(\boldsymbol{b}=(b_{1},\ldots,b_{\ell})\in\mathbb{N}^{\ell}\) _with_ \(b_{i}<\boldsymbol{a}(V)(\boldsymbol{a})\) _for all_ \(i\in[\ell]\)_._

_Furthermore, if \(k=0\), then the r-expressions_ itadd_, \(\mathsf{max}\), and \(\mathsf{min}\) are arithmetical._

Proof.: To express iterated addition, we first split the family of numbers into the positive and negative numbers. We take the sums over these two subfamilies separately and then combine the results using Lemma3.19. To take the sum over a family of nonnegative dyadic rationals, we apply Lemma3.16 for the numerator and Lemma3.15 for the denominator.

To express maximum and minimum, it clearly suffices to express the maximum and minimum of a family of nonnegative dyadic rationals \((p_{i}\cdot 2^{-s_{i}})_{i\in\mathcal{I}}\) for some definable finite index set \(\mathcal{I}\). Using Lemma3.15, we can determine \(s=\max_{i\in\mathcal{I}}s_{i}\). Then we need to determine maximum and minimum of the natural numbers \(q_{i}\coloneqq p_{i}2^{s-s_{i}}\), which we can do by applying Lemma3.17. 

For division, the situation is slightly more complicated, because the dyadic rationals are not closed under division. We only get an approximation. We use a \(0\)-ary function variable to control the additive approximation error.

**Lemma 3.21**.: _Let \(\boldsymbol{Z}_{1},\boldsymbol{Z}_{2}\) be r-schemas of type \(\varnothing\to\mathtt{r}\), and let \(W\) be a function variable of type \(\varnothing\to\mathtt{n}\). Then there is an arithmetical r-expression \(\mathsf{div}\) such that for all structures \(A\) and assignments \(\boldsymbol{a}\) over \(A\), if \(\left\langle\!\left\langle\boldsymbol{Z}_{2}\right\rangle\!\right\rangle^{(A, \,\boldsymbol{a})}\neq 0\) then_

Proof.: This follows easily from Lemma3.9.
### Evaluating Feedforward Neural Networks

The most important consequence the results of the previous section have for us is that we can simulate rational piecewise-linear FNNs.

Let us first see how we deal with the activation functions. To represent a rational piecewise linear function, we need an integer \(k\) as well as three families of dyadic rationals: the thresholds \((t_{i})_{1\leq i\leq k}\), the slopes \((a_{i})_{0\leq i\leq k}\) and the constant terms \((b_{i})_{0\leq i\leq k}\). An _\(L\)-schema_ of type \(\mathbf{t}\to\mathsf{L}\), for some \(\mathbf{t}\in\{\mathsf{v},\mathsf{n}\}^{k}\), is a tuple \(\mathbf{Z}=(Z_{\mathrm{len}},\mathbf{Z}_{\mathrm{th}},\mathbf{Z}_{\mathrm{sl}},\mathbf{Z}_{ \mathrm{co}})\), where \(Z_{\mathrm{len}}\) is a function variable of type \(\mathbf{t}\to\mathsf{n}\) and \(\mathbf{Z}_{\mathrm{th}},\mathbf{Z}_{\mathrm{sl}},\mathbf{Z}_{\mathrm{co}}\) are r-schemas of type \(\mathbf{n}\mathbf{t}\). Let \(A\) be a structure, \(\omega\) an assignment over \(A\), and \(\mathbf{c}\in A^{\mathbf{t}}\). Let \(k\coloneqq\omega(Z_{\mathrm{len}})(\mathbf{c})\). For \(1\leq i\leq k\), let \(t_{i}\coloneqq\llparent{\mathbf{Z}_{\mathrm{th}}}^{(A,\,\omega)}(i,\mathbf{c})\). For \(0\leq i\leq k\), let \(a_{i}\coloneqq\llparent{\mathbf{Z}_{\mathrm{sl}}}^{(A,\,\omega)}(i,\mathbf{c})\) and \(b_{i}\coloneqq\llparent{\mathbf{Z}_{\mathrm{co}}}^{(A,\,\omega)}(i,\mathbf{c})\). Then if \(t_{1}<\ldots<t_{k}\) and for all \(i\in[k]\) we have \(a_{i-1}t_{i}+b_{i-1}=a_{i}t_{i}+b_{i}\), we define \(\llparent{\mathbf{Z}}^{(A,\,\omega)}:\mathbb{R}\to\mathbb{R}\) to be the rational piecewise linear function with thresholds \(t_{i}\), slopes \(a_{i}\), and constants \(b_{i}\). The condition \(a_{i-1}t_{i}+b_{i-1}=a_{i}t_{i}+b_{i}\) guarantees that this function is continuous. Otherwise, we define \(\llparent{\mathbf{Z}}^{(A,\,\omega)}:\mathbb{R}\to\mathbb{R}\) to be identically \(0\). We can also define _\(L\)-expressions_ consisting of formulas of the appropriate types.

**Lemma 3.22**.: _Let \(\mathbf{Y}\) be an \(L\)-schema of type \(\varnothing\to\mathsf{L}\), and let \(\mathbf{Z}\) be an r-schema of type \(\varnothing\to\mathsf{r}\). Then there is an arithmetical r-expression \(\mathsf{apply}\) such that for all numerical assignments \(\omega\),_

Proof.: This follows easily from Lemma 3.19. 

To represent an FNN we need to represent the skeleton as well as all activation functions and parameters. An _F-schema_ of type \(\mathbf{t}\to\mathsf{F}\) for some \(\mathbf{t}\in\{\mathsf{v},\mathsf{n}\}^{k}\) is a tuple \(\mathbf{Z}=(Z_{\mathrm{V}},Z_{\mathrm{E}},\mathbf{Z}_{\mathrm{ac}},\mathbf{Z}_{\mathrm{wt }},\mathbf{Z}_{\mathrm{bi}})\) where \(Z_{\mathrm{V}}\) is a function variable \(\mathbf{t}\to\mathsf{n}\), \(Z_{\mathrm{E}}\) is a relation variable of type \(\{\mathsf{n}^{2}\mathbf{t}\}\), \(\mathbf{Z}_{\mathrm{ac}}\) is an L-schema of type \(\mathsf{n}\mathbf{t}\to\mathsf{L}\), \(\mathbf{Z}_{\mathrm{wt}}\) is an r-schema of type \(\mathsf{n}\mathbf{}^{2}\mathbf{t}\to\mathsf{r}\), and \(\mathbf{Z}_{\mathrm{bi}}\) is an r-schema of type \(\mathsf{n}\mathbf{t}\to\mathsf{r}\). Then for every structure \(A\), every assignment \(\omega\) over \(A\), and every tuple \(\mathbf{c}\in A^{\mathbf{t}}\), we define \((V,E,(\mathfrak{a}_{v})_{v\in V},(w_{e})_{e\in E},(b_{v})_{v\in V})\) as follows:

* \(V\coloneqq\{0,\ldots,\omega(Z_{\mathrm{V}})(\mathbf{c})\}\);
* \(E\coloneqq\{ij\in V^{2}\mid ij\mathbf{c}\in\omega(Z_{\mathrm{E}})\}\);
* \(\mathfrak{a}_{i}=\llparent{\mathbf{Z}_{\mathrm{ac}}}^{(A,\,\omega)}(i,\mathbf{c})\) for \(i\in V\);
* \(w_{ij}=\llparent{\mathbf{Z}_{\mathrm{wt}}}^{(A,\,\omega)}(i,j,\mathbf{c})\) for \(ij\in E\);
* \(b_{i}=\llparent{\mathbf{Z}_{\mathrm{bi}}}^{(A,\,\omega)}(i,\mathbf{c})\) for \(i\in V\).

Then if \((V,E)\) is a dag, \(\llparent{\mathbf{Z}}^{(A,\,\omega)}\coloneqq(V,E,(\mathfrak{a}_{v})_{v\in V},(w_ {e})_{e\in E},(b_{v})_{v\in V})\) is an FNN. The input nodes \(X_{1},\ldots,X_{p}\) of this FNN are the sources of the dag \((V,E)\) in their natural order (as natural numbers). Similarly, the output nodes \(Y_{1},\ldots,Y_{q}\) of the FNN are the sinks of the dag \((V,E)\) in their natural order. If \((V,E)\) is not a dag, we simply define \(\llparent{\mathbf{Z}}^{(A,\,\omega)}\) to be the trivial FNN with a single node, which computes the identity function. We can also define _F-expressions_ consisting of formulas of the appropriate types.
**Lemma 3.23**.: _Let \(\mathbf{Z}\) be an F-schema of type \(\varnothing\to\mathbb{F}\), and let \(\mathbf{X}\) be an r-schema of type \(\mathtt{n}\to\mathtt{r}\). Then for every \(t\geq 0\) there is an arithmetical r-expression \(\mathsf{eval}_{t}(y)\) such that the following holds. Let \(\omega\) be a numerical assignment and \(\mathfrak{F}\coloneqq\llparent{\mathbf{Z}}^{\omega}\). Suppose that the input dimension of \(\mathfrak{F}\) is \(p\), and let_

\[\mathbf{x}\coloneqq\big{(}\llparent{\mathbf{X}}^{\omega}\,(0),\ldots,\llparent{\mathbf{X}}^ {\omega}\,(p-1)\big{)}.\]

_Then for every node \(v\) of \(\mathfrak{F}\) of depth \(t\) it holds that_

\[f_{\mathfrak{F},v}(\mathbf{x})=\llparent{\mathsf{eval}_{t}}^{\omega}\,(v).\]

Proof.: Using the formulas for multiplication and iterated addition, it easy to construct \(\mathsf{eval}_{t}\) by induction on \(t\). 

**Corollary 3.24**.: _Let \(\mathbf{Z}\) be an F-schema of type \(\varnothing\to\mathbb{F}\), and let \(\mathbf{X}\) be r-schemas of type \(\mathtt{n}\to\mathtt{r}\). Then for every \(d>0\) there is an arithmetical r-expression \(\mathsf{eval}_{d}(y)\) such that the following holds. Let \(A\) be a structure, \(\omega\) an assignment, and \(\mathfrak{F}\coloneqq\llparent{\mathbf{Z}}^{(A,\,\omega)}\). Suppose that the depth of \((V,E)\) is at most \(d\), and let \(p\) be the input dimension and \(q\) the output dimension. Let_

\[\mathbf{x}=\big{(}\llparent{\mathbf{X}}^{\omega}\,(0),\ldots,\llparent{\mathbf{X}}^{(A,\, \omega)}\,(p-1)\big{)}.\]

_Then_

\[\mathfrak{F}(\mathbf{x})=\big{(}\llparent{\mathsf{eval}_{d}}^{(A,\,\omega)}\,(0), \ldots,\llparent{\mathsf{eval}_{d}}^{(A,\,\omega)}\,(q-1)\big{)}.\]

**Corollary 3.25**.: _Let \(\mathfrak{F}\) be a rational piecewise linear FNN of input dimension \(p\) and output dimension \(q\), and let \(\mathbf{X}_{1},\ldots,\mathbf{X}_{p}\) be r-schemas of type \(\varnothing\to\mathtt{r}\). Then for all \(i\in[q]\) there is an arithmetical r-expression \(\mathsf{eval}_{\mathfrak{F},i}\) such that for all structures \(A\) and assignments \(\omega\) over \(A\),_

### Fragments of FO+C

To describe the expressiveness of graph neural networks, we need to consider various fragments of FO+C. For \(k\geq 1\), the _\(k\)-variable fragment_\(\mathsf{FO}^{k}+\mathsf{C}\) of FO+C consists of all formulas with at most \(k\) vertex variables. Importantly, the number of number variables is unrestricted. We call an \(\mathsf{FO}^{k}+\mathsf{C}\)-formula _decomposable_ if it contains no relation variables or function variables and every subformula with exactly \(k\) free vertex variables is a Boolean combination of relational atoms and formulas with at most \(k-1\) free vertex variables. Equivalently, an \(\mathsf{FO}^{k}+\mathsf{C}\)-formula is decomposable if it contains no relation variables or function variables and every subformula of the form \(\theta\leq\theta^{\prime}\), for terms \(\theta,\theta^{\prime}\), has at most \(k-1\) free vertex variables. Note that this implies that a decomposable \(\mathsf{FO}^{k}+\mathsf{C}\)-formula contains no terms with \(k\) free vertex variables.

**Example 3.26**.: The \(\mathsf{FO}^{2}+\mathsf{C}\)-formula

\[\varphi(z)\coloneqq\exists x_{1}.\exists x_{2}.\big{(}E(x_{1},x_{2})\wedge z= \#y_{1}<\mathsf{ord}.E(x_{1},y_{1})\wedge z=\#y_{2}<\mathsf{ord}.E(x_{2},y_{2 })\big{)}\]
is decomposable, whereas the \(\mathsf{FO}^{2}+\mathsf{C}\)-formula

\[\psi(z)\coloneqq\exists x_{1}.\exists x_{2}.\Big{(}E(x_{1},x_{2})\wedge z=\big{(} \#y_{1}<\mathsf{ord}.E(x_{1},y_{1})\big{)}\cdot\big{(}\#y_{2}<\mathsf{ord}.E(x_{ 2},y_{2})\big{)}\Big{)}\]

is not. However, \(\psi(z)\) is decomposable if viewed as an \(\mathsf{FO}^{3}+\mathsf{C}\)-formula.

**Lemma 3.27**.: _Let \(\varphi\) be an \(\mathsf{FO}+\mathsf{C}\)-formula of vocabulary \(\tau\cup\{\leqslant\}\) with at most one free vertex variable and no relation or function variables. Then there is a decomposable \(\mathsf{FO}^{2}+\mathsf{C}\)-formula \(\varphi^{\prime}\) such that for all ordered \(\tau\)-structures \(A\) and all assignments \(\omega\) over \(A\) it holds that \(A\vDash\varphi\iff A\vDash\varphi^{\prime}\)._

Proof.: We first define a bijection between the vertices of the structure \(A\) and an initial segment of \(\mathbb{N}\). We simply let \(\mathsf{bij}(x,y)\coloneqq\#x^{\prime}.x^{\prime}\leqslant x=\#\big{(}y^{ \prime}\leq\mathsf{ord}\big{)}.y^{\prime}\leq y\). We introduce a distinguished number variable \(y_{x}\) for every vertex variable \(x\).

To obtain \(\varphi^{\prime}\) from \(\varphi\), we first replace quantification over \(x\) in counting terms by quantification over \(y_{x}\), that is, we replace \(\#(x,\ldots)\) by \(\#(y_{x}<\mathsf{ord},\ldots)\). Furthermore, we replace atomic formulas \(x=x^{\prime}\) by \(y_{x}=y_{x^{\prime}}\) (or, more formally, \(y_{x}\leq y_{x^{\prime}}\wedge y_{x^{\prime}}\leq y_{x}\)) and atomic \(R(x,x^{\prime})\) by \(\exists x_{1}.\exists x_{2}.\big{(}\mathsf{bij}(x_{1},y_{x})\wedge\mathsf{bij} (x_{2},y_{x^{\prime}})\wedge R(x_{1},x_{2})\big{)}\). Let \(\psi\) be the resulting formula. If \(\varphi\) has no free variables, we let \(\varphi^{\prime}\coloneqq\psi\). If \(\varphi\) has one free variable \(x\), we let

\[\varphi^{\prime}=\exists y_{x}<\mathsf{ord}.\big{(}\mathsf{bij}(x,y_{x})\wedge \psi\big{)}.\]

Then \(\varphi^{\prime}\) is equivalent to \(\varphi\), and it only contains the vertex variables \(x_{1},x_{2}\) and hence is in \(\mathsf{FO}^{2}+\mathsf{C}\). It is easy to check that the formula is decomposable. 

_Remark 3.28_.: Note that Lemma 3.27 implies that on ordered structures, every \(\mathsf{FO}^{2}+\mathsf{C}\)-formula with at most one free variable is equivalent to a decomposable formula. It is an open problem whether this holds on arbitrary structures.

The definition of decomposable \(\mathsf{FO}^{k}+\mathsf{C}\) is not particularly intuitive, at least at first glance. However, we wonder if "decomposable \(\mathsf{FO}^{k}+\mathsf{C}\)" is what we actually want as the \(k\)-variable fragment of \(\mathsf{FO}+\mathsf{C}\). This view is not only supported by Lemma 3.27, but also by the observation that the logic \(\mathsf{C}^{k}\) (the \(k\)-variable fragment of the extension of first-order logic by counting quantifiers \(\exists^{\geq n}x\)) is contained in decomposable \(\mathsf{FO}^{k}+\mathsf{C}\). Furthermore, characterisations of \(k\)-variable logics in terms of pebble games or the WL-algorithm only take atomic properties of \(k\)-tuples into account.

The _guarded fragment_\(\mathsf{GFO}+\mathsf{C}\) is a fragment of \(\mathsf{FO}^{2}+\mathsf{C}\) where quantification and counting is restricted to range over neighbours of a free variable. We fix two variables \(x_{1},x_{2}\). A _guard_ is an atomic formula of the form \(R(x_{i},x_{3-i})\) for some binary relation symbol \(R\). We inductively define the sets of \(\mathsf{GFO}+\mathsf{C}\)-_terms_ and \(\mathsf{GFO}+\mathsf{C}\)-_formulas_ as follows.

* All number variables and \(\mathsf{0},\mathsf{1},\mathsf{ord}\) are \(\mathsf{GFO}+\mathsf{C}\)-terms.
* For all \(\mathsf{GFO}+\mathsf{C}\)-terms \(\theta,\theta^{\prime}\), the expressions \(\theta+\theta^{\prime}\) and \(\theta\cdot\theta^{\prime}\) are \(\mathsf{GFO}+\mathsf{C}\)-terms.
* For all function variables \(U\) of type \((t_{1},\ldots,t_{k})\to\mathtt{n}\) and all tuples \((\xi_{1},\ldots,\xi_{k})\), where \(\xi_{i}\) is a vertex variable if \(t_{i}=\mathtt{v}\) and \(\xi_{i}\) is a \(\mathsf{GFO}+\mathsf{C}\)-term if \(t_{i}=\mathtt{n}\), the expression \(U(\xi_{1},\ldots,\xi_{k})\) is a \(\mathsf{GFO}+\mathsf{C}\)-term.
* For all \(\mathsf{GFO}+\mathsf{C}\)-terms \(\theta,\theta^{\prime}\), the expression \(\theta\leq\theta^{\prime}\) is a \(\mathsf{GFO}+\mathsf{C}\)-formula.
* All relational atoms whose variables are among \(x_{1},x_{2}\) are \(\mathsf{GFO}+\mathsf{C}\)-formulas.
* For all relation variables \(X\) of type \(\{(t_{1},\ldots,t_{k})\}\) and all tuples \((\xi_{1},\ldots,\xi_{k})\), where \(\xi_{i}\) is a vertex variable if \(t_{i}=\mathtt{v}\) and \(\xi_{i}\) is a \(\mathsf{GFO}+\mathsf{C}\)-term if \(t_{i}=\mathtt{n}\), the expression \(X(\xi_{1},\ldots,\xi_{k})\) is a \(\mathsf{GFO}+\mathsf{C}\)-formula.
* For all \(\mathsf{GFO}+\mathsf{C}\)-formulas \(\varphi,\psi\) the expressions \(\neg\varphi\) and \(\varphi\wedge\psi\) are \(\mathsf{GFO}+\mathsf{C}\)-formulas.
* For all \(\mathsf{GFO}+\mathsf{C}\)-formulas \(\varphi\), all guards \(\gamma\), all number variables \(y_{1},\ldots,y_{k}\), all \(\mathsf{GFO}+\mathsf{C}\)-terms \(\theta_{1},\ldots,\theta_{k}\), and \(i=1,2\), \[\#(x_{3-i},y_{1}<\theta_{1},\ldots,y_{k}<\theta_{k}).(\gamma\wedge\varphi),\] (3.N) is a \(\mathsf{GFO}+\mathsf{C}\)-term.
* For all \(\mathsf{GFO}+\mathsf{C}\)-formulas \(\varphi\), all number variables \(y_{1},\ldots,y_{k}\), and all \(\mathsf{GFO}+\mathsf{C}\)-terms \(\theta_{1},\ldots,\theta_{k}\), \[\#(y_{1}<\theta_{1},\ldots,y_{k}<\theta_{k}).\varphi,\] (3.O) is a \(\mathsf{GFO}+\mathsf{C}\)-term.

Observe that a \(\mathsf{GFO}+\mathsf{C}\)-term or \(\mathsf{GFO}+\mathsf{C}\)-formula either has at least one free vertex variable or contains no vertex variable at all. Note that we add \(\mathsf{ord}\) as a "built-in" constant that is always interpreted by the order of the input structure. We need access to the order of a structure to bound quantification on numbers, and the closed \(\mathsf{FO}^{2}+\mathsf{C}\)-term \(\mathsf{ord}=\#x.x=x\) defining the order is not in \(\mathsf{GFO}+\mathsf{C}\).

An r-expression is _guarded_ if all its formulas and terms are in \(\mathsf{GFO}+\mathsf{C}\).

_Remark 3.29_.: Our definition of the guarded fragment is relatively liberal in terms of which kind of formulas \(\varphi\) we allow inside the guarded counting operators in (3.N). In particular, we allow both \(x_{i}\) and \(x_{3-i}\) to occur freely in \(\varphi\). A more restrictive alternative definition, maybe more in the spirit of a modal logic, would be to stipulate that the variable \(x_{i}\) must not occur freely in \(\varphi\).

This is related to a similar choice we made in the definition of graph neural networks (see Remark 4.1).

By definition, \(\mathsf{GFO}+\mathsf{C}\) is contained in \(\mathsf{FO}^{2}+\mathsf{C}\). The converse does not hold. Let us introduce an intermediate fragment \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\) which extends \(\mathsf{GFO}+\mathsf{C}\) and is still in \(\mathsf{FO}^{2}+\mathsf{C}\). We call \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\) the _guarded fragment with global counting_. In addition to the guarded counting terms in (3.N), in \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\) formulas we also allow a restricted form of unguarded counting in the form

\[\#(x_{3-i},y_{1}<\theta_{1},\ldots,y_{k}<\theta_{k}).\varphi, \tag{3.P}\]

where the variable \(x_{i}\) must not occur freely in \(\varphi\). Intuitively, such a term makes a "global" calculation that is unrelated to the "local" properties of the free variable \(x_{i}\).

Let us call a \(\mathsf{GFO}+\mathsf{C}\)-formula or a \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\)-formula _decomposable_ if it is decomposable as an \(\mathsf{FO}^{2}+\mathsf{C}\)-formula.
**Lemma 3.30**.: _For every decomposable \(\mathsf{FO}^{2}+\mathsf{C}\)-formula \(\varphi\) there is a decomposable \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\)-formula \(\varphi^{\prime}\) such that for all graphs \(G\), possibly labelled, and all assignments \(\omega\) over \(G\) we have_

\[(G,\omega)\vDash\varphi\iff(G,\omega)\vDash\varphi^{\prime}.\]

Proof.: We write \(x\) and \(x^{\prime}\) to refer to the variables \(x_{1},x_{2}\), with the understanding that if \(x\) refers to \(x_{i}\) then \(x^{\prime}\) refers to \(x_{3-i}\). We need to replace unguarded terms by combinations of terms of the form (3.N), (3.O), and (3.P).

_Claim 1._ Every term \(\eta\coloneqq\#(x^{\prime},y_{1}<\theta_{1},\ldots,y_{k}<\theta_{k}).\psi\), where \(\psi\) is a \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\)-formula and the \(\theta_{i}\) are \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\)-terms, is equivalent to a \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\)-term.

Proof.: By Lemma 3.2, we may assume without loss of generality that the terms \(\theta_{i}\) do not contain the variables \(x,x^{\prime},y_{1},\ldots,y_{k}\) (and therefore the counting operator is nonadaptive). Indeed, by the lemma we can find a term \(\theta\) built from \(\mathsf{ord}\) and the free number variables of \(\eta\) such that \(\theta\) bounds all \(\theta_{i}\). Then \(\eta\) is equivalent to \(\#(x^{\prime},y_{1}<\theta,\ldots,y_{k}<\theta).(\psi\wedge y_{1}<\theta_{1} \wedge\ldots\wedge y_{k}<\theta_{k})\).

Also without loss of generality we may assume that \(k\geq 1\), because we can always append \(y<1\) for a fresh variable \(y\) in the counting operator without changing the result. To simplify the notation, let us assume that \(k=1\). The generalisation to larger \(k\) is straightforward. Hence

\[\eta=\#(x^{\prime},y<\theta).\psi \tag{3.Q}\]

and \(\theta\) is a term in which neither \(x\) nor \(x^{\prime}\) is free.

Since \(\psi\) is decomposable, we can re-write \(\psi\) as

\[(E(x,x^{\prime})\wedge\psi_{1})\vee(\neg E(x,x^{\prime})\wedge\neg x=x^{ \prime}\wedge\psi_{2})\vee\psi_{3},\]

where \(\psi_{1}\) and \(\psi_{2}\) are Boolean combinations of formulas with at most one free vertex variable (either \(x\) or \(x^{\prime}\)), and \(x^{\prime}\) does not occur freely in \(\psi_{3}\). To see this, note that in graphs all vertices \(x,x^{\prime}\) satisfy exactly one of \(E(x,x^{\prime})\), \(\neg E(x,x^{\prime})\wedge\neg x=x^{\prime}\), and \(x=x^{\prime}\). Thus \(\psi\) is equivalent to the disjunction

\[(E(x,x^{\prime})\wedge\psi)\vee(\neg E(x,x^{\prime})\wedge\neg x=x^{\prime} \wedge\psi)\vee(x=x^{\prime}\wedge\psi).\]

Then to obtain \(\psi_{1}\) and \(\psi_{2}\), we eliminate all atomic formulas \(E(x,x^{\prime}),x=x^{\prime}\) in both free variables from \(\psi\), and to obtain \(\psi_{3}\) we substitute \(x\) for all free occurrence of \(x^{\prime}\). Thus \(\eta\) is equivalent to the term

\[\#(x^{\prime},y<\theta).(E(x,x^{\prime})\wedge\psi_{1}) \tag{3.R}\] \[+\#(x^{\prime},y<\theta).(\neg E^{\prime}(x,x^{\prime})\wedge \neg x=x^{\prime}\wedge\psi_{2})\] \[+\#(y<\theta).\psi_{3}.\]

The first and the third term in this sum are already \(\mathsf{GFO}+\mathsf{C}\)-terms of the forms (3.N) and (3.O), respectively. We only need to worry about the second,

\[\eta_{2}\coloneqq\#(x^{\prime},y<\theta).(\neg E^{\prime}(x,x^{\prime}) \wedge\neg x=x^{\prime}\wedge\psi_{2}).\]
We can equivalently re-write \(\eta_{2}\) as

\[\#(x^{\prime},y<\theta).\psi_{2}\] \[\dot{\simeq} \#(x^{\prime},y<\theta).(E(x,x^{\prime})\wedge\psi_{2})\] \[\dot{\simeq} \#(y<\theta).\psi_{2}\tfrac{x}{x^{\prime}},\]

where \(\psi_{2}\tfrac{x}{x^{\prime}}\) denotes the formula obtained from \(\psi_{2}\) by replacing all free occurrences of \(x^{\prime}\) by \(x\). The second and the third term are already \(\mathsf{GFO}+\mathsf{C}\)-terms. We only need to worry about the first,

\[\eta_{2}^{\prime}\coloneqq\#(x^{\prime},y<\theta).\psi_{2}\]

Recall that \(\psi_{2}\) is a Boolean combination of formulas with only one free vertex variable. Bringing this Boolean combination into disjunctive normal form, we obtain an equivalent formula \(\bigvee_{i\in I}(\chi_{i}\wedge\chi_{i}^{\prime})\), where \(x\) does not occur freely in \(\chi_{i}^{\prime}\) and \(x^{\prime}\) does not occur freely in \(\chi_{i}\). We can further ensure that the disjuncts are mutually exclusive, that is, for every pair \(x,x^{\prime}\) there is at most one \(i\) such that that it satisfies \((\chi_{i}\wedge\chi_{i}^{\prime})\). For example, if \(I=\{1,2\}\), we note that \((\chi_{1}\wedge\chi_{1}^{\prime})\vee(\chi_{2}\wedge\chi_{2}^{\prime})\) is equivalent to

\[((\chi_{1}\wedge\chi_{2})\wedge(\chi_{1}^{\prime}\wedge\chi_{2}^{ \prime}))\] \[\vee((\chi_{1}\wedge\neg\chi_{2})\wedge(\chi_{1}^{\prime}\wedge \chi_{2}^{\prime}))\] \[\vee((\chi_{1}\wedge\chi_{2})\wedge(\chi_{1}^{\prime}\wedge \neg\chi_{2}^{\prime}))\] \[\vee((\chi_{1}\wedge\neg\chi_{2})\wedge(\chi_{1}^{\prime}\wedge \neg\chi_{2}^{\prime}))\] \[\vee((\neg\chi_{1}\wedge\chi_{2})\wedge(\chi_{1}^{\prime}\wedge \chi_{2}^{\prime}))\] \[\vee((\chi_{1}\wedge\chi_{2})\wedge(\neg\chi_{1}^{\prime}\wedge \chi_{2}^{\prime}))\] \[\vee((\neg\chi_{1}\wedge\chi_{2})\wedge(\neg\chi_{1}^{\prime} \wedge\chi_{2}^{\prime})).\]

Then \(\eta_{2}^{\prime}\) is equivalent to the term

\[\sum_{i\in I}\#(x^{\prime},y<\theta).(\chi_{i}\wedge\chi_{i}^{\prime}).\]

Consider a summand \(\eta_{2i}\coloneqq\#(x^{\prime},y<\theta).(\chi_{i}\wedge\chi_{i}^{\prime})\). Let \(\zeta\coloneqq\#x^{\prime}.(\chi_{i}\wedge\chi_{i}^{\prime})\) and note that for all graphs \(G\) and assignments \(\omega\) we have

\[[\![\eta_{2i}]\!]^{(G,\,\alpha)}=\sum_{b<[\![\theta]\!]^{(G,\,\alpha)}}[\![ \zeta]\!]^{(G,\,\alpha\,\tfrac{b}{y})}=\big{|}\big{\{}(b,c)\,\big{|}\,b<[\![ \theta]\!]^{(G,\,\alpha)}\,,c<[\![\zeta]\!]^{(G,\,\alpha\,\tfrac{b}{y})}\, \big{\}}\big{|}.\]

Let \(\eta_{2i}^{\prime}\coloneqq\#(y<\theta,z<\mathsf{ord}).z<\zeta\). Since we always have \([\![\zeta]\!]^{(G,\,\alpha)}\leq|G|\), the terms \(\eta_{2i}\) and \(\eta_{2i}^{\prime}\) are equivalent.

The final step is to turn \(\zeta\) into a \(\mathsf{GFO}+\mathsf{C}^{\mathsf{gc}}\)-term. Recall that \(\zeta=\#x^{\prime}.(\chi_{i}\wedge\chi_{i}^{\prime})\) and that \(x^{\prime}\) is not free in \(\chi_{i}\). If \(x\) does not satisfy \(\chi_{i}\), then the term \(\zeta\) evaluates to \(0\), and otherwise it has the same value as the term \(\#x^{\prime}.\chi_{i}^{\prime}\), which is of the form (3.P). Note that the term \(\#(y^{\prime}<1).\chi_{i}\), where \(y^{\prime}\) is a fresh number variable not occurring in \(\chi_{i}\), is of the form (3.O) and evaluates to \(1\) if \(\chi_{i}\) is satisfied and to \(0\) otherwise. Thus the term

\[\#(y^{\prime}<1).\chi_{i}\,\cdot\,\#x^{\prime}.\chi_{i}^{\prime}\]
is a \(\mathsf{GFO}+\mathsf{C}^{\text{gc}}\)-term equivalent to \(\zeta\). This completes the proof of the claim.

_Claim 2._ Every term \(\eta=\#(x,x^{\prime},y_{1}<\theta_{1},\ldots,y_{k}<\theta_{k}).\psi\), where \(\varphi\) is a \(\mathsf{GFO}+\mathsf{C}^{\text{gc}}\)-formula and the \(\theta_{i}\) are \(\mathsf{GFO}+\mathsf{C}^{\text{gc}}\)-terms, is equivalent to a \(\mathsf{GFO}+\mathsf{C}^{\text{gc}}\)-term.

Proof.: Arguing as in the proof of Claim 1, we may assume that

\[\eta=\#(x,x^{\prime},y<\theta).\psi, \tag{3.S}\]

where \(\theta\) is a term in which the variables \(x,x^{\prime}\) are not free.

We proceed very similarly to the proof of Claim 1. The first step is to rewrite the term as the sum

\[\#(x,x^{\prime},y<\theta).(E(x,x^{\prime})\wedge\psi_{1})\] \[+\#(x,x^{\prime},y<\theta).(\neg E^{\prime}(x,x^{\prime})\wedge \neg x=x^{\prime}\wedge\psi_{2}) \tag{3.T}\] \[+\#(x,y<\theta).\psi_{3},\]

where \(\psi_{1}\) and \(\psi_{2}\) are Boolean combinations of formulas with at most one free vertex variable and \(x^{\prime}\) is not free in \(\psi_{3}\). Then the third term is already of the form (3.P), and we only have to deal with the first two.

Let us look at the term \(\eta_{1}\coloneqq\#(x,x^{\prime},y<\theta).(E(x,x^{\prime})\wedge\psi_{1})\). Let \(\zeta_{1}\coloneqq\#(x^{\prime},y<\theta).(E(x,x^{\prime})\wedge\psi_{1})\). Then \(\zeta_{1}\) is a term of the form (3.N). Moreover, for every graph \(G\) and assignment \(\omega\) we have

\[[\![\eta_{1}]\!]^{(G,\omega)}=\sum_{a\in V(G)}[\![\zeta_{1}]\!]^{(G,\omega\, \omega)}=\big{|}\big{\{}(a,c)\,\big{|}\,a\in V(G),c<[\![\zeta_{1}]\!]^{(G,\omega \,\omega)}\,\big{\}}\big{|}.\]

We let

\[\eta^{\prime}_{1}\coloneqq\#(x,z<\mathsf{ord}).z<\zeta_{1}.\]

Then \(\eta^{\prime}_{1}\) is a term of the form (3.P) that is equivalent to \(\eta_{1}\).

It remains to deal with the second summand in (3.T), the term \(\eta_{2}\coloneqq\#(x,x^{\prime},y<\theta).(\neg E^{\prime}(x,x^{\prime}) \wedge\neg x=x^{\prime}\wedge\psi_{2})\). We rewrite this term as

\[\#(x,x^{\prime},y<\theta).\psi_{2}\] (3.U) \[\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!
Consider one of the terms in the sum, \(\eta^{\prime}_{2i}\coloneqq\#(x,x^{\prime},y<\theta).(\chi_{i}\wedge\chi^{\prime}_{ i})\). We let \(\zeta\coloneqq\#(x,x^{\prime}).(\chi_{i}\wedge\chi^{\prime}_{i})\) As in the proof of Claim 1, for all graphs \(G\) and assignments \(\omega\) we have

\[\big{[}\!\big{[}\eta^{\prime}_{2i}\big{]}^{(G,\omega)}=\sum_{b<[\![\theta]\!]^ {(G,\omega)}}\big{[}\![\zeta]\!\big{]}^{(G,\omega\,\tfrac{b}{y})}=\big{|}\big{\{} (b,c)\,\big{|}\,b<[\![\theta]\!]^{(G,\omega)}\,,c<[\![\zeta]\!]^{(G,\omega)}\, \big{\}}\big{|}.\]

Let \(\eta^{\prime\prime}_{2i}\coloneqq\#(y<\theta,z<\mathsf{ord}\cdot\mathsf{ord}).z<\zeta\). Since we always have \(\big{[}\![\zeta]\!\big{]}^{(G,\omega)}\leq|G|^{2}\), the terms \(\eta^{\prime}_{2i}\) and \(\eta^{\prime\prime}_{2i}\) are equivalent.

To turn \(\zeta\) into a \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\)-term \(\zeta^{\prime}\), we observe that for all graphs \(G\) and assignments \(\omega\) we have

\[\big{[}\![\zeta]\!\big{]}^{(G,\omega)}=\big{|}\big{\{}a\in V(G)\,\big{|}\, \big{|}\,(G,\omega\,\tfrac{a}{x}\big{)}\vDash\chi_{i}\big{\}}\big{|}\cdot\big{|} \big{\{}a^{\prime}\in V(G)\,\big{|}\,\big{(}G,\omega\,\tfrac{a^{\prime}}{x^{ \prime}}\big{)}\vDash\chi^{\prime}_{i}\big{\}}\big{|}.\]

We let \(\zeta^{\prime}\coloneqq\#x.\chi_{i}\cdot\#x^{\prime}.\chi^{\prime}_{i}\).

With these two claims, it is easy to inductively translate a decomposable \(\mathsf{FO}^{2}+\mathsf{C}\)-formula into an \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\)-formula. 

Combining Lemmas 3.30 and 3.27 with Corollary 3.6, we obtain the following.

**Corollary 3.31**.: _Let \(\mathds{O}\) be a unary query. Then \(L(\mathds{O})\) is in \(\mathsf{TC}^{0}\) if and only if \(\mathds{O}\) is definable in order-invariant \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}_{\mathrm{nu}}\)._

### Arithmetic in \(\mathsf{GFO}+\mathsf{C}\)

Since all arithmetical \(\mathsf{FO}+\mathsf{C}\)-formulas and terms are in \(\mathsf{GFO}+\mathsf{C}\), most results of Sections 3.4-3.6 apply to \(\mathsf{GFO}+\mathsf{C}\). Exceptions are Lemmas 3.15, 3.16, and 3.20 on iterated addition, which may involve vertex variables. Here we prove variants of these lemmas for the guarded fragment.

**Lemma 3.32**.: _Let \(X\) be a relation variable of type \(\{\mathsf{v}^{2}\mathsf{n}^{\ell}\}\), and let \(U,V\) be function variables of types \(\mathsf{v}^{2}\mathsf{n}^{\ell}\to\mathsf{n}\), \(\mathsf{v}^{2}\to\mathsf{n}\), respectively. Furthermore, let \(\gamma(x,x^{\prime})\) be a guard._

1. _There is a_ \(\mathsf{GFO}+\mathsf{C}\)_-term_ \(\mathsf{u}\mathsf{-itald}(x)\) _such that for all structures_ \(A\)_, assignments_ \(\omega\) _over_ \(A\)_, and_ \(a\in V(A)\)_,_ \[\big{[}\![\mathsf{u}\mathsf{-itald}]\!]^{(A,\omega)}\,(a)=\sum_{(a^{\prime},b )}\omega(U)(a,a^{\prime},\overline{b}),\] _where the sum ranges over all_ \((a^{\prime},\mathbf{b})\in V(A)\times\mathbb{N}^{\ell}\) _such that_ \(A\vDash\gamma(a,a^{\prime})\) _and_ \((a,a^{\prime},\mathbf{b})\in\omega(X)\) _and_ \(\mathbf{b}=(b_{1},\ldots,b_{\ell})\in\mathbb{N}^{\ell}\) _with_ \(b_{i}<\omega(V)(a,a^{\prime})\) _for all_ \(i\in[\ell]\)_._
2. _There are_ \(\mathsf{GFO}+\mathsf{C}\)_-terms_ \(\mathsf{u}\mathsf{-max}(x)\) _and_ \(\mathsf{u}\mathsf{-min}(x)\) _such that for all structures_ \(A\)_, assignments_ \(\omega\) _over_ \(A\)_, and_ \(a\in V(A)\)_,_ \[\big{[}\![\mathsf{u}\mathsf{-max}]\!]^{(A,\omega)}\,(a)=\max_{(a^{\prime}, \overline{b})}\omega(U)(a,a^{\prime},\overline{b}),\]
\[\left[\mathsf{u}\text{-}\mathsf{min}\right]^{(A,a)}(a)=\min_{(a^{\prime},\vec{b})} \alpha(U)(a,a^{\prime},\vec{b}),\] _where_ \(\max\) _and_ \(\min\) _range over all_ \((a^{\prime},\vec{b})\in V(A)\times\mathbb{N}^{\ell}\) _such that_ \(A\vDash\gamma(a,a^{\prime})\) _and_ \((a,a^{\prime},\vec{b})\in\omega(X)\) _and_ \(\vec{b}=(b_{1},\ldots,b_{\ell})\in\mathbb{N}^{\ell}\) _with_ \(b_{i}<\omega(V)(a,a^{\prime})\) _for all_ \(i\in[\ell]\)_._

Proof.: The proof is very similar to the proof of Lemma 3.15, we just have to make sure that the terms we define are guarded. Again, we assume for simplicity that \(\ell=1\).

We let

\[\mathsf{u}\text{-}\mathsf{i}\mathsf{t}\mathsf{add}(x)\coloneqq\#\big{(}x^{ \prime},y<V(x,x^{\prime}),z<U(\vec{x},y)\big{)}.\big{(}\gamma(x,x^{\prime}) \wedge X(x,x^{\prime},y)\big{)}\]

and

\[\mathsf{u}\text{-}\mathsf{max}(x) \coloneqq\#z<\mathsf{u}\text{-}\mathsf{i}\mathsf{t}\mathsf{add}( x).\exists(x^{\prime},y<V(x,x^{\prime})).\big{(}\gamma(x,x^{\prime})\wedge X(x,x ^{\prime},y)\wedge z<U(x,x^{\prime})\big{)},\] \[\mathsf{u}\text{-}\mathsf{min}(x) \coloneqq\#z<\mathsf{u}\text{-}\mathsf{i}\mathsf{t}\mathsf{add}( x).\forall(x^{\prime},y<V(x,x^{\prime})).\big{(}\gamma(x,x^{\prime})\wedge X(x,x ^{\prime},y)\to z<U(x,x^{\prime})\big{)}.\]

**Lemma 3.33**.: _Let \(\vec{Z}\) be an r-schema of type \(\mathsf{v}^{2}\mathsf{n}^{\ell}\to\mathtt{r}\). Let \(X\) be a relation variable of type \(\{\mathsf{v}^{2}\mathsf{n}^{\ell}\}\), and let \(V\) be a function variable of type \(\mathsf{v}^{2}\to\mathtt{n}\). Furthermore, let \(\gamma(x,x^{\prime})\) be a guard._

1. _There is a guarded r-expression_ \(\mathsf{i}\mathsf{t}\mathsf{add}(x)\) _such that for all structures_ \(A\)_, assignments_ \(\omega\) _over_ \(A\)_, and_ \(a\in V(A)\) _we have_ \[\langle\!\langle\mathsf{i}\mathsf{t}\mathsf{add}\rangle\!\rangle^{(A,a)}\,(a )=\sum_{(a^{\prime},\vec{b})}\langle\!\langle\vec{Z}\rangle\!\rangle^{(A,a)} \,(a,a^{\prime},\vec{b}),\] _where the sum ranges over all_ \((a^{\prime},\vec{b})\in V(A)\times\mathbb{N}^{\ell}\) _such that_ \(A\vDash\gamma(a,a^{\prime})\) _and_ \((a,a^{\prime},\vec{b})\in\omega(X)\) _and_ \(\vec{b}=(b_{1},\ldots,b_{\ell})\in\mathbb{N}^{\ell}\) _with_ \(b_{i}<\omega(V)(a,a^{\prime})\) _for all_ \(i\in[\ell]\)_._
2. _There are guarded r-expressions_ \(\mathsf{max}(x)\) _and_ \(\mathsf{min}(x)\) _such that for all structures_ \(A\)_, assignments_ \(\omega\) _over_ \(A\)_, and_ \(a\in V(A)\) _we have_ \[\langle\!\langle\mathsf{max}\rangle\!\rangle^{(A,a)}\,(a) =\max_{(a^{\prime},\vec{b})}\langle\!\langle\vec{Z}\rangle\! \rangle^{(A,a)}\,(a,a^{\prime},\vec{b}),\] \[\langle\!\langle\mathsf{min}\rangle\!\rangle^{(A,a)}\,(a) =\min_{(a^{\prime},\vec{b})}\langle\!\langle\vec{Z}\rangle\! \rangle^{(A,a)}\,(a,a^{\prime},\vec{b}),\] _where_ \(\max\) _and_ \(\min\) _range over all_ \((a^{\prime},\vec{b})\in V(A)\times\mathbb{N}^{\ell}\) _such that_ \(A\vDash\gamma(a,a^{\prime})\) _and_ \((a,a^{\prime},\vec{b})\in\omega(X)\) _and_ \(\vec{b}=(b_{1},\ldots,b_{\ell})\in\mathbb{N}^{\ell}\) _with_ \(b_{i}<\omega(V)(a,a^{\prime})\) _for all_ \(i\in[\ell]\)_._

Proof.: The proof is an easy adaptation of the proof of Lemma 3.20, arguing as in the proof of Lemma 3.32 to make sure that we obtains guarded formulas and terms.
## 4 Graph Neural Networks

We will work with standard message passing graph neural networks (GNNs5) [10]. A GNN consists of a finite sequence of layers. A _GNN layer_ of input dimension \(p\) and output dimension \(q\) is a triple \(\mathfrak{L}=(\mathsf{msg},\mathsf{agg},\mathsf{comb})\) of functions: a _message function_\(\mathsf{msg}:\mathbb{R}^{2p}\to\mathbb{R}^{p^{\prime}}\), an _aggregation function_\(\mathsf{agg}\) mapping finite multisets of vectors in \(\mathbb{R}^{p^{\prime}}\) to vectors in \(\mathbb{R}^{p^{\prime\prime}}\), and a _combination function_\(\mathsf{comb}:\mathbb{R}^{p+p^{\prime\prime}}\to\mathbb{R}^{q}\). A _GNN_ is a tuple \(\mathfrak{N}=(\mathfrak{L}^{(1)},\ldots,\mathfrak{L}^{(d)})\) of GNN layers, where the output dimension \(q^{(i)}\) of \(\mathfrak{L}^{(i)}\) matches the input dimension \(p^{(i+1)}\) of \(\mathfrak{L}^{(i+1)}\). We call \(q^{(0)}\coloneqq p^{(1)}\) the input dimension of \(\mathfrak{N}\) and \(q^{(d)}\) the output dimension.

Footnote 5: We use the abbreviation GNN, but MPNN is also very common.

To define the semantics, let \(\mathfrak{L}=(\mathsf{msg},\mathsf{agg},\mathsf{comb})\) be a GNN layer of input dimension \(p\) and output dimension \(q\). It computes a function \(\mathfrak{L}\colon\mathcal{GF}_{p}\to\mathcal{FG}_{q}\) (as for circuits and feedforward neural networks, we use the same letter to denote the network and the function it computes) defined by \(\mathfrak{L}(G,\mathpzc{x})\coloneqq(G,\mathpzc{y})\), where \(\mathpzc{y}:V(G)\to\mathbb{R}^{q}\) is defined by

\[\mathpzc{x}(v)\coloneqq\mathsf{comb}\Bigg{(}\mathpzc{x}(v),\mathsf{agg} \Big{(}\Big{\{}\mathsf{msg}\big{(}\mathpzc{x}(v),\mathpzc{x}(w)\big{)}\; \Big{|}\;w\in N_{G}(v)\Big{\}}\Big{)}\Bigg{)}. \tag{4.A}\]

A GNN \(\mathfrak{N}=(\mathfrak{L}^{(1)},\ldots,\mathfrak{L}^{(d)})\) composes the transformations computed by its layers \(\mathfrak{L}^{(i)}\), that is, it computes the function \(\mathfrak{N}\colon\mathcal{GF}_{q^{(0)}}\to\mathcal{GF}_{q^{(d)}}\) defined by

\[\mathfrak{N}(G,\mathpzc{x})\coloneqq\mathfrak{L}^{(d)}\circ\mathfrak{L}^{(d- 1)}\circ\ldots\circ\mathfrak{L}^{(1)}.\]

It will be convenient to also define \(\widetilde{\mathfrak{N}}\) as the function mapping \((G,\mathpzc{x})\) to the signal \(\mathpzc{x}^{\prime}\in\mathcal{S}_{q^{(d)}}(G)\) such that \(\mathfrak{N}(G,\mathpzc{x})=(G,\mathpzc{x}^{\prime})\), so \(\mathfrak{N}(G,\mathpzc{x})=(G,\widetilde{\mathfrak{N}}(G,\mathpzc{x}))\), and similarly \(\widetilde{\mathfrak{L}}\) for a single layer \(\mathfrak{L}\).

Our version of GNNs corresponds to the _message passing neural networks_ due to [10]. Another version that can be found in the literature, cleanly formalised as the _aggregate-combine GNNs_ in [3], only allows the messages to depend on the vertex they are sent form. So the \(\mathsf{msg}(\mathpzc{x}(v),\mathpzc{x}(w))\) in (4.A) becomes \(\mathsf{msg}(\mathpzc{x}(w))\). On the surface, our version is more powerful, but it is not clear to us if it is really more expressive.

The reason we decided to use the version with messages depending on both endvertices of the edge is that in practical work we also found it beneficial to use this version. However, slightly adapting the logic (cf. Remark 3.29), our results also have a version for aggregate-combine GNNs.

So far, we have defined GNNs as an abstract computation model computing transformations between graph signals. To turn them into deep learning models, we represent the functions that specify the layers by feedforward neural networks. More precisely, we assume that the message functions \(\mathsf{msg}\) and the combination functions \(\mathsf{comb}\) of all GNN layers are specified by FNNs \(\mathfrak{F}_{\mathsf{msg}}\) and \(\mathfrak{F}_{\mathsf{comb}}\). Furthermore, we assume that the aggregation function \(\mathsf{agg}\) is either summation \(\mathsf{SUM}\) or arithmetic mean \(\mathsf{MEAN}\) or maximum \(\mathsf{MAX}\). Note that this means that the aggregation function does not change the dimension,
that is, we always have \(p^{\prime}=p^{\prime\prime}\) (referring to the description of GNN layers above). To be able to deal with isolated nodes as well, we define \(\mathsf{SUM}(\varnothing)\coloneqq\mathsf{MEAN}(\varnothing)\coloneqq\mathsf{ MAX}(\varnothing)\coloneqq\mathbf{0}\).

If the FNNs \(\mathfrak{F}_{\mathsf{msg}}\) and \(\mathfrak{F}_{\mathsf{comb}}\) on all layers are (rational) piecewise linear, we call the GNN _(rational) piecewise linear_. Similarly, if they are rpl-approximable, we call the GNN _rpl-approximable_.

We mention a few extensions of the basic GNN model. Most importantly, in a GNN with _global readout_[3] (or, equivalently, a GNN with a _virtual node_[10]) in each round the nodes also obtain the aggregation of the states of all nodes in addition to the messages they receive from their neighbours. So the state update rule (4.A) becomes

\[\mathsf{agg}^{\prime}\Big{(}\Big{\{}\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!
only depends on the neighbourhood of \(v\). The _global_ bound (4.C) is simpler, but a bit weaker.

Proof.: Clearly, (4.B) implies (4.C), so we only have to prove the local bound (4.B). Let \(\mathsf{msg},\mathsf{agg},\mathsf{comb}\) be the message, aggregation, and combination functions of \(\mathfrak{L}\). Let \(\mathfrak{F}_{\mathsf{msg}}\) and \(\mathfrak{F}_{\mathsf{comb}}\) be FNNs computing \(\mathsf{msg}\) and \(\mathsf{comb}\), respectively, and let \(\gamma_{\mathsf{msg}}\coloneqq\gamma(\mathfrak{F}_{\mathsf{msg}})\) and \(\gamma_{\mathsf{comb}}\coloneqq\gamma(\mathfrak{F}_{\mathsf{comb}})\) be the constants of Lemma 2.5.

Let \(G\) be a graph, \(\mathbf{x}\in\mathcal{S}_{p}(G)\), and \(v\in V(G)\). Then for all \(w\in N_{G}(v)\) we have

Since for every multiset \(M\) we have \(\mathsf{agg}(M)\leq|M|\cdot m\), where \(m\) is the the maximum absolute value of the entries of \(M\), it follows that

Since \(\gamma_{\mathsf{msg}}\geq 1\), this implies

Hence

\[\big{\|}\widetilde{\mathfrak{L}}(G,x)(v)\big{\|}_{\infty} =\big{\|}\mathsf{comb}\big{(}\big{(}\mathbf{x}(v),\mathbf{x}(v)\big{)} \big{)}\big{\|}_{\infty}\] \[\leq\gamma_{\mathsf{comb}}\cdot\big{(}\gamma_{\mathsf{msg}} \big{(}\big{\|}x|_{N[v]}\big{\|}_{\infty}+1\big{)}\max\{\deg(v),1\}+1\big{)}\] \[\leq 2\gamma_{\mathsf{comb}}\gamma_{\mathsf{msg}}\cdot\big{(} \big{\|}x|_{N[v]}\big{\|}_{\infty}+1\big{)}\max\{\deg(v),1\}.\]

We let \(\gamma\coloneqq 2\gamma_{\mathsf{comb}}\gamma_{\mathsf{msg}}\). 

**Lemma 4.3**.: _Let \(\mathfrak{L}\) be a GNN layer of input dimension \(p\). Then there is a \(\lambda=\lambda(\mathfrak{L})\in\mathbb{N}_{>0}\) such that for all graphs \(G\), all signals \(x,\mathbf{x}^{\prime}\in\mathcal{S}_{p}(G)\), and all vertices \(v\in V(G)\) we have_

\[\big{|}\widetilde{\mathfrak{L}}(G,x)(v)-\widetilde{\mathfrak{L}} (G,\mathbf{x}^{\prime})(v)\big{\|}_{\infty} \leq\lambda\left\|\mathbf{x}|_{N[v]}-\mathbf{x}^{\prime}|_{N[v]}\right\| _{\infty}\max\{\deg(v),1\} \tag{4.D}\] \[\leq\lambda\left\|\mathbf{x}-\mathbf{x}^{\prime}\right\|_{\infty}|G|. \tag{4.E}\]

Proof.: Again, the local bound (4.D) implies the global bound (4.E). So we only need to prove (4.D). Let \(\mathsf{msg},\mathsf{agg},\mathsf{comb}\) be the message, aggregation, and combination functions of \(\mathfrak{L}\). Let \(\mathfrak{F}_{\mathsf{msg}}\) and \(\mathfrak{F}_{\mathsf{comb}}\) be FNNs computing \(\mathsf{msg}\) and \(\mathsf{comb}\), respectively, and let \(\lambda_{\mathsf{msg}}\coloneqq\lambda(\mathfrak{F}_{\mathsf{msg}})\) and \(\lambda_{\mathsf{comb}}\coloneqq\lambda(\mathfrak{F}_{\mathsf{comb}})\) be their Lipschitz constants (cf. Lemma 2.5).

Let \(G\) be a graph, \(\mathbf{x},\mathbf{x}^{\prime}\in\mathcal{S}_{p}(G)\), and \(\mathbf{y}\coloneqq\widetilde{\mathfrak{L}}(G,\mathbf{x})\), \(\mathbf{y}^{\prime}\coloneqq\widetilde{\mathfrak{L}}(G,\mathbf{x}^{\prime})\). Let \(v\in V(G)\). For all \(w\in N(v)\) we have

\[\big{|}\mathsf{msg}(x(v),\mathbf{x}(w))-\mathsf{msg}(\mathbf{x}^{\prime}(v),\mathbf{x}^{ \prime}(w))\big{|}_{\infty}\leq\lambda_{\mathsf{msg}}\big{|}(\mathbf{x}(v),\mathbf{x} (w))-(\mathbf{x}^{\prime}(v),\mathbf{x}^{\prime}(w))\big{|}_{\infty}\,.\]
Thus for

\[x(v) \coloneqq\mathsf{agg}\Big{(}\big{\{}\mathsf{msg}(x(v),x(w))\big{|} \,w\in N_{G}(v)\big{\}}\Big{)},\] \[x^{\prime}(v) \coloneqq\mathsf{agg}\Big{(}\big{\{}\mathsf{msg}(x^{\prime}(v),x^{ \prime}(w))\big{|}\,w\in N_{G}(v)\big{\}}\Big{)}\]

we have

\[\big{\|}x(v)-x^{\prime}(v)\big{\|}_{\infty}\leq\lambda_{\mathsf{msg}}\big{\|}x |_{N[v]}-x^{\prime}|_{N[v]}\big{\|}_{\infty}\deg(v). \tag{4.F}\]

It follows that

\[\big{\|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\! \big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|} \!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!\big{|}\!
_such that the following holds for all graphs \(G\) and assignments \(\omega\) over \(G\). Let \(\mathbf{x}\in\mathcal{S}_{p}(G)\) be the signal defined by_

\[\mathbf{x}(v)\coloneqq\left(\llparent{\mathbf{X}_{1}}\right)^{\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!
r-expression itadd of Lemma 3.33 to obtain guarded r-expressions \(\boldsymbol{\sigma}_{i}\), for \(i\in[r]\), such that for all \(v\in V(G)\) we have \[\langle\!\langle\boldsymbol{\sigma}_{i}\rangle\!\rangle^{(G,a)}\,(v)=\sum_{v^{ \prime}\in N(v)}\langle\!\langle\boldsymbol{\mu}_{i}\rangle\!\rangle^{(G,a)} \,(v,v^{\prime}).\] (5.E) Then we substitute the r-expressions \(\boldsymbol{\sigma}_{i}(x)\) for the variables \(\boldsymbol{Z}_{i}\) in the formulas \(\boldsymbol{\gamma}_{j}\) of Claim 2 and obtain the desired r-expressions \(\mathsf{l}\text{-}\mathsf{eval}_{j}(x)\) such that \[\Big{(}\,\langle\!\langle\mathsf{l}\text{-}\mathsf{eval}_{1} \rangle\!\rangle^{(G,a)}\,(v),\ldots,\langle\!\langle\mathsf{l}\text{-} \mathsf{eval}_{q}\rangle\!\rangle^{(G,a)}\,(v)\,\Big{)}\] \[\qquad=\mathsf{comb}\Big{(}\,\langle\!\langle\boldsymbol{X}_{1} \rangle\!\rangle^{(G,a)}\,(v),\ldots,\langle\!\langle\boldsymbol{X}_{p} \rangle\!\rangle^{(G,a)}\,(v),\langle\!\langle\boldsymbol{\sigma}_{1}\rangle \!\rangle^{(G,a)}\,(v),\ldots,\langle\!\langle\boldsymbol{\sigma}_{r}\rangle\! \rangle^{(G,a)}\,(v)\Big{)}\] \[\qquad=\mathpzc{z}(v).\] Thus in this case, the r-expressions \(\mathsf{l}\text{-}\mathsf{eval}_{j}(x)\) even define \(\mathpzc{z}\) exactly. Of course this implies that they satisfy (5.D). _Case 2:_agg = MAX. We can argue as in Case 1, using the r-expression \(\mathsf{max}\) of Lemma 3.33 instead of itadd. Again, we obtain r-expressions \(\mathsf{l}\text{-}\mathsf{eval}_{j}(x)\) that define \(\mathpzc{z}\) exactly. _Case 3:_agg = MEAN. The proof is similar to Case 1 and Case 2, but we need to be careful. We cannot define the mean of a family of numbers exactly, but only approximately, because of the division it involves. Exactly as in Case 1 we define r-expressions \(\boldsymbol{\sigma}_{i}(x)\) satisfying (5.E). Recall that \(\lambda\) is a Lipschitz constant for comb. Let \(\boldsymbol{\delta}(x)\coloneqq\#x^{\prime}.E(x,x^{\prime})\) be a term defining the degree of a vertex. Using Lemma 3.21 we can construct an r-expression \(\boldsymbol{\nu}_{i}\) such that \[\left|\frac{\langle\!\langle\boldsymbol{\sigma}_{i}\rangle\!\rangle^{(G,a)} \,(v)}{\langle\!\langle\boldsymbol{\delta}\rangle\!\rangle^{(G,a)}\,(v)}- \langle\!\langle\boldsymbol{\nu}_{i}\rangle\!\rangle^{(G,a)}\,(v)\right|<2^{-a \,(W)-\lambda}\leq\lambda^{-1}2^{-a\,(W)}\] if \(\langle\!\langle\boldsymbol{\delta}\rangle\!\rangle^{(G,a)}\,(v)=\deg(v)\neq 0\) and \(\langle\!\langle\boldsymbol{\nu}_{i}\rangle\!\rangle^{(G,a)}\,(v)=0\) otherwise. Thus, letting \[x(v) \coloneqq\text{MEAN}\big{(}\big{\{}\mathsf{msg}(x(v),x(v^{ \prime}))\,\big{|}\,v^{\prime}\in N(v)\big{\}}\big{)}\] \[=\begin{cases}0&\text{if }\deg(v)=0,\\ \big{(}\frac{\langle\!\langle\boldsymbol{\sigma}_{1}\rangle\!\rangle^{(G,a)} \,(v)}{\langle\!\langle\boldsymbol{\delta}\rangle\!\rangle^{(G,a)}\,(v)}, \ldots,\frac{\langle\!\langle\boldsymbol{\sigma}_{r}\rangle\!\rangle^{(G,a)} \,(v)}{\langle\!\langle\boldsymbol{\delta}\rangle\!\rangle^{(G,a)}\,(v)}\bigg{)} &\text{otherwise}\end{cases}\] we have \[\left\|x(v)-\Big{(}\,\langle\!\langle\boldsymbol{\nu}_{1}\rangle\!\rangle^{(G,a)}\,(v),\ldots,\langle\!\langle\boldsymbol{\nu}_{r}\rangle\!\rangle^{(G,a)} \,(v)\Big{)}\right\|_{\infty}\leq\lambda^{-1}2^{-a\,(W)}\] for all \(v\in V(G)\). By the Lipschitz continuity of comb, this implies \[\left\|\mathsf{comb}\big{(}x(v),x(v)\big{)}-\mathsf{comb}\Big{(}x(v),\langle\! \langle\boldsymbol{\nu}_{1}\rangle\!\rangle^{(G,a)}\,(v),\ldots,\langle\! \langle\boldsymbol{\nu}_{r}\rangle\!\rangle^{(G,a)}\,(v)\Big{)}\right\|_{ \infty}\leq 2^{-a\,(W)}.\] (5.F)
We substitute the r-expressions \(\mathbf{\nu}_{i}(x)\) for the variables \(\mathbf{Z}_{i}\) in the formulas \(\mathbf{\gamma}_{j}\) of Claim 2 and obtain r-expressions \(\mathsf{l}\mathsf{-eval}_{j}(x)\) such that

\[\Big{(}\,\langle\!\langle\mathsf{l}\mathsf{-eval}_{1}\rangle\! \rangle^{(G,a)}\,(v),\ldots,\langle\!\langle\mathsf{l}\mathsf{-eval}_{q} \rangle\!\rangle^{(G,a)}\,(v)\Big{)}\] \[\quad=\mathsf{comb}\Big{(}\,\langle\!\langle\mathbf{X}_{1}\rangle\! \rangle^{(G,a)}\,(v),\ldots,\langle\!\langle\mathbf{X}_{p}\rangle\!\rangle^{(G,a)} \,(v),\langle\!\langle\mathbf{\nu}_{1}\rangle\!\rangle^{(G,a)}\,(v),\ldots,\langle\! \langle\mathbf{\nu}_{r}\rangle\!\rangle^{(G,a)}\,(v)\Big{)}\] \[\quad=\mathsf{comb}\Big{(}\,x(v),\langle\!\langle\mathbf{\nu}_{1} \rangle\!\rangle^{(G,a)}\,(v),\ldots,\langle\!\langle\mathbf{\nu}_{r}\rangle\! \rangle^{(G,a)}\,(v)\Big{)}.\]

Since \(\mathbf{\nu}(v)=\mathsf{comb}\big{(}\,x(v),\mathbf{\varkappa}(v)\big{)}\), the assertion (5.D) follows from (5.F). 

Proof of Theorem 5.1.: We fix a graph \(G\) and assignment \(\mathbf{\omega}\) over \(G\) for the presentation of the proof; as usual the formulas we shall define will not depend on this graph and assignment. Let \(x\in\mathcal{S}_{p}(G)\) be the signal defined in (5.A).

Suppose that \(\mathfrak{N}=(\mathfrak{L}^{(1)},\ldots,\mathfrak{L}^{(d)})\). Let \(p^{(i-1)}\) be the input dimension of \(\mathfrak{L}^{(i)}\), and let \(p^{(i)}\) be the output dimension. Then \(p=p^{(0)}\) and \(q=p^{(d)}\). Moreover, let \(\mathbf{x}^{(0)}\coloneqq\mathbf{x}\) and \(\mathbf{x}^{(i)}\coloneqq\widetilde{\mathfrak{L}}^{(i)}(G,\mathbf{x}^{(i-1)})\) for \(i\in[d]\). Note that \(\mathbf{x}^{(d)}=\mathbf{\varkappa}\).

For every \(i\in[d]\), let \(\lambda^{(i)}\coloneqq\lambda(\mathfrak{L}^{(i)})\) be the constant of Lemma 4.3. We inductively define a sequence of \(\mathsf{GFO}+\mathsf{C}\)-terms \(\mathsf{err}^{(i)}(x)\), which will give us the desired error bounds. We let \(\mathsf{err}^{(d)}(x)\coloneqq W(x)\). To define \(\mathsf{err}^{(i)}(x)\) for \(0\leq i<d\), we first note that by Lemma 3.32, for every \(\mathsf{GFO}+\mathsf{C}\)-term \(\theta(x)\) there is a \(\mathsf{GFO}+\mathsf{C}\)-term \(\mathsf{maxN}_{\theta}(x)\) such that for every \(v\in V(G)\) we have

\[[\![\mathsf{maxN}_{\theta}]\!]^{(G,a)}\,(v)=\max\Big{\{}\,[\![\theta]\!]^{(G,a )}\,(w)\,\Big{|}\,w\in N_{G}[v]\Big{\}}.\]

We let \(\mathsf{dg}(x)\coloneqq(\#x^{\prime}.E(x,x^{\prime}))+1\) and

\[\mathsf{err}^{(i)}(x)\coloneqq\mathsf{maxN}_{\mathsf{err}^{(i+1)}}(x)+\lambda ^{(i+1)}\cdot\mathsf{maxN}_{\mathsf{dg}}(x)+1.\]

Letting

\[k^{(i)}(v)\coloneqq\left[\!\left|\mathsf{err}^{(i)}\right|\!\right]^{(G,a)}(v),\]

for every \(v\in V(G)\) and \(0\leq i<d\), we have

\[k^{(i)}(v)=\max\big{\{}k^{(i+1)}(w)\,\big{|}\,w\in N_{G}[v]\big{\}}+\lambda^{( i+1)}\max\big{\{}\,\deg_{G}(w)+1\,\big{|}\,w\in N_{G}[v]\big{\}}+1. \tag{5.G}\]

Furthermore, \(k^{(d)}(v)=\mathbf{\omega}(W)(v)\).

Now for \(i\in[d]\) and \(j\in[p^{(i)}]\) we shall define guarded r-expressions \(\mathbf{\rho}_{j}^{(i)}(x)\) such that for all \(v\in V(G)\), with

\[\mathbf{\varkappa}^{(i)}(v)\coloneqq\Big{(}\,\langle\!\langle\mathbf{\rho}_{1}^{(i)} \rangle\!\rangle\,(v),\ldots,\langle\!\langle\mathbf{\rho}_{p^{(i)}}^{(i)}\rangle \!\rangle\,(v)\Big{)}\]

we have

\[\left\|\mathbf{x}^{(i)}(v)-\mathbf{\varkappa}^{(i)}(v)\right\|_{\infty}\leq 2^{-k^{(i)}(v)}. \tag{5.H}\]
For \(i=d\) and with \(\mathsf{gnn}\text{-}\mathsf{eval}_{j}\coloneqq\boldsymbol{\rho}_{j}^{(d)}\), this implies (5.B) and hence the assertion of the theorem.

To define \(\boldsymbol{\rho}_{j}^{(1)}(x)\), we apply Lemma 5.2 to the first layer \(\mathfrak{L}^{(1)}\) and substitute \(\mathsf{err}^{(0)}\) for \(W\) in the resulting-expression. Then (5.H) for \(i=1\) follows directly from Lemma 5.2 and the fact that \(k^{(0)}(v)\geq k^{(1)}(v)\) for all \(v\).

For the inductive step, let \(2\leq i\leq d\) and suppose that we have defined \(\boldsymbol{\rho}_{j}^{(i-1)}(x)\) for all \(j\in[p^{(i-1)}]\). To define \(\boldsymbol{\rho}_{j}^{(i)}(x)\), we apply Lemma 5.2 to the \(i\)th layer \(\mathfrak{L}^{(i)}\) and substitute \(\boldsymbol{\rho}_{1}^{(i-1)},\ldots,\boldsymbol{\rho}_{p^{(i-1)}}^{(i-1)}\) for \(\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{p^{(i-1)}}\) and \(\mathsf{err}^{(i-1)}\) for \(W\) in the resulting r-expression. Then by Lemma 5.2, for all \(v\in V(G)\) we have

\[\left\|\widehat{\mathfrak{L}}^{(i)}(G,\varkappa^{(i-1)})(v)-\varkappa^{(i)}(v )\right\|_{\infty}\leq 2^{-k^{(i-1)}(v)}. \tag{5.I}\]

Moreover, by Lemma 4.3 applied to \(\mathfrak{L}^{(i)}\) and \(x\coloneqq\varkappa^{(i-1)}\), \(\varkappa^{\prime}\coloneqq\varkappa^{(i-1)}\) we have

\[\left\|\varkappa^{(i)}(v)-\widehat{\mathfrak{L}}^{(i)}(G, \varkappa^{(i-1)})(v)\right\|_{\infty}\] \[\leq\lambda^{(i)}\max\Big{\{}\left\|\varkappa^{(i-1)}(w)- \varkappa^{(i-1)}(w)\right\|_{\infty}\Bigm{|}w\in N_{G}[v]\Big{\}}\big{(} \deg_{G}(v)+1\big{)}\] \[\leq\lambda^{(i)}\max\Big{\{}2^{-k^{(i-1)}(w)}\Bigm{|}w\in N_{G} [v]\Big{\}}\big{(}\deg_{G}(v)+1\big{)}\] \[\leq\max\Big{\{}2^{-k^{(i-1)}(w)+\lambda^{(i)}(\cdot\deg_{G}(v)+1 )}\Bigm{|}w\in N_{G}[v]\Big{\}}.\]

We choose a \(w\in N_{G}[v]\) minimising \(k^{(i-1)}(w)\). Then

\[\left\|\varkappa^{(i)}(v)-\widehat{\mathfrak{L}}^{(i)}(G,\varkappa^{(i-1)})( v)\right\|\leq 2^{-(k^{(i-1)}(w)-\lambda^{(i)}(\cdot\deg_{G}(v)+1))}. \tag{5.J}\]

Combining (5.I) and (5.J) by the triangle inequality, we get

\[\left\|\varkappa^{(i)}(v)-\varkappa^{(i)}(v)\right\|\leq 2^{-k^{(i-1)}(v)}+2^{- (k^{(i-1)}(w)-\lambda^{(i)}(\cdot\deg_{G}(v)+1))}.\]

Observe that \(k^{(i-1)}(v)\geq k^{(i)}(v)+1\) and

\[k^{(i-1)}(w)-\lambda^{(i)}\cdot(\deg_{G}(v)+1)\] \[=\max\big{\{}k^{(i)}(v^{\prime})\bigm{|}v^{\prime}\in N_{G}[w] \big{\}}+\lambda^{(i)}\max\big{\{}\deg_{G}(v^{\prime})+1\bigm{|}v^{\prime}\in N _{G}[w]\big{\}}+1\] \[\qquad\qquad-\lambda^{(i)}\cdot(\deg_{G}(v)+1)\] \[\geq k^{(i)}(v)+\lambda^{(i)}\big{(}\deg_{G}(v)+1\big{)}+1- \lambda^{(i)}\cdot(\deg_{G}(v)+1)\qquad\qquad\text{ since }v\in N_{G}[w]\] \[=k^{(i)}(v)+1.\]

Thus

\[\left\|\varkappa^{(i)}(v)-\varkappa^{(i)}(v)\right\|\leq 2^{-k^{(i)}(v)-1}+2^{- k^{(i)}(v)-1}=2^{-k^{(i)}(v)},\]

which proves (5.H) and hence the theorem.
Since logics define queries, when comparing the expressiveness of graph neural networks with that of logics, it is best to focus on queries. Recall from Section 2.5 that we identified the \(\ell\)-labelled graphs and graphs with an \(\ell\)-dimensional Boolean signal. A unary query on the class \(\mathscr{SG}_{\ell}^{\mathrm{bool}}\) is an equivariant signal transformations from \(\mathscr{SG}_{\ell}^{\mathrm{bool}}\) to \(\mathscr{SG}_{1}^{\mathrm{bool}}\).

We say that a GNN \(\mathfrak{N}\)_computes_ a unary query \(\mathcal{G}:\mathscr{SG}_{\ell}^{\mathrm{bool}}\to\mathscr{SG}_{1}^{\mathrm{bool}}\) if for all \((G,\mathcal{G})\in\mathscr{SG}_{\ell}^{\mathrm{bool}}\) and all \(v\in V(G)\) it holds that

\[\begin{cases}\widetilde{\mathfrak{N}}(G,\mathcal{G})(v)\geq\frac{2}{3}&\text{ if }\mathcal{G}(G,\mathcal{G})(v)=1,\\ \widetilde{\mathfrak{N}}(G,\mathcal{G})(v)\leq\frac{1}{3}&\text{if }\mathcal{G}(G, \mathcal{G})(v)=0.\end{cases} \tag{5.K}\]

Note that if we allow \(\mathrm{lsig}\) or \(\mathrm{relu}\) activations, we can replace the \(\geq\frac{2}{3}\) and \(\leq\frac{1}{3}\) in (5.K) by \(=1\) and \(=0\) and thus require \(\widetilde{\mathfrak{N}}(G,\mathcal{G})(v)=\mathcal{G}(G,\mathcal{G})(v)\). We simply apply the transformation \(\mathrm{lsig}(3x-1)\) to the output. It maps the interval \((-\infty,\frac{1}{3}]\) to \(0\) and the interval \([\frac{2}{3},\infty)\) to \(1\). With other activations such as the logistic function, this is not possible, which is why we chose our more flexible definition.

**Corollary 5.3**.: _Every unary query on \(\mathscr{GS}_{\ell}\) that is computable by a rational piecewise linear GNN is definable in \(\mathsf{GFO}+\mathsf{C}\)._

Note that this Corollary is Theorem 1.4 stated in the introduction.

The reader may wonder if the converse of the previous corollary holds, that is, if every query definable in \(\mathsf{GFO}+\mathsf{C}\) is computable by a rational piecewise linear GNN. It is not; we refer the reader to Remark 7.6.

_Remark 5.4_.: As mentioned earlier, there are versions of the theorem for all the extensions of basic GNNs that we discussed in Section 4. In particular, there is a version for graph level functions and the logic \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\), and also for GNNs with global readout and \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\).

## 6 The Non-Uniform Case: GNNs with Arbitrary Weights and Families of GNNs

Now we consider the general case where the weights in the neural networks are arbitrary real numbers. We also drop the assumption that the activation functions be piecewise linear, only requiring rpl approximability. The price we pay is a non-uniformity on the side of the logic or the Boolean circuits and a slightly weaker approximation bound as well as a boundedness assumption on the input signal.

**Theorem 6.1**.: _Let \(\mathfrak{N}\) be an rpl-approximable GNN of input dimension \(p\) and output dimension \(q\). Let \(\mathbf{X}_{1},\ldots,\mathbf{X}_{p}\) be r-schemas of type \(\mathsf{v}\to\mathsf{r}\), and let \(W,W^{\prime}\) be function variables of type \(\varnothing\to\mathsf{n}\)._

_Then there are a r-expressions \(\mathsf{gnn}\)-\(\mathsf{eval}_{1}(x)\), \(\ldots\), \(\mathsf{gnn}\)-\(\mathsf{eval}_{q}(x)\) in \(\mathsf{GFO}+\mathsf{C}_{\mathrm{nu}}\) such that the following holds for all graphs \(G\) and assignments \(\mathscr{L}\) over \(G\). Let \(\mathbf{x}\in\mathscr{S}_{p}(G)\) be the signal defined by_

\[\mathbf{x}(v)\coloneqq\Big{(}\langle\!\langle\mathbf{X}_{1}\rangle\!\rangle^{(G, \mathscr{L})}\,(v),\ldots,\langle\!\langle\mathbf{X}_{p}\rangle\!\rangle^{(G, \mathscr{L})}\,(v)\Big{)}, \tag{6.A}\]
_and let \(\boldsymbol{y}=\widehat{\mathfrak{N}}(G,\boldsymbol{x})\). Assume that \(\left\|\boldsymbol{x}\right\|_{\infty}\leq\omega(W)\) and that \(\omega(W^{\prime})\neq 0\). Then for all \(v\in V(G)\),_

\[\left\|\boldsymbol{y}(v)-\left(\,\langle\!\langle\mathsf{gnn-eval}_{1}\rangle \!\rangle^{(G,e)}\,(v),\ldots,\langle\!\langle\mathsf{gnn-eval}_{q}\rangle\! \rangle^{(G,e)}\,(v)\right)\right\|_{\infty}\leq\frac{1}{\omega(W^{\prime})} \tag{6.B}\]

Let us comment on the role of the two \(0\)-ary functions (that is, constants) \(W,W^{\prime}\). We introduce them to add flexibility in the bounds. Their values depend on the assignment \(\omega\), which means that we can freely choose them. For example, we can let \(\omega(W)=\omega(W^{\prime})=n\coloneqq|G|\). Then we get an approximation error of \(1/n\) for input signals bounded by \(n\). Or we could let \(\omega(W)=1\) and \(\omega(W^{\prime})=100\). Then we get an approximation error of \(1\%\) for Boolean input signals.

Since we move to a non-uniform regime anyway, to obtain the most general results we may as well go all the way to a non-uniform GNN model where we have different GNNs for every size of the input graphs.

We need additional terminology. We define the _bitsize_\(\operatorname{bsize}(\mathfrak{F})\) of a rational piecewise linear FNN \(\mathfrak{F}\) to be the sum of the bitsizes of its skeleton, all its weights and biases, and all its activations. We define the _weight_ of an arbitrary FNN \(\mathfrak{F}=(V,E,(\mathfrak{a}_{v})_{v\in V},\boldsymbol{w},\boldsymbol{b})\) to be

\[\operatorname{wt}(\mathfrak{F})\coloneqq|V|+\left\|\boldsymbol{w}\right\|_{ \infty}+\left\|\boldsymbol{b}\right\|_{\infty}+\max_{v\in V}\left(\lambda( \mathfrak{a}_{v})+\mathfrak{a}_{v}(0)\right)\]

Here \(\lambda(\mathfrak{a}_{v})\) denotes the minimal Lipschitz constant of the Lipschitz continuous function \(\mathfrak{a}_{v}\). The _size_\(\operatorname{size}(\mathfrak{F})\) of a rational piecewise linear FNN \(\mathfrak{F}\) is the maximum of its bitsize and its weight. The _depth_\(\operatorname{dp}(\mathfrak{F})\) of an FNN \(\mathfrak{F}\) is the depth of its skeleton, that is, the length of a longest path from an input node to an output node of \(\mathfrak{F}\).

The _weight_\(\operatorname{wt}(\mathfrak{N})\) of a GNN \(\mathfrak{N}\) is the sum of the weights of the FNNs for the message and combination functions of all layers of \(\mathfrak{N}\). The _bitsize_\(\operatorname{bsize}(\mathfrak{N})\) and the _size_\(\operatorname{size}(\mathfrak{N})\) of a rational piecewise linear GNN \(\mathfrak{N}\) is the sum of the (bit)sizes of all its FNNs. The _skeleton_ of a GNN \(\mathfrak{N}\) consists of the directed acyclic graphs underlying the FNNs for the message and combination functions of all layers of \(\mathfrak{N}\).Thus if two GNNs have the same skeleton they have the same number of layers and the same input and output dimensions on all layers, but they may have different activation functions and different weights. The _depth_\(\operatorname{dp}(\mathfrak{N})\) of a GNN \(\mathfrak{N}\) is the number of layers of \(\mathfrak{N}\) times the maximum depth of all its FNNs.

Let \(\mathcal{N}=(\mathfrak{N}^{(n)})_{n\in\mathbb{N}}\) be a family of GNNs. Suppose that the input dimension of \(\mathfrak{N}^{(n)}\) is \(p^{(n)}\) and the output dimension is \(q^{(n)}\). It will be convenient to call \((p^{(n)})_{n\in\mathbb{N}}\) the _input dimension_ of \(\mathcal{N}\) and \((q^{(n)})_{n\in\mathbb{N}}\) the _output dimension_. Then for every graph \(G\) of order \(n\) and every \(\boldsymbol{x}\in\mathcal{S}_{p(n)}(G)\) we let \(\mathcal{N}(G,\boldsymbol{x})\coloneqq\mathfrak{N}^{(n)}(G,\boldsymbol{x})\) and \(\widetilde{\mathcal{N}}(G,\boldsymbol{x})\coloneqq\widehat{\mathfrak{N}}^{( n)}(G,\boldsymbol{x})\). Thus \(\mathcal{N}\) computes a generalised form of signal transformation where the input and output dimension depend on the order of the input graph.

We say that \(\mathcal{N}\) is of _polynomial weight_ if there is a polynomial \(\pi(X)\) such that \(\operatorname{wt}(\mathfrak{N}^{(n)})\leq\pi(n)\) for all \(n\). _Polynomial (bit)size_ is defined similarly. The family \(\mathcal{N}\) is of _bounded depth_ if there is a \(d\in\mathbb{N}\) such that \(\operatorname{dp}(\mathfrak{N}^{(n)})\leq d\) for all \(n\). The family \(\mathcal{N}\) is _rpl approximable_ if there is a polynomial \(\pi^{\prime}(X,Y)\) such that for all \(n\in\mathbb{N}_{>0}\) and all
\(\varepsilon>0\), every activation function of \(\mathfrak{N}^{(n)}\) is \(\varepsilon\)-approximable by a rational piecewise linear function of bitsize at most \(\pi^{\prime}(\varepsilon^{-1},n)\).

**Theorem 6.2**.: _Let \(\mathscr{N}\) be an rpl-approximable polynomial-weight, bounded-depth family of GNNs of input dimension \((p^{(n)})_{n\in\mathbb{N}}\) and output dimension \((q^{(n)})_{n\in\mathbb{N}}\). Let \(\mathbf{X}\) be an r-schema of type \(\mathtt{vn}\to\mathtt{r}\), and let \(W,W^{\prime}\) be function variables of type \(\varnothing\to\mathtt{n}\)._

_Then there is an r-expression \(\mathtt{gnn}\text{-}\mathsf{eval}(x,y)\) in \(\mathsf{GFO}+\mathsf{C}_{\mathtt{n}\mathtt{u}}\) such that the following holds for all graphs \(G\) and assignments \(\mathpzc{a}\) over \(G\). Let \(n\coloneqq|G|\), and let \(\mathbf{x}\in\mathscr{S}_{p^{(n)}}(G)\) be the signal defined by_

\[\mathbf{x}(v)\coloneqq\Big{(}\left\langle\!\left\langle\mathbf{X}\right\rangle\! \right\rangle^{(G,a)}(v,0),\ldots,\left\langle\!\left\langle\mathbf{X}\right\rangle \!\right\rangle^{(G,a)}(v,p^{(n)}-1)\Big{)}. \tag{6.C}\]

_Assume that \(\left\|\!\left\langle\mathbf{x}\right\rangle\!\right\|_{\infty}\leq\mathpzc{a}(W)\) and that \(\mathpzc{a}(W^{\prime})\neq 0\). Let \(\mathbf{\chi}=\widetilde{\mathscr{N}}(G,\mathbf{x})\in\mathscr{S}_{q^{(n)}}(G)\). Then for all \(v\in V(G)\),_

\[\left\|\!\left\langle\mathpzc{c}(v)-\big{(}\left\langle\!\left\langle\mathtt{ gnn}\text{-}\mathsf{eval}\right\rangle\!\right\rangle^{(G,a)}(v,0),\ldots,\left\langle\! \left\langle\mathtt{gnn}\text{-}\mathsf{eval}\right\rangle\!\right\rangle^{( G,a)}(v,q^{(n)}-1)\big{)}\right\|_{\infty}\leq\frac{1}{\mathpzc{a}(W^{\prime})} \tag{6.D}\]

Observe that Theorem 6.2 implies Theorem 6.1, because we can simply let \(\mathscr{N}\) be the family consisting of the same GNN for every \(n\). So we only need to prove Theorem 6.2. The basic idea of the proof is simple. We exploit the continuity of the functions computed by FNNs and GNNs not only in terms of the input signals but also in terms of the weights and the biases. This allows us to approximate the functions computed by GNNs with arbitrary real weights by GNNs with rational weights. However, the bitsize of the rationals we need to get a sufficiently precise approximation depends on the size of the input graph, and this leads to the non-uniformity.

Before we delve into the proof, let us state one important corollary. Extending the definition for single GNNs in the obvious way, we say that a family \(\mathscr{N}=(\mathfrak{N}^{(n)})_{n\in\mathbb{N}}\) of GNNs _computes_ a unary query \(\mathscr{O}:\mathscr{SG}_{p}^{\mathrm{bool}}\to\mathscr{SG}_{1}^{\mathrm{bool}}\) on \(p\)-labelled graphs if for all \(n\in\mathbb{N}\), all \((G,\mathpzc{a})\in\mathscr{SG}_{p}^{\mathrm{bool}}\) with \(n=|G|\), and all \(v\in V(G)\) it holds that

\[\begin{cases}\widetilde{\mathfrak{N}}^{(n)}(G,\mathpzc{a})(v)\geq\frac{2}{3}& \text{if }\mathscr{O}(G,\mathpzc{a})(v)=1,\\ \widetilde{\mathfrak{N}}^{(n)}(G,\mathpzc{a})(v)\leq\frac{1}{3}&\text{if } \mathscr{O}(G,\mathpzc{a})(v)=0.\end{cases}\]

**Corollary 6.3**.: _Every unary query on \(\mathscr{SG}_{p}^{\mathrm{bool}}\) that is computable by an rpl-approximable polynomial-weight bounded-depth family of GNNs is definable in \(\mathsf{GFO}+\mathsf{C}_{\mathrm{m}\mathtt{u}}\)._

_Remark 6.4_.: The exact analogues of Theorems 6.1 and 6.2 hold for GNNs with global readout and the logic \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\), with only small modifications of the proof.

### Bounds and Approximations for FNNs

In this section, we shall prove that we can approximate rpl approximable FNNs by rational piecewise linear FNNs whose size is bounded in terms of the approximation ratio. For this, we first need to establish bounds on the Lipschitz constant and growth of an FNN in terms of its structure, its activation functions, and its parameters.
Throughout this section, we let \(\mathfrak{A}=\big{(}V,E,(\mathfrak{a}_{v})_{v\in V}\big{)}\) be an FNN architecture of input dimension \(p\) and output dimension \(q\). We let \(d\) be the depth and the maximum in-degree of the directed graph \((V,E)\). Without loss of generality, we assume \(\Delta\geq 1\) and thus \(d\geq 1\). If \(\Delta=0\), we simply add a dummy edge of weight \(0\) to the network. Moreover, we let \(\lambda\in\mathbb{N}_{>0}\) be a Lipschitz constant for all activation function \(\mathfrak{a}_{v}\) for \(v\in V\), and we let

\[\mu=\max\{v\in V\mid\mathfrak{a}_{v}(0)\}.\]

For vectors \(\boldsymbol{x}\in\mathbb{R}^{p}\), \(\boldsymbol{w}\in\mathbb{R}^{E}\), \(\boldsymbol{b}\in\mathbb{R}^{V}\), we assume that \(\boldsymbol{x}=(x_{1},\ldots,x_{p})\), \(\boldsymbol{w}=(w_{e})_{e\in E}\), and \(\boldsymbol{b}=(b_{v})_{v\in V}\).

In the first two lemmas we analyse how the growth and variation of the functions \(g_{\boldsymbol{A},v}\mathfrak{A}(\boldsymbol{x},\boldsymbol{w},\boldsymbol{b})\) and \(\mathfrak{A}(\boldsymbol{x},\boldsymbol{w},\boldsymbol{b})\) depend on the constants \(d,\Delta,\lambda,\mu\) and \(\left\|\boldsymbol{x}\right\|_{\infty},\left\|\boldsymbol{w}\right\|_{\infty},\left\|\boldsymbol{b}\right\|_{\infty}\) (more precisely than in Lemma 2.5).

**Lemma 6.5**.: _Let \(\gamma\coloneqq 2\Delta\lambda\max\{\lambda,\mu\}\). Then for all \(\boldsymbol{x}\in\mathbb{R}^{p}\), \(\boldsymbol{b}\in\mathbb{R}^{V}\), and \(\boldsymbol{w}\in\mathbb{R}^{E}\), and all \(v\in V\) of depth \(t\) we have_

\[\big{|}g_{\mathfrak{A},v}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})\big{|} \leq\gamma^{t}(\left\|\boldsymbol{w}\right\|_{\infty}+1)^{t}\big{(}\left\| \boldsymbol{x}\right\|_{\infty}+\left\|\boldsymbol{b}\right\|_{\infty}+1). \tag{6.E}\]

Proof.: Note that for all \(x\in\mathbb{R}\) we have

\[\left|\mathfrak{a}_{v}(x)\right|\leq\lambda|x|+\mu. \tag{6.F}\]

For all input nodes \(X_{i}\) we have

\[\big{|}g_{\mathfrak{A},X_{i}}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w}) \big{|}=\left|x_{i}\right|\leq\left\|\boldsymbol{x}\right\|_{\infty}. \tag{6.G}\]

This implies (6.E) for \(t=0\).

_Claim 1._ For all nodes \(v\in V\) of depth \(t\geq 1\) we have

\[\big{|}g_{\mathfrak{A},v}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})\big{|} \leq\big{(}\Delta\lambda\left\|\boldsymbol{w}\right\|_{\infty}\big{)}^{t} \left\|\boldsymbol{x}\right\|_{\infty}+\sum_{s=0}^{t-1}\big{(}\Delta\lambda \left\|\boldsymbol{w}\right\|_{\infty}\big{)}^{s}\big{(}\lambda\left\| \boldsymbol{b}\right\|_{\infty}+\mu\big{)}. \tag{6.H}\]

Proof.: We prove (6.H) by induction on \(t\geq 1\). Suppose that \(v\in V\) is a node of depth \(t\), and let \(v_{1},\ldots,v_{k}\) be its in-neighbours. Let \(b\coloneqq b_{v}\) and \(w_{i}\coloneqq w_{v_{i}v}\) for \(i\in[k]\). Moreover, let \(y_{i}\coloneqq g_{\mathfrak{A},v_{i}}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})\) and \(\boldsymbol{y}=(y_{1},\ldots,y_{k})\). If \(t=1\), by (6.G) we have

\[\left\|\boldsymbol{y}\right\|_{\infty}\leq\left\|\boldsymbol{x}\right\|_{\infty} \tag{6.I}\]

If \(t>1\), by the induction hypothesis we have

\[\left\|\boldsymbol{y}\right\|_{\infty}\leq\big{(}\Delta\lambda\left\| \boldsymbol{w}\right\|_{\infty}\big{)}^{t-1}\left\|\boldsymbol{x}\right\|_{ \infty}+\sum_{s=0}^{t-2}\big{(}\Delta\lambda\left\|\boldsymbol{w}\right\|_{ \infty}\big{)}^{s}\big{(}\lambda\left\|\boldsymbol{b}\right\|_{\infty}+\mu \big{)}. \tag{6.J}\]

Thus

\[\big{|}g_{\mathfrak{A},v}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})\big{|} =\Big{|}\mathfrak{a}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y_{i}\Big{)}\Big{|}\]
\[\leq\left|\lambda\middle|b+\sum_{i=1}^{k}w_{i}y_{i}\middle|+\mu\right| \text{by (\ref{eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eqeq:eq:eqeq:eq:eq:eq:eq:eq:eqeq:eq:eqeq:eq:eqeq:eq:eq:eqeq:eqeq:eq:eqeq:eqeq:eq:eqeq:eq:eqeq:eqeq:eqeq:eqeq:eqeqeq:eqeq:eqeq:eqeq:eqeqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeqeq:eqeq:eqeqeq:eqeq:eqeqeqeq:eqeq:eqeqeq:eqeqeq:eqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeqeq:eqeqeqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeqeq:eqeqeq:eqeqeqeq:eqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeqeqeqeqeq:eqeqeqeqeqeqeqeqeqeqeqeqeqeqeq:
\(\mathbf{y}=(y_{1},\ldots,y_{k})\) and \(\mathbf{y}^{\prime}=(y^{\prime}_{1},\ldots,y^{\prime}_{k})\) with \(y_{i}\coloneqq g_{\mathfrak{A},v_{i}}(\mathbf{x},\mathbf{b},\mathbf{w})\) and \(y^{\prime}_{i}\coloneqq g_{\mathfrak{A},v_{i}}(\mathbf{x}^{\prime},\mathbf{b},\mathbf{w})\). Then

\[\big{|}g_{\mathfrak{A},v}(\mathbf{x},\mathbf{b},\mathbf{w})-g_{\mathfrak{A},v} (\mathbf{x},\mathbf{b}^{\prime},\mathbf{w}^{\prime})\big{|} =\Big{|}\mathfrak{a}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y_{i}\Big{)}- \mathfrak{a}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y^{\prime}_{i}\Big{)}\Big{|}\] \[\leq\lambda\Big{(}\sum_{i=1}^{k}w_{i}|y_{i}-y^{\prime}_{i}|\Big{)}\] \[\leq\lambda\Delta\left\|\mathbf{w}\right\|_{\infty}\left\|\mathbf{y}-\bm {y}^{\prime}\right\|_{\infty}.\]

Since by the induction hypothesis we have \(\left\|\mathbf{y}-\mathbf{y}^{\prime}\right\|_{\infty}\leq(\lambda\Delta\left\|\mathbf{ w}\right\|_{\infty})^{t-1}\left\|\mathbf{x}-\mathbf{x}^{\prime}\right\|_{\infty}\), the assertion (6.K) follows. 

**Lemma 6.7**.: _Let \(\nu\coloneqq(4\Delta\lambda\gamma)^{d}\), where \(\gamma\) is the constant of Lemma 6.5. Then for all \(\varepsilon\in\mathbb{R}\), \(\mathbf{x}\in\mathbb{R}^{p}\), \(\mathbf{b},\mathbf{b}^{\prime}\in\mathbb{R}^{V}\), and \(\mathbf{w},\mathbf{w}^{\prime}\in\mathbb{R}^{E}\) with_

\[0\leq\max\big{\{}\|\mathbf{b}-\mathbf{b}^{\prime}\|_{\infty},\|\mathbf{w}-\mathbf{w}^{\prime }\|_{\infty}\big{\}}\leq\varepsilon\leq 1 \tag{6.L}\]

_we have_

\[\big{\|}\mathfrak{A}(\mathbf{x},\mathbf{b},\mathbf{w})-\mathfrak{A}(\mathbf{x},\mathbf{b}^{\prime},\mathbf{w}^{\prime})\big{\|}_{\infty}\leq\nu\big{(}\left\|\mathbf{w}\right\|_{\infty }+1\big{)}^{d}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\|\mathbf{b}\right\|_{ \infty}+1\big{)}\varepsilon.\]

Proof.: Let \(\mathbf{x}\in\mathbb{R}^{p}\) and \(\varepsilon\in[0,1]\), \(\mathbf{b},\mathbf{b}^{\prime}\in\mathbb{R}^{V}\), \(\mathbf{w},\mathbf{w}^{\prime}\in\mathbb{R}^{E}\) such that (6.L) holds.

We shall prove by induction on \(t\) that for all nodes \(v\in V\) of depth \(t\) we have

\[\big{|}g_{\mathfrak{A},v}(\mathbf{x},\mathbf{b},\mathbf{w})-g_{\mathfrak{A},v}(\mathbf{x},\bm {b}^{\prime},\mathbf{w}^{\prime})\big{|}\leq\big{(}4\Delta\lambda\gamma(\left\|\bm {w}\right\|_{\infty}+1)\big{)}^{t}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\| \mathbf{b}\right\|_{\infty}+1\big{)}\varepsilon. \tag{6.M}\]

Applied to the output nodes \(v=Y_{i}\) of depth \(\leq d\), this yields the assertion of the lemma.

Nodes of depth \(t=0\) are input nodes, and we have

\[\big{|}g_{\mathfrak{A},X_{i}}(\mathbf{x},\mathbf{b},\mathbf{w})-g_{\mathfrak{A},X_{i}}( \mathbf{x},\mathbf{b}^{\prime},\mathbf{w}^{\prime})\big{|}=|x_{i}-x_{i}|=0. \tag{6.N}\]

For the inductive step, let \(v\in V\) be a node of depth \(t>0\), and let \(v_{1},\ldots,v_{k}\) be its in-neighbours. Let \(b\coloneqq b_{v},b^{\prime}\coloneqq b^{\prime}_{v}\) and \(w_{i}\coloneqq w_{v_{i}v},w^{\prime}_{i}\coloneqq w^{\prime}_{v_{i}v}\) for \(i\in[k]\). Moreover, let \(\mathbf{y}=(y_{1},\ldots,y_{k})\) and \(\mathbf{y}^{\prime}=(y^{\prime}_{1},\ldots,y^{\prime}_{k})\) with \(y_{i}\coloneqq g_{\mathfrak{A},v_{i}}(\mathbf{x},\mathbf{b},\mathbf{w})\) and \(y^{\prime}_{i}\coloneqq g_{\mathfrak{A},v_{i}}(\mathbf{x},\mathbf{b}^{\prime},\mathbf{w}^ {\prime})\).

_Claim 1_.: \[\big{|}g_{\mathfrak{A},v}(\mathbf{x},\mathbf{b},\mathbf{w})-g_{\mathfrak{A},v}(\mathbf{x},\bm {b}^{\prime},\mathbf{w}^{\prime})\big{|}\leq\Delta\lambda\left\|\mathbf{y}-\mathbf{y}^{ \prime}\right\|_{\infty}\left\|\mathbf{w}\right\|_{\infty}+\Delta\lambda\big{(} \left\|\mathbf{y}\right\|_{\infty}+\left\|\mathbf{y}-\mathbf{y}^{\prime}\right\|_{\infty}+1 \big{)}\varepsilon\]

Proof.: By the definition of \(g_{\mathfrak{A},v}\) and the Lipschitz continuity of the activation functions we have

\[\big{|}g_{\mathfrak{A},v}(\mathbf{x},\mathbf{b},\mathbf{w})-g_{\mathfrak{A},v }(\mathbf{x},\mathbf{b}^{\prime},\mathbf{w}^{\prime})\big{|} =\Big{|}\mathfrak{a}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y_{i}\Big{)} -\mathfrak{a}_{v}\Big{(}b^{\prime}+\sum_{i=1}^{k}w^{\prime}_{i}y^{\prime}_{i} \Big{)}\Big{|}\] \[\leq\lambda\cdot\Big{(}|b-b^{\prime}|+\sum_{i=1}^{k}|w_{i}y_{i}-w^ {\prime}_{i}y^{\prime}_{i}|\Big{)}\]
Observe that \(\left|b-b^{\prime}\right|\leq\left\|\mathbf{b}-\mathbf{b}^{\prime}\right\|_{\infty}\leq\varepsilon\) and

\[y_{i}w_{i}-y_{i}^{\prime}w_{i}^{\prime} =(y_{i}-y_{i}^{\prime})w_{i}+y_{i}^{\prime}(w_{i}-w_{i}^{\prime})\] \[=(y_{i}-y_{i}^{\prime})w_{i}+(y_{i}^{\prime}-y_{i})(w_{i}-w_{i}^{ \prime})+y_{i}(w_{i}-w_{i}^{\prime})\] \[\leq\left\|\mathbf{y}-\mathbf{y}^{\prime}\right\|_{\infty}\left\|\mathbf{w} \right\|_{\infty}+\varepsilon\left\|\mathbf{y}-\mathbf{y}^{\prime}\right\|_{\infty}+ \varepsilon\left\|\mathbf{y}\right\|_{\infty}.\]

Hence

\[\left|g_{\mathfrak{A},v}(\mathbf{x},\mathbf{b},\mathbf{w})-g_{\mathfrak{A},v} (\mathbf{x},\mathbf{b}^{\prime},\mathbf{w}^{\prime})\right| \leq\lambda\Big{(}\varepsilon+\Delta\big{(}\left\|\mathbf{y}-\mathbf{y}^ {\prime}\right\|_{\infty}\left\|\mathbf{w}\right\|_{\infty}+\left\|\mathbf{y}-\mathbf{y}^ {\prime}\right\|_{\infty}\varepsilon+\left\|\mathbf{y}\right\|_{\infty} \varepsilon\big{)}\Big{)}\] \[\leq\Delta\lambda\left\|\mathbf{y}-\mathbf{y}^{\prime}\right\|_{\infty} \left\|\mathbf{w}\right\|_{\infty}+\Delta\lambda\big{(}\left\|\mathbf{y}\right\|_{ \infty}+\left\|\mathbf{y}-\mathbf{y}^{\prime}\right\|_{\infty}+1\big{)}\varepsilon.\]

This proves the claim.

By the inductive hypothesis (6.M), we have

\[\left\|\mathbf{y}-\mathbf{y}^{\prime}\right\|_{\infty}\leq\big{(}4\Delta\lambda \gamma(\left\|\mathbf{w}\right\|_{\infty}+1)\big{)}^{t-1}\big{(}\left\|\mathbf{x} \right\|_{\infty}+\left\|\mathbf{b}\right\|_{\infty}+1\big{)}\varepsilon. \tag{6.O}\]

Thus

\[\Delta\lambda\left\|\mathbf{y}-\mathbf{y}^{\prime}\right\|_{\infty}\left\|\mathbf{w} \right\|_{\infty}\leq 4^{t-1}\big{(}\Delta\lambda\gamma(\left\|\mathbf{w} \right\|_{\infty}+1)\big{)}^{t}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\| \mathbf{b}\right\|_{\infty}+1\big{)}\varepsilon \tag{6.P}\]

and

\[\Delta\lambda\left\|\mathbf{y}-\mathbf{y}^{\prime}\right\|_{\infty}\varepsilon \leq\Delta\lambda\big{(}4\Delta\lambda\gamma(\left\|\mathbf{w} \right\|_{\infty}+1)\big{)}^{t-1}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\| \mathbf{b}\right\|_{\infty}+1\big{)}\varepsilon^{2}\] \[\leq 4^{t-1}\big{(}\Delta\lambda\gamma(\left\|\mathbf{w}\right\|_{ \infty}+1)\big{)}^{t}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\|\mathbf{b} \right\|_{\infty}+1\big{)}\varepsilon\qquad\quad\text{ because }\varepsilon\leq 1. \tag{6.Q}\]

By Lemma 6.5 we have \(\left\|\mathbf{y}\right\|_{\infty}\leq\gamma^{t-1}\big{(}\left\|\mathbf{w}\right\|_{ \infty}+1\big{)}^{t-1}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\|\mathbf{b} \right\|_{\infty}+1\big{)}\) and thus

\[\Delta\lambda\left\|\mathbf{y}\right\|_{\infty}\varepsilon \leq\Delta\lambda\gamma^{t-1}\big{(}\left\|\mathbf{w}\right\|_{ \infty}+1\big{)}^{t-1}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\|\mathbf{b} \right\|_{\infty}+1\big{)}\varepsilon\] \[\leq\big{(}\Delta\lambda\gamma(\left\|\mathbf{w}\right\|_{\infty}+1) \big{)}^{t}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\|\mathbf{b}\right\|_{\infty} +1\big{)}\varepsilon \tag{6.R}\]

Plugging (6.P), (6.Q), and (6.R) into Claim 1, we get

\[\leq 4^{t-1}\big{(}\Delta\lambda\gamma(\left\|\mathbf{w}\right\|_{ \infty}+1)\big{)}^{t}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\|\mathbf{b} \right\|_{\infty}+1\big{)}\varepsilon\] \[\leq 4^{t-1}\big{(}\Delta\lambda\gamma(\left\|\mathbf{w}\right\|_{ \infty}+1)\big{)}^{t}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\|\mathbf{b} \right\|_{\infty}+1\big{)}\varepsilon\] \[\leq 4^{t}\big{(}\Delta\lambda\gamma(\left\|\mathbf{w}\right\|_{ \infty}+1)\big{)}^{t}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\|\mathbf{b} \right\|_{\infty}+1\big{)}\varepsilon.\]

This proves (6.M) and thus completes the proof of the lemma. 

**Lemma 6.8**.: _Let \(\beta=(4\Delta\lambda\gamma)^{d}\), where \(\gamma\) is the constant of Lemma 6.5. Let \(\varepsilon>0\). For all \(v\in V\), let \(\mathfrak{a}_{v}^{\prime}:\mathbb{R}\to\mathbb{R}\) be an \(\varepsilon\)-approximation of \(\mathfrak{a}_{v}\) that is Lipschitz continuous with constant \(2\lambda\), and let \(\mathfrak{A}^{\prime}=(V,E,(\mathfrak{a}_{v}^{\prime})_{v\in V})\). Then for all \(\mathbf{x}\in\mathbb{R}^{p}\), \(\mathbf{b}\in\mathbb{R}^{V}\), and \(\mathbf{w}\in\mathbb{R}^{E}\),_

\[\left\|\mathfrak{A}(\mathbf{x},\mathbf{b},\mathbf{w})-\mathfrak{A}^{\prime}(\mathbf{x},\mathbf{b}, \mathbf{w})\right\|_{\infty}\leq\beta\big{(}\left\|\mathbf{w}\right\|_{\infty}+1 \big{)}^{d}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\|\mathbf{b}\right\|_{\infty} +1\big{)}\varepsilon.\]
Proof.: We shall prove by induction on \(t\) that for all nodes \(v\in V\) of depth \(t\) we have

(6.S)

This yields the assertion of the lemma.

Nodes of depth \(t=0\) are input nodes, and we have

\[\big{|}g_{\mathfrak{A},X_{i}}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})-g_{ \mathfrak{A}^{\prime},X_{i}}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w}) \big{|}=|x_{i}-x_{i}|=0.\]

For the inductive step, let \(v\in V\) be a node of depth \(t>0\), and let \(v_{1},\ldots,v_{k}\) be its in-neighbours. Let \(b\coloneqq b_{v}\) and \(w_{i}\coloneqq w_{v_{i}v}\) for \(i\in[k]\). Moreover, let \(\boldsymbol{y}=(y_{1},\ldots,y_{k})\) and \(\boldsymbol{y}^{\prime}=(y_{1}^{\prime},\ldots,y_{k}^{\prime})\) with \(y_{i}\coloneqq g_{\mathfrak{A},v_{i}}(\boldsymbol{x},\boldsymbol{b}, \boldsymbol{w})\) and \(y_{i}^{\prime}\coloneqq g_{\mathfrak{A}^{\prime},v_{i}}(\boldsymbol{x}, \boldsymbol{b},\boldsymbol{w})\).

_Claim 1_.: \[\big{|}g_{\mathfrak{A},v}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})-g_{ \mathfrak{A}^{\prime},v}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})\big{|} \leq 2\gamma^{t}\big{(}\|\boldsymbol{w}\|+1\big{)}^{t}\big{(}\| \boldsymbol{x}\|_{\infty}+\|\boldsymbol{b}\|_{\infty}+1\big{)}\varepsilon+2 \Delta\lambda\left\|\boldsymbol{w}\right\|_{\infty}\left\|\boldsymbol{y}- \boldsymbol{y}^{\prime}\right\|_{\infty}\]

Proof.: We have

\[\big{|}g_{\mathfrak{A},v}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})-g_{ \mathfrak{A}^{\prime},v}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})\big{|} =\Big{|}\mathfrak{a}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y_{i}\Big{)} -\mathfrak{a}^{\prime}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y_{i}^{\prime}\Big{)}\Big{|} \tag{6.T}\] \[\leq\Big{|}\mathfrak{a}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y_{i}\Big{)} -\mathfrak{a}^{\prime}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y_{i}\Big{)}\Big{|}\] \[\quad+\Big{|}\mathfrak{a}^{\prime}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i }y_{i}\Big{)}-\mathfrak{a}^{\prime}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y_{i}^{ \prime}\Big{)}\Big{|}\]

By Lemma 6.5 we have

\[\Big{|}\mathfrak{a}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y_{i}\Big{)}\Big{|}=\big{|} g_{\mathfrak{A},v}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w})\big{|} \leq\gamma^{t}\big{(}\|\boldsymbol{w}\|+1\big{)}^{t}\big{(}\| \boldsymbol{x}\|_{\infty}+\|\boldsymbol{b}\|_{\infty}+1\big{)}. \tag{6.U}\]

Since \(\mathfrak{a}^{\prime}_{v}\)\(\varepsilon\)-approximates \(\mathfrak{a}_{v}\), this implies

\[\leq 2\gamma^{t}\big{(}\|\boldsymbol{w}\|+1\big{)}^{t}\big{(}\| \boldsymbol{x}\|_{\infty}+\|\boldsymbol{b}\|_{\infty}+1\big{)}\varepsilon. \tag{6.V}\]

Furthermore, by the Lipschitz continuity of \(\mathfrak{a}^{\prime}_{v}\) we have

\[\Big{|}\mathfrak{a}^{\prime}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y_{i}\Big{)}- \mathfrak{a}^{\prime}_{v}\Big{(}b+\sum_{i=1}^{k}w_{i}y_{i}^{\prime}\Big{)} \Big{|}\leq 2\lambda\sum_{i=1}^{k}|w_{i}|\cdot|y_{i}-y_{i}^{\prime}|\leq 2 \Delta\lambda\left\|\boldsymbol{w}\right\|_{\infty}\left\|\boldsymbol{y}- \boldsymbol{y}^{\prime}\right\|_{\infty}. \tag{6.W}\]

The assertion of the claim from (6.T), (6.V), and (6.W).

By the inductive hypothesis (6.S), we have

\[\big{|}\boldsymbol{y}-\boldsymbol{y}^{\prime}\big{|}_{\infty}\leq(4\Delta \lambda\gamma)^{t-1}\big{(}\|\boldsymbol{w}\|_{\infty}+1\big{)}^{t-1}\big{(} \|\boldsymbol{x}\|_{\infty}+\|\boldsymbol{b}\|_{\infty}+1\big{)}\varepsilon.\]
Thus by the claim,

\[\big{|}g_{\mathfrak{A},v}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{ w})-g_{\mathfrak{A}^{\prime},v}(\boldsymbol{x},\boldsymbol{b},\boldsymbol{w}) \big{|} \leq 2\gamma^{t}\big{(}\|\boldsymbol{w}\|+1\big{)}^{t}\big{(}\| \boldsymbol{x}\|_{\infty}+\|\boldsymbol{b}\|_{\infty}+1\big{)}\varepsilon+2 \Delta\lambda\left\|\boldsymbol{w}\right\|_{\infty}\left\|\boldsymbol{y}- \boldsymbol{y}^{\prime}\right\|_{\infty}\] \[\leq 2\gamma^{t}\big{(}\|\boldsymbol{w}\|+1\big{)}^{t}\big{(}\| \boldsymbol{x}\|_{\infty}+\|\boldsymbol{b}\|_{\infty}+1\big{)}\varepsilon\] \[\quad+2\cdot 4^{t-1}(\Delta\lambda\gamma)^{t}\big{(}\left\| \boldsymbol{w}\right\|_{\infty}+1\big{)}^{t}\big{(}\left\|\boldsymbol{x} \right\|_{\infty}+\|\boldsymbol{b}\|_{\infty}+1\big{)}\varepsilon\] \[\leq(4\gamma\Delta\lambda)^{t}\big{(}\left\|\boldsymbol{w} \right\|_{\infty}+1\big{)}^{t}\big{(}\left\|\boldsymbol{x}\right\|_{\infty}+ \left\|\boldsymbol{b}\right\|_{\infty}+1\big{)}\varepsilon\]

This proves (6.S) and hence the lemma. 

**Lemma 6.9**.: _Let \(f:\mathbb{R}\to\mathbb{R}\) be Lipschitz continuous with constant \(\lambda>0\) such that \(f\) is rpl approximable. Then for every \(\varepsilon>0\) there is a rational piecewise linear function \(L\) of bitsize polynomial in \(\varepsilon^{-1}\) such that \(L\) is an \(\varepsilon\)-approximation of \(f\) and \(L\) is Lipschitz continuous with constant \((1+\varepsilon)\lambda\)._

Proof.: Let \(0<\varepsilon\leq 1\) and \(\varepsilon^{\prime}\coloneqq\frac{\varepsilon}{10}\). Let \(L^{\prime}\) be a piecewise linear \(\varepsilon^{\prime}\)-approximation of \(f\) of bitsize polynomial in \(\varepsilon^{-1}\). Let \(t_{1}<\ldots<t_{n}\) be the thresholds of \(L^{\prime}\), and let \(a_{0},\ldots,a_{n}\) and \(b_{0},\ldots,b_{n}\) be its slopes and constants. Then \(|a_{0}|\leq(1+\varepsilon)\lambda\); otherwise the slope of the linear function \(a_{0}x+b_{0}\) would be too large (in absolute value) to approximate the function \(f\) whose slope is bounded by \(\lambda\). For the same reason, \(|a^{\prime}_{n}|\leq(1+\varepsilon)\lambda\).

Let \(s\coloneqq t_{1}\) and \(s^{\prime}\coloneqq t_{n}\). We subdivide the interval \([s,s^{\prime}]\) into sufficiently small subintervals (of length at most \(\varepsilon^{\prime}\cdot\lambda^{-1}\)). Within each such interval, \(f\) does not change much, because it is Lipschitz continuous, and we can approximate it sufficiently closely by a linear function with parameters whose bitsize is polynomially bounded in \(\varepsilon^{-1}\). The slope of theses linear functions will not be significantly larger than \(\lambda\), because the slope of \(f\) is bounded by \(\lambda\). We can combine all these linear pieces with the linear functions \(a_{0}x+b_{0}\) for the interval \((-\infty,s]\) and \(a_{n}x+b_{n}\) for the interval \([s^{\prime},\infty)\) to obtain the desired piecewise linear approximation of \(f\). 

Now we are ready to prove the main result of this subsection.

**Lemma 6.10**.: _For every \(d\in\mathbb{N}_{>0}\) there is a polynomial \(\pi^{\prime}(X,Y)\) such that the following holds. Let \(\mathfrak{F}=\big{(}V,E,(\mathfrak{a}_{v})_{v\in V},\boldsymbol{w},\boldsymbol{ b}\big{)}\) be an rpl-approximable FNN architecture of depth \(d\). Let \(\varepsilon>0\). Then there exists a rational piecewise-linear FNN \(\mathfrak{F}^{\prime}=\big{(}V,E,(\mathfrak{a}^{\prime}_{v})_{v\in V}, \boldsymbol{w}^{\prime},\boldsymbol{b}^{\prime}\big{)}\) of size at most \(\pi^{\prime}\big{(}\varepsilon^{-1},\operatorname{wt}(\mathfrak{F})\big{)}\) such that for all \(v\in V\) it holds that \(\lambda(\mathfrak{a}^{\prime}_{v})\leq 2\lambda(\mathfrak{a}_{v})\) and for all \(\boldsymbol{x}\in\mathbb{R}^{p}\) it holds that_

\[\big{\|}\mathfrak{F}(\boldsymbol{x})-\mathfrak{F}^{\prime}(\boldsymbol{x}) \big{\|}_{\infty}\leq\big{(}\left\|\boldsymbol{x}\right\|_{\infty}+1\big{)}\varepsilon.\]

Note that \(\mathfrak{F}^{\prime}\) has the same skeleton as \(\mathfrak{F}\).

Proof.: Without loss of generality we assume \(\varepsilon\leq 1\). Let \(\mathfrak{A}\coloneqq\big{(}V,E,(\mathfrak{a}_{v})_{v\in V}\big{)}\). Define the parameters \(\Delta,\lambda,\mu\) with respect to \(\mathfrak{A}\) as before. Note that \(\Delta,\lambda,\mu\leq\operatorname{wt}(\mathfrak{F})\). Choose the constants \(\gamma\) according to Lemma 6.5, \(\nu\) according to Lemma 6.7, and \(\beta\) according to Lemma 6.8 and note that for fixed \(d\) they depend polynomially on \(\Delta,\lambda,\mu\) and hence on \(\operatorname{wt}(\mathfrak{F})\).
Let \(\alpha\coloneqq 2\nu\big{(}\left\|\mathbf{w}\right\|_{\infty}+1\big{)}^{d}\big{(} \left\|\mathbf{b}\right\|_{\infty}+1\big{)}\). Let \(\mathbf{w}^{\prime}\in\mathbb{Z}\big{[}\frac{1}{2}\big{]}^{E}\), \(\mathbf{b}^{\prime}\in\mathbb{Z}\big{[}\frac{1}{2}\big{]}^{V}\) such that \(\|\mathbf{w}-\mathbf{w}^{\prime}\|_{\infty}\leq\frac{\varepsilon}{\alpha}\) and \(\|\mathbf{b}-\mathbf{b}^{\prime}\|_{\infty}\leq\frac{\varepsilon}{\alpha}\). Clearly, we can choose such \(\mathbf{w}^{\prime}=(w_{e}^{\prime})_{eeE}\) and \(\mathbf{b}^{\prime}=(b_{v}^{\prime})_{v\in V}\) such that all their entries have bitsize polynomial in \(\frac{\alpha}{\varepsilon}\), which is polynomial in \(\varepsilon^{-1}\) and in \(\operatorname{wt}(\mathfrak{F})\).. Then by Lemma 6.7, for all \(\mathbf{x}\in\mathbb{R}^{p}\) we have

\[\big{\|}\mathfrak{A}(\mathbf{x},\mathbf{b},\mathbf{w})-\mathfrak{A}(\mathbf{x},\mathbf{b}^{\prime },\mathbf{w}^{\prime})\big{\|}_{\infty}\leq\nu\big{(}\left\|\mathbf{w}\right\|_{\infty }+1\big{)}^{d}\big{(}\left\|\mathbf{x}\right\|_{\infty}+\left\|\mathbf{b}\right\|_{ \infty}+1\big{)}\frac{\varepsilon}{\alpha}\leq\big{(}\left\|\mathbf{x}\right\|_{ \infty}+1\big{)}\frac{\varepsilon}{2},\]

Let \(\alpha^{\prime}\coloneqq 2\beta\big{(}\left\|\mathbf{w}^{\prime}\right\|_{\infty}+1 \big{)}^{d}\big{(}\left\|\mathbf{b}^{\prime}\right\|_{\infty}+1\big{)}\). For every \(v\in V\), we let \(\mathfrak{a}_{v}^{\prime}\) be a rational piecewise-linear function of bitsize polynomial in in \(\varepsilon^{-1}\) that is an \(\frac{\varepsilon}{\alpha^{\prime}}\)-approximation of \(\mathfrak{a}_{v}\) and Lipschitz continuous with constant \(2\lambda\). Such an \(\mathfrak{a}_{v}^{\prime}\) exists by Lemma 6.9, because \(\mathfrak{a}_{v}\) is rpl-approximable and Lipschitz continuous with constant \(\lambda\). Let \(\mathfrak{A}^{\prime}\coloneqq\big{(}V,E,(\mathfrak{a}_{v}^{\prime})_{v\in V} \big{)}\). By Lemma 6.8, for all \(\mathbf{x}\in\mathbb{R}^{p}\) we have

\[\big{\|}\mathfrak{A}(\mathbf{x},\mathbf{b}^{\prime},\mathbf{w}^{\prime})-\mathfrak{A}^{ \prime}(\mathbf{x},\mathbf{b}^{\prime},\mathbf{w}^{\prime})\big{\|}_{\infty}\leq\beta \big{(}\left\|\mathbf{w}^{\prime}\right\|_{\infty}+1\big{)}^{d}\big{(}\left\|\mathbf{ x}\right\|_{\infty}+\left\|\mathbf{b}^{\prime}\right\|_{\infty}+1\big{)}\frac{ \varepsilon}{\alpha^{\prime}}\leq\big{(}\left\|\mathbf{x}\right\|_{\infty}+1\big{)} \frac{\varepsilon}{2}.\]

Overall,

\[\big{\|}\mathfrak{F}(\mathbf{x})-\mathfrak{F}^{\prime}(\mathbf{x})\big{\|}_ {\infty} =\big{\|}\mathfrak{A}(\mathbf{x},\mathbf{b},\mathbf{w})-\mathfrak{A}^{ \prime}(\mathbf{x},\mathbf{b}^{\prime},\mathbf{w}^{\prime})\big{\|}_{\infty}\] \[\leq\big{\|}\mathfrak{A}(\mathbf{x},\mathbf{b},\mathbf{w})-\mathfrak{A}(\mathbf{ x},\mathbf{b}^{\prime},\mathbf{w}^{\prime})\big{\|}_{\infty}+\big{\|}\mathfrak{A}(\mathbf{x}, \mathbf{b}^{\prime},\mathbf{w}^{\prime})-\mathfrak{A}^{\prime}(\mathbf{x},\mathbf{b}^{\prime },\mathbf{w}^{\prime})\big{\|}_{\infty}\] \[\leq\big{(}\left\|\mathbf{x}\right\|_{\infty}+1\big{)}\varepsilon.\]

### Bounds and Approximations for GNNs

We start with a more explicit version of Lemmas 4.2 and 4.3, the growth bounds for GNN layers. Recall that the depth of a GNN layer is the maximum of the depths of the FNNs for the combination and the message function.

**Lemma 6.11**.: _For every \(d\) there is a polynomial \(\pi(X)\) such that the following holds. Let \(\mathfrak{L}\) be a GNN layer of depth \(d\), and let \(p\) be the input dimension of \(\mathfrak{L}\). Then for all graphs \(G\) and all signals \(\mathbf{x}\in\mathcal{S}_{p}(G)\) we have_

\[\big{\|}\mathfrak{L}(G,\mathbf{x})\big{\|}_{\infty}\leq\pi\big{(}\operatorname{wt} (\mathfrak{L})\big{)}\big{(}\left\|\mathbf{x}\right\|_{\infty}+1\big{)}|G|. \tag{6.X}\]

Proof.: The proof of Lemma 4.2 yields

\[\big{\|}\mathfrak{L}(G,\mathbf{x})(v)\big{\|}_{\infty}\leq 2\gamma_{\mathsf{msg}} \gamma_{\mathsf{comb}}\big{(}\left\|\mathbf{x}\right\|_{\infty}+1\big{)}|G|,\]

where \(\gamma_{\mathsf{msg}}\), \(\gamma_{\mathsf{comb}}\) are growth bounds for the message and combination functions of \(\mathfrak{L}\). It follows from Lemma 6.5 that \(\gamma_{\mathsf{msg}}\), \(\gamma_{\mathsf{comb}}\) can be chosen polynomial in the weight of \(\mathfrak{L}\). 

**Lemma 6.12**.: _For every \(d\) there is a polynomial \(\pi(X)\) such that the following holds. Let \(\mathfrak{L}\) be a GNN layer of depth \(d\), and let \(p\) be the input dimension of \(\mathfrak{L}\). Then for all graphs \(G\) and all signals \(\mathbf{x},\mathbf{x}^{\prime}\in\mathcal{S}_{p}(G)\) we have_

\[\big{\|}\mathfrak{L}(G,\mathbf{x})-\mathfrak{L}(G,\mathbf{x}^{\prime})\big{\|}_{\infty} \leq\pi\big{(}\operatorname{wt}(\mathfrak{L})\big{)}\left\|\mathbf{x}-\mathbf{x}^{ \prime}\right\|_{\infty}|G|. \tag{6.Y}\]
Proof.: The proof of Lemma 4.3 yields

\[\left\|\widetilde{\mathfrak{L}}(G,x)(v)-\widetilde{\mathfrak{L}}(G,x^{\prime})(v) \right\|_{\infty}\leq\lambda_{\mathsf{msg}}\lambda_{\mathsf{comb}}\left\|x-x^ {\prime}\right\|_{\infty}|G|,\]

where \(\lambda_{\mathsf{msg}}\), \(\lambda_{\mathsf{comb}}\) are Lipschitz constants for the message and combination functions of \(\mathfrak{L}\). It follows from Lemma 6.6 that \(\lambda_{\mathsf{msg}}\), \(\lambda_{\mathsf{comb}}\) can be chosen polynomial in the weight of \(\mathfrak{L}\). 

**Corollary 6.13**.: _For every \(d\) there is a polynomial \(\pi(X)\) such that the following holds. Let \(\mathfrak{N}\) be a GNN of depth \(d\), and let \(p\) be the input dimension of \(\mathfrak{N}\). Then for all graphs \(G\) and all signals \(x,x^{\prime}\in\mathcal{S}_{p}(G)\) we have_

\[\left\|\widetilde{\mathfrak{N}}(G,x)-\widetilde{\mathfrak{N}}(G,x^{\prime}) \right\|_{\infty}\leq\pi\big{(}\operatorname{wt}(\mathfrak{L})\big{)}\left\| x-x^{\prime}\right\|_{\infty}|G|^{d}.\]

**Lemma 6.14**.: _For every \(d\in\mathbb{N}_{>0}\) there exist polynomials \(\pi(X)\) and \(\pi^{\prime}(X,Y)\) such that the following holds. Let \(\mathfrak{L}\) be an rpl-approximable GNN layer of depth \(d\), and let \(p\) be the input dimension of \(\mathfrak{L}\). Then for all \(\varepsilon>0\) there exists a rational piecewise-linear GNN layer \(\mathfrak{L}^{\prime}\) of size at most \(\pi^{\prime}(\varepsilon^{-1},\operatorname{wt}(\mathfrak{L}))\) with the same skeleton as \(\mathfrak{L}\) such that the Lipschitz constants of the activation functions of \(\mathfrak{L}^{\prime}\) are at most twice the Lipschitz constants of the corresponding activation functions in \(\mathfrak{L}\) and for all graphs \(G\), all signals \(x,x^{\prime}\in\mathcal{S}_{p}(G)\), and all vertices \(v\in V(G)\) it holds that_

Proof.: Let \(\mathsf{msg}:\mathbb{R}^{2p}\to\mathbb{R}^{r}\) and \(\mathsf{comb}:\mathbb{R}^{p+r}\to\mathbb{R}^{q}\) be the message and combination functions of \(\mathfrak{L}\), and let \(\mathfrak{F}_{\mathsf{msg}}\) and \(\mathfrak{F}_{\mathsf{comb}}\) be the FNNs for these functions. By Lemma 6.10 there are rational piecewise linear FNNs \(\mathfrak{F}^{\prime}_{\mathsf{msg}}\) and \(\mathfrak{F}^{\prime}_{\mathsf{comb}}\) of size polynomial in \(\operatorname{wt}(\mathfrak{F}_{\mathsf{msg}}),\operatorname{wt}(\mathfrak{ F}_{\mathsf{comb}})\leq\operatorname{wt}(\mathfrak{L})\) with activation functions of Lipschitz constants at most twice the Lipschitz constants of the corresponding activation functions in \(\mathfrak{F}_{\mathsf{msg}}\), \(\mathfrak{F}_{\mathsf{comb}}\) such that for all \(x,x^{\prime}\in\mathbb{R}^{p}\) and \(z\in\mathbb{R}^{r}\) we have

\[\left\|\mathsf{msg}(x,x^{\prime})-\mathsf{msg}^{\prime}(x,x^{ \prime})\right\|_{\infty}\leq\big{(}\left\|(x,x^{\prime})\right\|_{\infty}+1 \big{)}\varepsilon,\] (6.Z) \[\left\|\mathsf{comb}(x,z)-\mathsf{comb}^{\prime}(x,z)\right\|_{\infty} \leq\big{(}\left\|(x,z)\right\|_{\infty}+1\big{)}\varepsilon.\] (6.AA)

Let \(\mathfrak{L}^{\prime}\) be the GNN layer with message function \(\mathsf{msg}^{\prime}\), combination function \(\mathsf{comb}^{\prime}\), and the same aggregation function \(\mathsf{agg}\) as \(\mathfrak{L}\). By Lemma 6.5 there is an \(\alpha\in\mathbb{N}_{>0}\) that is polynomial in \(\operatorname{wt}(\mathfrak{F}_{\mathsf{msg}})\) and hence polynomial in \(\operatorname{wt}(\mathfrak{L})\) such that for all \(x,x^{\prime}\in\mathbb{R}^{p}\) it holds that

\[\left\|\mathsf{msg}(x,x^{\prime})\right\|_{\infty}\leq\alpha\big{(}\max\big{\{} \left\|x\right\|_{\infty},\left\|x^{\prime}\right\|_{\infty}\big{\}}+1\big{)}.\] (6.AB)

By Lemma 6.6 there is an \(\alpha^{\prime}\in\mathbb{N}_{>0}\) polynomial in \(\operatorname{wt}(\mathfrak{F}^{\prime}_{\mathsf{comb}})\) and hence polynomial in \(\operatorname{wt}(\mathfrak{L})\) such that for all \(x,x^{\prime}\in\mathbb{R}^{p},z,z^{\prime}\in\mathbb{R}^{r}\) it holds that

\[\left\|\mathsf{comb}^{\prime}(x,z)-\mathsf{comb}^{\prime}(x^{\prime},z^{ \prime})\right\|_{\infty}\leq\alpha^{\prime}\max\big{\{}\left\|x-x^{\prime} \right\|_{\infty},\left\|z-z^{\prime}\right\|_{\infty}\big{\}}.\] (6.AC)
By Lemma 6.12 there is an \(\alpha^{\prime\prime}\in\mathbb{N}_{>0}\) polynomial in \(\operatorname{wt}(\mathfrak{L}^{\prime})\leq\operatorname{size}(\mathfrak{L}^{ \prime})\) and thus polynomial in \(\operatorname{wt}(\mathfrak{L})\) such that for all \(G\) and \(\boldsymbol{x}\in\mathcal{S}_{p}(G)\),

\[\big{\|}\widetilde{\mathfrak{L}}^{\prime}(G,\boldsymbol{x})-\widetilde{ \mathfrak{L}}^{\prime}(G,\boldsymbol{x}^{\prime})\big{\|}_{\infty}\leq\alpha^{ \prime\prime}\,\big{\|}\boldsymbol{x}-\boldsymbol{x}^{\prime}\big{\|}_{\infty }\,|G|.\] (6.AD)

Let \(G\) be a graph of order \(n=|G|\) and \(v\in V(G)\). Furthermore, let \(\boldsymbol{x},\boldsymbol{x}^{\prime}\in\mathcal{S}_{p}(G)\) and \(\boldsymbol{y}\coloneqq\widetilde{\mathfrak{L}}(G,\boldsymbol{x})\), \(\boldsymbol{y}^{\prime}\coloneqq\widetilde{\mathfrak{L}}^{\prime}(G, \boldsymbol{x})\), \(\boldsymbol{y}^{\prime\prime}\coloneqq\widetilde{\mathfrak{L}}^{\prime}(G, \boldsymbol{x}^{\prime})\). Then

\[\big{\|}\widetilde{\mathfrak{L}}(G,\boldsymbol{x})(v)-\widetilde{\mathfrak{ L}}^{\prime}(G,\boldsymbol{x}^{\prime})(v)\big{\|}_{\infty}\leq\big{\|} \boldsymbol{y}(v)-\boldsymbol{y}^{\prime}(v)\big{\|}_{\infty}+\big{\|} \boldsymbol{y}^{\prime}(v)-\boldsymbol{y}^{\prime\prime}(v)\big{\|}_{\infty}\,.\]

By (6.AD) we have

\[\big{\|}\boldsymbol{y}^{\prime}(v)-\boldsymbol{y}^{\prime\prime}(v)\big{\|}_{ \infty}\leq\alpha^{\prime\prime}n\,\big{\|}\boldsymbol{x}-\boldsymbol{x}^{ \prime}\big{\|}_{\infty}\,.\] (6.AE)

Thus we need to bound \(\|\boldsymbol{x}(v)-\boldsymbol{y}^{\prime}(v)\|_{\infty}\). Let

\[\boldsymbol{x}(v) \coloneqq\operatorname{\mathsf{agg}}\Bigl{(}\big{\{}\mathsf{ msg}(\boldsymbol{x}(v),\boldsymbol{x}(w))\,\big{|}\,w\in N_{G}(v)\big{\}} \Bigr{)},\] \[\boldsymbol{x}^{\prime}(v) \coloneqq\operatorname{\mathsf{agg}}\Bigl{(}\big{\{}\mathsf{ msg}(\boldsymbol{x}^{\prime}(v),\boldsymbol{x}^{\prime}(w))\,\big{|}\,w\in N_{G}(v) \big{\}}\Bigr{)}.\]

Then by (6.Z), we have

\[\big{\|}\boldsymbol{x}(v)-\boldsymbol{x}^{\prime}(v)\big{\|}_{\infty}\leq n \left(\big{\|}\boldsymbol{x}\big{\|}_{\infty}+1\right)\varepsilon.\] (6.AF)

Furthermore, by (6.AB), for all \(w\in N(v)\) we have

\[\big{\|}\mathsf{msg}\bigl{(}\boldsymbol{x}(v),\boldsymbol{x}(w)\bigr{)} \big{\|}_{\infty}\leq\alpha\Bigl{(}\big{\|}\boldsymbol{x}\big{\|}_{\infty}+1 \Bigr{)}\]

Thus, since \(\alpha\geq 1\) and \(n\geq 1\),

\[\big{\|}\big{(}\boldsymbol{x}(v),\boldsymbol{x}(v)\big{)}\big{\|}_{\infty}= \max\Bigl{\{}\,\|\boldsymbol{x}(v)\|_{\infty}\,,\|\boldsymbol{x}(v)\|_{\infty }\,\Bigr{\}}\leq\alpha n\Bigl{(}\big{\|}\boldsymbol{x}\big{\|}_{\infty}+1 \Bigr{)}.\] (6.AG)

Putting things together, we get

\[\big{\|}\boldsymbol{y}(v)-\boldsymbol{y}^{\prime}(v)\big{\|}_{\infty} =\big{\|}\operatorname{\mathsf{comb}}\bigl{(}\boldsymbol{x}(v), \boldsymbol{x}(v)\bigr{)}-\operatorname{\mathsf{comb}}^{\prime}\bigl{(} \boldsymbol{x}(v),\boldsymbol{x}^{\prime}(v)\bigr{)}\big{\|}_{\infty}\] \[\leq\big{\|}\operatorname{\mathsf{comb}}\bigl{(}\boldsymbol{x}(v ),\boldsymbol{x}(v)\bigr{)}-\operatorname{\mathsf{comb}}^{\prime}\bigl{(} \boldsymbol{x}(v),\boldsymbol{x}(v)\bigr{)}\big{\|}_{\infty}\] \[\quad+\big{\|}\operatorname{\mathsf{comb}}^{\prime}\bigl{(} \boldsymbol{x}(v),\boldsymbol{x}(v)\bigr{)}-\operatorname{\mathsf{comb}}^{ \prime}\bigl{(}\boldsymbol{x}(v),\boldsymbol{x}^{\prime}(v)\bigr{)}\big{\|}_{\infty}\] \[\leq\big{(}\,\|(\boldsymbol{x}(v),\boldsymbol{x}(v))\|_{\infty}+1 \big{)}\varepsilon\] by (6.AA) \[\quad+\alpha^{\prime}\,\big{\|}\boldsymbol{x}(v)-\boldsymbol{x} ^{\prime}(v)\big{\|}_{\infty}\] (6.AC) \[\leq 2\alpha n\Bigl{(}\big{\|}\boldsymbol{x}\big{\|}_{\infty}+1 \Bigr{)}\varepsilon+\alpha^{\prime}n\bigl{(}\big{\|}\boldsymbol{x}\big{\|}_{ \infty}+1\bigr{)}\varepsilon\] by (6.AG) and (6.AF) \[\leq(2\alpha+\alpha^{\prime})n\bigl{(}\,\big{\|}\boldsymbol{x} \big{\|}_{\infty}+1\bigr{)}\varepsilon\]

Combined with (6.AE), this yields the assertion of the lemma. 

Now we are ready to prove the main lemma of this section.
**Lemma 6.15**.: _For every \(d\in\mathbb{N}_{>0}\) there exist a polynomials \(\pi(X,Y)\) such that the following holds. Let \(\mathfrak{N}\) be an rpl-approximable GNN of depth \(d\), and let \(p\) be the input dimension of \(\mathfrak{N}\). Then for all \(\varepsilon>0\) there exists a rational piecewise-linear GNN \(\mathfrak{N}^{\prime}\) of size at most \(\pi(\varepsilon^{-1},\operatorname{wt}(\mathfrak{N}))\) with the same skeleton as \(\mathfrak{N}\) such that the Lipschitz constants of the activation functions of \(\mathfrak{N}^{\prime}\) are at most twice the Lipschitz constants of the corresponding activation functions in \(\mathfrak{N}\) and for all graphs \(G\) and all signals \(\boldsymbol{x}\in\mathcal{S}_{p}(G)\) it holds that_

\[\big{\|}\widehat{\mathfrak{N}}(G,\boldsymbol{x})-\widehat{\mathfrak{N}}^{ \prime}(G,\boldsymbol{x})\big{\|}_{\infty}\leq|G|^{d}\big{(}\left\|\boldsymbol {x}\right\|_{\infty}+1\big{)}\varepsilon.\]

Proof.: Suppose that \(\mathfrak{N}=(\mathfrak{L}_{1},\ldots,\mathfrak{L}_{d})\). For every \(t\in[d]\), let \(p_{t-1}\) be the input dimension of \(\mathfrak{L}_{t}\). Then \(p=p_{0}\). By Lemma 6.11 there is an \(\alpha\) polynomial in \(\operatorname{wt}(\mathfrak{N})\) such that for all \(t\in[d]\), \(G\), and \(\boldsymbol{x}\in\mathcal{S}_{p_{t-1}}(G)\) we have

\[\big{\|}\widetilde{\mathfrak{L}}_{t}(G,\boldsymbol{x})\big{\|}_{\infty}\leq \alpha|G|\big{(}\left\|\boldsymbol{x}\right\|_{\infty}+1\big{)}.\] (6.AH)

Let \(\pi^{\prime}(X)\) be the polynomial of Lemma 6.14,

\[\alpha^{\prime}\coloneqq\max_{t\in[d]}\pi^{\prime}\big{(}\operatorname{wt}( \mathfrak{L}_{t})\big{)},\]

and

\[\beta \coloneqq 3\max\{\alpha,\alpha^{\prime}\},\] \[\varepsilon^{\prime} \coloneqq\frac{\varepsilon}{\beta^{d}}.\]

Note that \(\varepsilon^{\prime}\) is polynomial in \(\operatorname{wt}(\mathfrak{N})\). For every \(t\in[d]\), we apply Lemma 6.14 to \(\mathfrak{L}_{t}\) and \(\varepsilon^{\prime}\) and obtain a rational piecewise linear GNN layer \(\mathfrak{L}_{t}^{\prime}\) such that for all graphs \(G\) and all signals \(\boldsymbol{x},\boldsymbol{x}^{\prime}\in\mathcal{S}_{p_{t-1}}(G)\) we have

\[\big{\|}\widetilde{\mathfrak{L}}_{t}(G,\boldsymbol{x})-\widetilde{\mathfrak{ L}}_{t}^{\prime}(G,\boldsymbol{x}^{\prime})\big{\|}_{\infty}\leq\alpha^{ \prime}|G|\Big{(}\left\|\boldsymbol{x}-\boldsymbol{x}^{\prime}\right\|_{ \infty}+\big{(}\left\|\boldsymbol{x}\right\|_{\infty}+1\big{)}\varepsilon^{ \prime}\Big{)}.\] (6.AI)

Let \(G\) be a graph of order \(n\coloneqq|G|\), and \(\boldsymbol{x}\in\mathcal{S}_{p}(G)\). Let \(\boldsymbol{x}_{0}\coloneqq\boldsymbol{x}_{0}^{\prime}\coloneqq\boldsymbol{x}\), and for \(t\in[d]\), let \(\boldsymbol{x}_{t}\coloneqq\widetilde{\mathfrak{L}}_{t}(G,\boldsymbol{x}_{t-1})\) and \(\boldsymbol{x}_{t}^{\prime}\coloneqq\widetilde{\mathfrak{L}}_{i}^{\prime}(G, \boldsymbol{x}_{t-1}^{\prime})\). We shall prove that for all \(t\in\{0,\ldots,d\}\) we have

\[\left\|\boldsymbol{x}_{t}\right\|_{\infty} \leq\beta^{t}n^{t}\big{(}\left\|\boldsymbol{x}\right\|_{\infty}+1 \big{)},\] (6.AJ) \[\big{\|}\boldsymbol{x}_{t}-\boldsymbol{x}_{t}^{\prime}\big{\|}_{\infty} \leq\beta^{t}n^{t}\big{(}\left\|\boldsymbol{x}\right\|_{\infty}+1 \big{)}\varepsilon^{\prime}.\] (6.AK)

Since \(\beta^{d}\varepsilon^{\prime}=\varepsilon\) and \(\boldsymbol{x}_{d}=\widehat{\mathfrak{N}}(G,\boldsymbol{x})\), \(\boldsymbol{x}_{d}^{\prime}\coloneqq\widehat{\mathfrak{N}}^{\prime}(G, \boldsymbol{x})\), (6.AK) implies the assertion of the lemma.

We prove (6.AJ) and (6.AK) by induction on \(t\). The base step \(t=0\) is trivial, because \(\boldsymbol{x}_{0}=\boldsymbol{x}\) and \(\boldsymbol{x}_{0}=\boldsymbol{x}^{\prime}\). For the inductive step, let \(t\geq 1\). By (6.AH) and the induction hypothesis we have

\[\left\|\boldsymbol{x}_{t}\right\|_{\infty}\leq\alpha n\big{(}\left\| \boldsymbol{x}_{t-1}\right\|_{\infty}+1\big{)}\]
\[\leq\alpha n\Big{(}\beta^{t-1}n^{t-1}\big{(}\left\|\boldsymbol{x} \right\|_{\infty}+1\big{)}+1\Big{)}\] \[\leq\beta^{t}n^{t}\big{(}\left\|\boldsymbol{x}\right\|_{\infty}+1 \big{)},\]

where the last inequality holds because \(2\alpha\leq\beta\). This proves (6.AJ).

By (6.AJ) we have

\[\left\|\boldsymbol{x}_{t}-\boldsymbol{x}_{t}^{\prime}\right\|_{\infty}\leq \alpha^{\prime}n\Big{(}\left\|\boldsymbol{x}_{t-1}-\boldsymbol{x}_{t-1}^{ \prime}\right\|_{\infty}+\big{(}\left\|\boldsymbol{x}_{t-1}\right\|_{\infty}+ 1\big{)}\varepsilon^{\prime}\Big{)}.\] (6.AL)

By induction hypothesis (6.AK),

\[\alpha^{\prime}n\left\|\boldsymbol{x}_{t-1}-\boldsymbol{x}_{t-1}^ {\prime}\right\|_{\infty} \leq\alpha^{\prime}n\beta^{t-1}n^{t-1}\big{(}\left\|\boldsymbol{x }\right\|_{\infty}+1\big{)}\varepsilon^{\prime}\] \[\leq\frac{1}{3}\beta^{t}n^{t}\big{(}\left\|\boldsymbol{x}\right\| _{\infty}+1\big{)}\varepsilon^{\prime}.\] (6.AM)

By induction hypothesis (6.AJ),

\[\alpha^{\prime}n\big{(}\left\|\boldsymbol{x}_{t-1}\right\|_{ \infty}+1\big{)}\varepsilon^{\prime} \leq\alpha^{\prime}n\Big{(}\beta^{t-1}n^{t-1}\big{(}\left\| \boldsymbol{x}\right\|_{\infty}+1\big{)}+1\Big{)}\varepsilon^{\prime}\] \[\leq\frac{2}{3}\beta^{t}n^{t}\big{(}\left\|\boldsymbol{x}\right\| _{\infty}+1\big{)}\varepsilon^{\prime}\] (6.AN)

Plugging (6.AM) and (6.AN) into (6.AL), we obtain the desired inequality (6.AK). 

### Proof of Theorem 6.2

Let us first remark that we cannot directly apply Theorem 5.1 (the "uniform theorem") to a family of rational piecewise linear GNN approximating the GNNs in our family \(\mathscr{N}\). The reason is that in Theorem 5.1 the GNN is "hardwired" in the formula, whereas in our non-uniform setting we obtain a different GNN for every input size. Instead, we encode the sequence of rational piecewise linear GNNs we obtain into the numerical built-in relations. Then our formula evaluates these GNNs directly on the numerical side of the structures.

Proof of Theorem 6.2.: Let \(\mathscr{N}=(\mathfrak{N}^{(n)})_{n\in\mathbb{N}}\). Furthermore, let \(d\) be an upper bound on the depth of all the \(\mathfrak{N}^{(n)}\). Without loss of generality we assume that every \(\mathfrak{N}^{(n)}\) has exactly \(d\) layers \(\mathfrak{L}_{1}^{(n)},\ldots,\mathfrak{L}_{d}^{(n)}\). For \(t\in[d]\), let \(p_{t-1}^{(n)}\) and \(p_{t}^{(n)}\) be the input and output dimension of \(\mathfrak{L}_{t}^{(n)}\). Then \(p^{(n)}\!=\!p_{0}^{(n)}\) is the input dimension of \(\mathfrak{N}^{(n)}\) and \(q^{(n)}\!:=\!p_{d}^{(n)}\) is the output dimension. By the definition of the weight, the \(p_{t}^{(n)}\) are polynomially bounded in \(n\). Let \(\lambda^{(n)}\in\mathbb{N}\) be a Lipschitz constant for all activation functions in \(\mathfrak{N}^{(n)}\). By the definition of the weight of an FNN and GNN we may choose \(\lambda^{(n)}\) polynomial in \(\operatorname{wt}(\mathfrak{N}^{(n)})\) and thus in \(n\).

For all \(k,n\in\mathbb{N}_{>0}\), we let \(\mathfrak{N}^{(n,k)}\) be a rational piecewise linear GNN with the same skeleton as \(\mathfrak{N}^{(n)}\) of size polynomial in \(\operatorname{wt}(\mathfrak{N}^{(n)})\), hence also polynomial in \(n\), and \(k\) such that all activation functions of \(\mathfrak{N}^{(n,k)}\) have Lipschitz constant at most \(2\lambda^{(n)}\), and for all graphs \(G\) of order \(n\) and all signals \(\boldsymbol{x}\in\mathscr{S}_{p}(G)\) it holds that

\[\left\|\mathfrak{\Re}^{(n)}(G,\boldsymbol{x})-\mathfrak{\Re}^{(n,k)}(G, \boldsymbol{x})\right\|_{\infty}\leq\frac{n^{d}}{k}\big{(}\left\|\boldsymbol{x }\right\|_{\infty}+1\big{)}.\] (6.AO)
Such an \(\mathfrak{N}^{(n,k)}\) exists by Lemma 6.15. Let \(\mathfrak{L}_{1}^{(n,k)},\ldots,\mathfrak{L}_{d}^{(n,k)}\) be the layers of \(\mathfrak{N}^{(n,k)}\).

We want to describe the GNNs \(\mathfrak{N}^{(n,k)}\) with built-in relations, using an encoding similar to the F-schemes. We cannot just use the same encoding as for the F-schemes because the non-uniform logic \(\mathsf{FO}+\mathsf{C}_{nu}\) does not allow for built-in numerical functions.6

Footnote 6: The reader may wonder why we do not simply allow for built-in function variables to avoid this difficulty. The reason is that then we could built terms whose growth is no longer polynomially bounded in the size of the input structure, which would make the logic too powerful. In particular, the logic would no longer be contained in \(\mathsf{TC}^{0}\).

Let \(t\in[d]\). In the following, we define the built-in relations that describe the \(t\) layers \(\mathfrak{L}_{t}^{(n,k)}\) of all the \(\mathfrak{N}^{(n,k)}\). We need to describe FNNs \(\mathfrak{F}_{\mathsf{msg}}^{(n,k)}\) and \(\mathfrak{F}_{\mathsf{comb}}^{(n,k)}\) for the message and combination functions of the layers, and in addition we need to describe the aggregation function. For the aggregation functions we use three relations \(A_{t}^{\mathsf{SUM}},A_{t}^{\mathsf{MEAN}},A_{t}^{\mathsf{MAX}}\subseteq \mathbb{N}^{2}\), where

\[A_{t}^{\mathsf{SUM}}\coloneqq\big{\{}(n,k)\in\mathbb{N}^{2}\,\big{|}\,\, \mathfrak{a}_{t}^{(n,k)}=\mathsf{SUM}\big{\}},\]

and \(A_{t}^{\mathsf{MEAN}}\) and \(A_{t}^{\mathsf{MAX}}\) are defined similarly. For each of the two FNNs \(\mathfrak{F}_{\mathsf{msg}}^{(n,k)}\) and \(\mathfrak{F}_{\mathsf{comb}}^{(n,k)}\) we use 18 relations. We only describe the encoding of \(\mathfrak{F}_{\mathsf{msg}}^{(n,k)}\) using relations \(M_{t}^{1},\ldots,M_{t}^{18}\). The encoding of \(\mathfrak{F}_{\mathsf{comb}}^{(n,k)}\) is analogous using a fresh set of 18 relations \(C_{t}^{1},\ldots,C_{t}^{18}\).

Say,

\[\mathfrak{F}_{\mathsf{msg}}^{(n,k)}=\Big{(}V^{(n,k)},E^{(n,k)},(\mathfrak{a}_ {v}^{(n,k)})_{v\in V^{(n,k)}},(w_{e}^{(n,k)})_{e\in E^{(n,k)}},(b_{v}^{(n,k)} )_{v\in V^{(n,k)}}\Big{)},\]

where without loss of generality we assume that \(V^{(n,k)}\) is an initial segment of \(\mathbb{N}\). We use relation \(M_{t}^{1}\subseteq\mathbb{N}^{3}\) and \(M_{t}^{2}\subseteq\mathbb{N}^{4}\) to describe the vertex set \(V\) and the edge set \(E\), letting

\[M_{t}^{1} \coloneqq\big{\{}(n,k,v)\,\big{|}\,v\in V^{(n,k)}\big{\}},\] \[M_{t}^{2} \coloneqq\big{\{}(n,k,v,w)\,\big{|}\,vw\in E^{(n,k)}\big{\}}.\]

As the bitsize of the skeleton of \(\mathfrak{N}^{(n)}\) and hence \(|V^{(n,k)}|\) is polynomially bounded in \(n\), there is an arithmetical term \(\theta_{V}(y,y^{\prime})\) such that for all graphs \(G\) and all assignments \(\omega\) we have

\[[\![\theta_{V}]\!]^{(G,\omega)}\,(n,k)=|V^{(n,k)}|\]

This term uses the constant \(\mathsf{ord}\) as well as the built-in relation \(M_{t}^{1}\). It does not depend on the graph \(G\) or the assignment \(\omega\).

For the weights, we use the relations \(M_{t}^{3},M_{t}^{4},M_{t}^{5}\subseteq\mathbb{N}^{5}\), letting

\[M_{t}^{3} \coloneqq\big{\{}(n,k,v,w,r)\,\big{|}\,(v,w)\in E^{(n,k)}\text{ with }w_{e}^{(n,k)}=(-1)^{r}2^{-s}m\big{\}},\] \[M_{t}^{4} \coloneqq\big{\{}(n,k,v,w,s)\,\big{|}\,(v,w)\in E^{(n,k)}\text{ with }w_{e}^{(n,k)}=(-1)^{r}2^{-s}m\big{\}},\] \[M_{t}^{5} \coloneqq\big{\{}(n,k,v,w,i)\,\big{|}\,(v,w)\in E^{(n,k)}\text{ with }w_{e}^{(n,k)}=(-1)^{r}2^{-s}m\text{ and }\operatorname{ Bit}(i,m)=1\big{\}}.\]

We always assume that \(w_{e}^{(n,k)}=(-1)^{r}2^{-s}m\) is the _canonical representation_ of \(w_{e}^{(n,k)}\) with \(r=0,m=0,s=0\) or \(r\in\{0,1\}\) and \(s=0\) and \(m\neq 0\) or \(r\in\{0,1\}\) and \(s\neq 0\) and \(m\neq 0\)
odd. As the bitsize of the numbers \(w_{e}^{(n,k)}\) is polynomial in \(k\), we can easily construct an arithmetical r-expression \(\boldsymbol{\rho}_{w}(y,y^{\prime},z,z^{\prime})\) such that for all graphs \(G\) and assignments \(\omega\) and \(n,k,v,w\in\mathbb{N}\) such that \((v,w)\in E^{(n,k)}\),

\[\langle\!\langle\boldsymbol{\rho}_{w}\rangle\!\rangle^{(G,\,\omega)}\,(n,k,v,w )=w_{e}^{(n,k)}.\]

This r-expression depends on the built-in relations \(M_{t}^{i}\), but not on \(G\) or \(\omega\).

Similarly, we define the relations \(M_{t}^{6},M_{t}^{7},M_{t}^{8}\subseteq\mathbb{N}^{4}\) for the biases and an r-expression \(\boldsymbol{\rho}_{b}(y,y^{\prime},z)\) such that for all graphs \(G\) and assignments \(\omega\) and \(n,k,v\in\mathbb{N}\) with \(v\in V^{(n,k)}\),

\[\langle\!\langle\boldsymbol{\rho}_{b}\rangle\!\rangle^{(G,\,\omega)}\,(n,k,v) =b_{v}^{(n,k)}.\]

To store the activation functions \(\mathfrak{a}_{v}^{(n,k)}\) we use the remaining ten relations \(M_{t}^{9}\subseteq\mathbb{N}^{4}\), \(M_{t}^{10},\ldots,M_{t}^{18}\subseteq\mathbb{N}^{5}\). The relation \(M_{t}^{9}\) is used to store the number \(m_{v}^{(n,k)}\) of thresholds:

\[M_{t}^{9}\coloneqq\{(n,k,v,m_{v}^{(n,k)})\,\big{|}\,v\in V^{(n,k)}\}.\]

As the bitsize of \(\mathfrak{a}_{v}^{(n,k)}\) is polynomial in \(k\), the number \(m_{v}^{(n,k)}\) is bounded by a polynomial in \(k\). Thus we can construct an arithmetical term \(\theta_{\mathfrak{a}}(y,y^{\prime},z)\) such that for all graphs \(G\), all assignments \(\omega\), all \(n,k\in\mathbb{N}\), and all \(v\in V^{(n,k)}\) we have

\[\llbracket\theta_{\mathfrak{a}}\rrbracket^{(G,\,\omega)}\,(n,k,v)=m_{v}^{(n,k)}\]

Of course this term needs to use the built-in relation \(M_{t}^{9}\). It does not depend on the graph \(G\) or the assignment \(\omega\).

The relations \(M_{t}^{10},N_{t}^{11},N_{t}^{12}\) are used to store thresholds. Say, the thresholds of \(\mathfrak{a}_{v}^{(n,k)}\) are \(t_{v,1}^{(n,k)}<\ldots<t_{v,m}^{(n,k)}\), where \(m=m_{v}^{(n,k)}\). We let

\[N_{t}^{10} \coloneqq\big{\{}(n,k,v,i,r)\,\big{|}\,v\in V^{(n,k)},1\leq i\leq m _{v}^{(n,k)}\text{ with }t_{v,i}^{(n,k)}=(-1)^{r}2^{-s}m\big{\}},\] \[N_{t}^{11} \coloneqq\big{\{}(n,k,v,i,s)\,\big{|}\,v\in V^{(n,k)},1\leq i\leq m _{v}^{(n,k)}\text{ with }t_{v,i}^{(n,k)}=(-1)^{r}2^{-s}m\big{\}},\] \[N_{t}^{12} \coloneqq\big{\{}(n,k,v,i,j)\,\big{|}\,v\in V^{(n,k)},1\leq i\leq m _{v}^{(n,k)}\text{ with }t_{v,i}^{(n,k)}=(-1)^{r}2^{-s}m\text{ and }\text{ Bit}(j,m)=1\big{\}}.\]

We always assume that \(t_{v,i}^{(n,k)}=(-1)^{r}2^{-s}m\) is the canonical representation of \(t_{v,i}^{(n,k)}\). As the bitsize of \(\mathfrak{a}_{v}^{(n,k)}\) is polynomial in \(k\), we can construct an arithmetical r-expression \(\boldsymbol{\rho}_{t}(y,y^{\prime},z,z^{\prime})\) such that for all graphs \(G\) and assignments \(\omega\) and \(n,k,v,i\in\mathbb{N}\) with \(v\in V^{(n,k)}\), \(i\in[m_{v}^{(n,k)}]\),

\[\langle\!\langle\boldsymbol{\rho}_{t}\rangle\!\rangle^{(G,\,\omega)}\,(n,k,v,i) =t_{v,i}^{(n,k)}.\]

Similarly, we use the relations \(N_{t}^{13},N_{t}^{14},N_{t}^{15}\) to represent the slopes of \(\mathfrak{a}_{v}^{(n,k)}\) and the relations \(N_{t}^{16},N_{t}^{17},N_{t}^{18}\) to represent the constants. Furthermore, we construct arithmetical r-expressions \(\boldsymbol{\rho}_{s}(y,y^{\prime},z,z^{\prime})\) and \(\boldsymbol{\rho}_{c}(y,y^{\prime},z,z^{\prime})\) to access them. We can combine the term \(\theta_{\mathfrak{a}}\) and the r-expressions \(\boldsymbol{\rho}_{t}(y,y^{\prime},z,z^{\prime})\), \(\boldsymbol{\rho}_{s}(y,y^{\prime},z,z^{\prime})\), \(\boldsymbol{\rho}_{c}(y,y^{\prime},z,z^{\prime})\) to an L-expression
\(\mathbf{\chi}(y,y^{\prime},z)\) such that for all graphs \(G\), all assignments \(\mathpzc{a}\), all \(n,k\in\mathbb{N}\), and all \(v\in V^{(n,k)}\) we have

\[\left\langle\!\left\langle\mathbf{\chi}\right\rangle\!\right\rangle^{(G,\mathpzc{a} )}(n,k,v)=\mathfrak{a}_{v}^{(n,k)}.\]

Then we can combine the term \(\theta_{V}\), the relation \(N_{t}^{E}\), the r-expressions \(\mathbf{\rho}_{w},\mathbf{\rho}_{b}\), and the L-expression \(\mathbf{\chi}\) to an F-expression \(\mathbf{\varphi}_{t}^{\mathsf{msg}}(y,y^{\prime})\) such that for all graphs \(G\), all assignments \(\mathpzc{a}\), and all \(n,k\in\mathbb{N}\) we have

\[\left\langle\!\left\langle\mathbf{\varphi}_{t}^{\mathsf{msg}}\right\rangle\! \right\rangle^{(G,\mathpzc{a})}(n,k)=\mathfrak{F}_{\mathsf{msg}}^{(n,k)}.\]

Similarly, we obtain an F-expression \(\mathbf{\varphi}_{t}^{\mathsf{comb}}(y,y^{\prime})\) for the combination function which uses the relation \(C_{t}^{1},\ldots,C_{t}^{18}\) that represent \(\mathfrak{F}_{\mathsf{comb}}^{(n,k)}\).

In addition to the \(\mathfrak{N}^{(n,k)}\), we also need access to the Lipschitz constants \(\lambda^{(n)}\). We use one more built-in relation

\[L\coloneqq\{(n,\lambda^{(n)})\,\big{|}\,n\in\mathbb{N}\}\subseteq\mathbb{N}^ {2}.\]

As \(\lambda^{(n)}\) is polynomially bounded in \(n\), there is a term \(\theta_{\lambda}(y)\) using the built-in relation \(L\) such that for all \(G\), \(\mathpzc{a}\) and all \(n\),

\[\left[\!\left[\theta_{\lambda}\right]\!\right]^{(G,\mathpzc{a})}(n)=\lambda^ {(n)}.\]

_Claim 1._ For all \(t\in[d]\), there is an arithmetical term \(\eta_{t}(y,y^{\prime})\) such that for all \(n,k\in\mathbb{N}\) and all \(G\), \(\mathpzc{a}\), the value \(\left[\!\left[\eta_{t}\right]\!\right]^{(G,\mathpzc{a})}(n,k)\) is a Lipschitz constant for the combination function \(\mathsf{comb}_{t}^{(n,k)}\) of \(\mathfrak{L}_{t}^{(n,k)}\).

_Proof._ Using Lemma 6.6 we can easily obtain such a Lipschitz constant using the fact that \(2\lambda^{(n)}\) is a Lipschitz constant for all activation functions of \(\mathsf{comb}_{t}^{(n,k)}\), and using the term \(\theta_{\lambda}(y)\) to access \(\lambda^{(n)}\) and the F-expression \(\mathbf{\varphi}_{t}^{\mathsf{comb}}(y,y^{\prime})\) to access the FNN for \(\mathsf{comb}_{t}^{(n,k)}\).

_Claim 2._ For all \(t\in[d]\), there is an arithmetical term \(\zeta_{t}(y,y^{\prime})\) such that for all \(n,k\in\mathbb{N}\), all \(G\), \(\mathpzc{a}\), and all \(\mathpzc{x},\mathpzc{x}^{\prime}\in\mathcal{S}_{p_{t-1}}(G)\),

\[\left|\!\left|\mathfrak{F}_{t}^{(n,k)}(G,\mathpzc{x})-\mathfrak{F}_{t}^{(n,k )}(G,\mathpzc{x})\right|\!\right|_{\infty}\leq\left[\!\left\langle\mathpzc{c} \right\rvert\!\right]^{(G,\mathpzc{a})}(n,k)n\left|\!\left|\mathpzc{x}- \mathpzc{x}^{\prime}\right|\!\right|_{\infty}\]

_Proof._ To construct this term we use Lemma 6.12.

Note that the bound \(\left[\!\left\langle\mathpzc{c}\right\rvert\!\right]^{(G,\mathpzc{a})}(n,k)\) is independent of the graph \(G\) and the assignment \(\mathpzc{a}\) and only depends on \(\mathfrak{F}_{t}^{(n,k)}\) and \(\lambda^{(n)}\).

The following claim is an analogue of Lemma 5.2.

_Claim 3._ Let \(t\in[d]\). Let \(\mathbf{X}\) be an r-schema of type \(\mathsf{vn}\to\mathtt{r}\), and let \(W\) be a function variable of type \(\varnothing\to\mathtt{n}\). Then there is a guarded r-expression \(\mathsf{l}\mathsf{-eval}_{t}(y,y^{\prime},x,y^{\prime\prime})\) such
that the following holds for all \(n,k\in\mathbb{N}\), all graphs \(G\), and all assignments \(\omega\) over \(G\). Let \(x\in\mathcal{S}_{p_{t-1}^{(n)}}(G)\) be the signal defined by

\[x(v)\coloneqq\left(\langle\!\langle X\rangle\!\rangle^{(G,a)}\,(v,0),\ldots, \langle\!\langle X\rangle\!\rangle^{(G,a)}\,(v,p_{t-1}^{(n)}-1)\right)\]

and let \(y\coloneqq\widetilde{\mathfrak{L}}_{t}^{(n,k)}(G,x)\in\mathcal{S}_{p_{t}^{(n) }}(G)\). Then for all \(v\in V(G)\),

\[\left\|y\!\!\langle v\rangle-\left(\langle\!\langle 1\text{-}\text{eval} \rangle\!\rangle^{(G,a)}\,(n,k,v,0),\ldots,\langle\!\langle 1\text{-}\text{eval} \rangle\!\rangle^{(G,a)}\,(n,k,v,p_{t}^{(n)}-1)\right)\right\|_{\infty}\leq 2^{-a \,(W)}\]

Proof.: The proof is completely analogous to the proof of Lemma 5.2, except that in the proof of Claims 1 and 2 we use Lemma 3.23 instead of Corollary 3.25 to evaluate the FNNs computing the message function and combination function. We substitute suitable instantiations of the r-expression \(\boldsymbol{\chi}(y,y^{\prime})\) and the L-expression \(\boldsymbol{\psi}(y,y^{\prime})\) for the r-schemas \(\boldsymbol{Z}_{v},\boldsymbol{Z}_{e}\) representing the parameters of the FNN and the L-schemas \(\boldsymbol{Y}_{v}\) representing the activation functions in Lemma 3.23.

Furthermore, in Case 3 of the proof of Lemma 5.2 (handling MEAN-aggregation) we need a term that defines a Lipschitz constant for the combination function of \(\mathfrak{L}_{t}^{(n,k)}\). (In the proof of Lemma 5.2, this is the constant \(\lambda\).) We can use the term \(\eta_{t}(y,y^{\prime})\) of Claim 1.

The next claim is the analogue of Theorem 5.1 for our setting with built-in relations.

_Claim 4._  Let \(\boldsymbol{X}\) be an r-schema of type \(\mathtt{vn\to r}\), and let \(W\) be a function variable of type \(\varnothing\to\mathtt{n}\). Then there is a guarded r-expression \(\mathsf{n\text{-}eval}(y,y^{\prime},x,y^{\prime\prime})\) such that the following holds for all \(n,k\in\mathbb{N}\), all graphs \(G\), and all assignments \(\omega\) over \(G\). Let \(\boldsymbol{x}\in\mathcal{S}_{p^{(n)}}(G)\) be the signal defined by

\[\boldsymbol{x}(v)\coloneqq\left(\langle\!\langle X\rangle\!\rangle^{(G,a)}\, (v,0),\ldots,\langle\!\langle X\rangle\!\rangle^{(G,a)}\,(v,p^{(n)}-1)\right)\!,\]

and let \(y\coloneqq\widetilde{\mathfrak{N}}^{(n,k)}(G,x)\in\mathcal{S}_{q^{(n)}}(G)\). Then for all \(v\in V(G)\),

\[\left\|y\!\!\langle v\rangle-\left(\langle\!\langle\mathsf{n\text{-}eval} \rangle\!\rangle^{(G,a)}\,(n,k,v,0),\ldots,\langle\!\langle\mathsf{n\text{-} eval}\rangle\!\rangle^{(G,a)}\,(n,k,v,q^{(n)}-1)\right)\right\|_{\infty}\leq 2^{-a \,(W)}\] (6.AP)

Proof.: The proof is completely analogous to the proof of Theorem 5.1, using Claim 3 instead of Lemma 5.2. In the proof of Theorem 5.1 we need access to term that defines a constant \(\lambda^{(t)}\) that bounds the growth of the \(t\)th layer of \(\mathfrak{N}^{(n,k)}\). We use the term \(\zeta_{t}(y,y^{\prime})\) of Claim 2.

What remains to be done is choose the right \(k\) to achieve the desired approximation error in (6.D). We will define \(k\) using a closed term err that depends on \(W,W^{\prime}\) as well as the order of the input graph. We let

\[\mathsf{err}\coloneqq 2\mathsf{ord}\cdot\left(W+1\right)\cdot W^{\prime}.\]
In the following, let us assume that \(G\) is a graph of order \(n\) and \(\omega\) is an assignment over \(G\) satisfying the two assumptions \(\omega(W^{\prime})\neq 0\) and \(\left\|\boldsymbol{x}\right\|_{\infty}\leq\omega(W)\) for the signal \(\boldsymbol{x}\in\mathcal{S}_{p^{(n)}}(G)\) defined by \(\boldsymbol{X}\) as in (6.C). Let

\[k\coloneqq\llbracket\mathsf{err}\rrbracket^{(G,\,a)}=2n(\omega(W)+1)\omega(W^{ \prime})\geq 2n\big{(}\left\|\boldsymbol{x}\right\|_{\infty}+1\big{)}\omega(W^{ \prime})\]

and thus

\[n\big{(}\left\|\boldsymbol{x}\right\|_{\infty}+1\big{)}\frac{1}{k}\leq\frac{1 }{2\omega(W^{\prime})}.\]

Thus by (6.AO),

\[\left\|\widetilde{\mathfrak{N}}^{(n)}(G,\boldsymbol{x})-\widetilde{\mathfrak{ N}}^{(n,k)}(G,\boldsymbol{x})\right\|_{\infty}\leq\frac{1}{2\omega(W^{ \prime})}.\] (6.AQ)

Now we let \(\mathsf{gnn-eval}(x,y)\) be the formula obtained from the formula \(\mathsf{gnn-eval}(y,y^{\prime},x,y^{\prime\prime})\) of Claim 4 by substituting \(\mathsf{ord}\) for \(y\), \(\mathsf{err}\) for \(y^{\prime}\), \(y\) for \(y^{\prime\prime}\), and \(W^{\prime}\) for \(W\). Then by (6.AP), we have

\[\left\|\widetilde{\mathfrak{N}}^{(n,k)}(G,\boldsymbol{x})(v)- \big{(}\left(\langle\mathsf{gnn-eval}\rangle\right)^{(G,\,a)}(v,0),\ldots, \langle\langle\mathsf{gnn-eval}\rangle\rangle^{(G,\,a)}(v,q^{(n)}-1)\big{)} \right\|_{\infty}\] \[\leq 2^{-a(W^{\prime})}\leq\frac{1}{2\omega(W^{\prime})}\]

Combined with (6.AQ), this yields

\[\left\|\widetilde{\mathfrak{N}}(G,\boldsymbol{x})(v)-\big{(}\left(\langle \mathsf{gnn-eval}\rangle\right)^{(G,\,a)}(v,0),\ldots,\langle\langle\mathsf{ gnn-eval}\rangle\rangle^{(G,\,a)}(v,q^{(n)}-1)\big{)}\right\|_{\infty}\leq\frac{1}{ \omega(W^{\prime})},\]

that is, the desired inequality (6.D). 

## 7 A Converse

The main result of this section is a converse of Corollary 6.3. For later reference, we prove a slightly more general lemma that not only applies to queries over labelled graphs, that is, graphs with Boolean signals, but actually to queries over graphs with integer signals within some range.

**Lemma 7.1**.: _Let \(U_{1},\ldots,U_{p}\) be function variables of type \(\mathtt{v}\to\mathtt{n}\), and let \(\varphi(x)\) be a \(\mathsf{GFO}+\mathsf{C}_{\mathrm{nu}}\)-formula that contains no relation or function variables except possibly the \(U_{i}\). Then there is a polynomial-size bounded-depth family \(\mathcal{N}\) of rational piecewise-linear GNNs of input dimension \(p\) such that for all graphs \(G\) of order \(n\) and all assignments \(\omega\) over \(G\) the following holds. Let \(\omega\in\mathcal{S}_{p}(G)\) be the signal defined by_

\[\omega(v)\coloneqq\big{(}\omega(U_{1})(v),\ldots,\omega(U_{p})(v)\big{)}. \tag{7.A}\]

_Assume that \(\omega(U_{i})(v)<n\) for all \(i\in[p]\) and \(v\in V(G)\). Then for all \(v\in V(G)\), \(\widetilde{\mathcal{N}}(G,\omega)\in\{0,1\}\) and_

\[\widetilde{\mathcal{N}}(G,\omega)(v)=1\iff(G,\omega)\vDash\varphi(v).\]

_Furthermore, all GNNs in \(\mathcal{N}\) only use \(\mathrm{lsig}\)-activations and \(\mathsf{SUM}\)-aggregation._
In the following, we will use the following more suggestive notation for the setting of the lemma: for a signal \(\iota\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}V(G)\to\{0,\ldots,n-1\}^ {p}\), we write

\[(G,\iota)\vDash\varphi(v)\]

if \((G,\iota)\vDash\varphi(v)\) for some assignments \(\iota\) with \(\iota(U_{i})(v)=\iota(v)_{i}\) for all \(i\in[p]\) and \(v\in V(G)\). This is reasonable because \(\varphi\) only depends on the assignment to \(U_{i}\) and to \(x\). Thus if \((G,\iota)\vDash\varphi(v)\) then \((G,\iota)\vDash\varphi(v)\) for all assignments \(\iota\) with with \(\iota(U_{i})(v)=\iota(v)_{i}\).

To prove Lemma 7.1, we need the following.

**Lemma 7.2**.: _Let \(m,n\in\mathbb{N}\). Then there is an FNN \(\mathfrak{F}\) of input dimension \(m\) and output dimension \(m\cdot n\) such that for all \(\vec{x}=(x_{0},\ldots,x_{m-1})\in\{0,\ldots,n-1\}^{m}\) the following holds. Suppose that \(\mathfrak{F}(\vec{x})=\vec{y}=(y_{0},\ldots,y_{mn-1})\). Then for \(k=in+j\), \(0\leq i<m,0\leq j<n\),_

\[y_{k}=\begin{cases}1&\text{if $j=x_{i}$},\\ 0&\text{otherwise}.\end{cases}\]

\(\mathfrak{F}\) _has size \(O(mn)\), depth \(2\), and it only uses \(\operatorname{lsig}\) activations._

**Example 7.3**.: Suppose that \(m=3,n=4\), and \(\vec{x}=(1,3,0)\). Then we want \(\mathfrak{F}(\vec{x})\) to be

\[(0,1,0,0,\ 0,0,0,1,\ 1,0,0,0).\]

Proof of Lemma 7.2.: Observe that the function \(f(x)=\operatorname{lsig}(x+1)-\operatorname{lsig}(x)\) satisfies \(f(0)=1\) and \(f(k)=0\) for all integers \(k\neq 0\). We design the network such that

\[y_{im+j}=\operatorname{lsig}(x_{i}-j+1)-\operatorname{lsig}(x_{i}-j).\]

On the middle layer we compute the values \(\operatorname{lsig}(x_{i}-j+1)\) and \(\operatorname{lsig}(x_{i}-j)\) for all \(i,j\). 

Proof of Lemma 7.1.: Let us fix an \(n\in\mathbb{N}\). We need to define a GNN \(\mathfrak{N}\) of size polynomial in \(n\) and of depth only depending on \(\varphi\), but not on \(n\), such that for every graph \(G\) of order \(n\), every signal \(\iota:V(G)\to\{0,\ldots,n-1\}^{p}\), and every \(v\in V(G)\) it holds that \(\mathfrak{G}(G,\iota)(v)\in\{0,1\}\) and

\[(G,\iota)\vDash\varphi(v)\iff\mathfrak{N}(G,\iota)(v)=1. \tag{7.B}\]

We will prove (7.B) by induction on the formula \(\varphi(x)\). Let us understand the structure of this formula. The formula uses two vertex variables \(x_{1},x_{2}\). We usually refer to them as \(x,x^{\prime}\), with the understanding that if \(x=x_{i}\) then \(x^{\prime}=x_{3-i}\). In addition, the formula uses an arbitrary finite set of number variables, say, \(\{y_{1},\ldots,y_{\ell}\}\). When we want to refer to any of these variables without specifying a particular \(y_{i}\), we use the notations like \(y,y^{\prime},y^{j}\).

By a slight extension of Lemma 3.2 to a setting where we allow function variables, but require them to take values smaller than the order of the input structure, all subterms of \(\varphi(x)\) are polynomially bounded in the order \(n\) of the input graph. Thus in every
counting subterm \(\#(x^{\prime},y^{1}<\theta_{1},\ldots,y^{k}<\theta_{k}).\big{(}E(x,x^{\prime}) \wedge\psi\big{)}\) or \(\#(y^{1}<\theta_{1},\ldots,y^{k}<\theta_{k}).\psi\), we may replace the \(\theta_{i}\) by a fixed closed term \(\theta\coloneqq(\mathsf{ord}+1)^{r}\) for some constant \(r\in\mathbb{N}\) and rewrite the counting terms as \(\#(x^{\prime},y^{1}<\theta,\ldots,y^{k}<\theta).\big{(}E(x,x^{\prime})\wedge y ^{1}<\theta_{1}\wedge\ldots\wedge y^{k}<\theta_{k}\wedge\psi\big{)}\) and \(\#(y^{1}<\theta,\ldots,y^{k}<\theta).\big{(}y^{1}<\theta_{1}\wedge\ldots\wedge y ^{k}<\theta_{k}\wedge\psi\big{)}\), respectively. We fix \(\theta\) for the rest of the proof and assume that it is used as the bound in all counting terms appearing in \(\varphi\).

Arguably the most important building blocks of the formula \(\varphi\) are subterms of the form

\[\#(x^{\prime},y^{1}<\theta,\ldots,y^{k}<\theta).\big{(}E(x,x^{\prime})\wedge \psi\big{)}. \tag{7.C}\]

Let us call these the _neighbourhood terms_ of \(\varphi\). Note that the only guards available in our vocabulary of graphs are \(E(x,x^{\prime})\) and \(E(x^{\prime},x)\), and since we are dealing with undirected graphs these two are equivalent. This is why we always assume that the guards \(\gamma\) of subterms \(\#(x^{\prime},y_{1}<\theta,\ldots,y_{k}<\theta).(\gamma\wedge\ldots)\) are of the form \(E(x,x^{\prime})\). Furthermore, we may assume that atoms \(E(x,x^{\prime})\) only appear as guards of neighbourhood terms. This assumption is justified by the observation that atoms \(E(x,x^{\prime})\) must always occur within some neighbourhood term (otherwise both \(x\) and \(x^{\prime}\) would occur freely in \(\varphi\)), and it would make no sense to have either \(E(x,x^{\prime})\) or its negation in the subformula \(\psi\) in (7.C) unless it appeared within a neighbourhood term of \(\psi\). We may also assume that \(\varphi\) has no atomic subformulas \(E(x,x)\), because they always evaluate to false (in the undirected simple graphs we consider), or \(x=x\), because they always evaluate to true.

In the following, we will use the term "expression" to refer to subformulas and subterms of \(\varphi\), and we denote expressions by \(\xi\). An _vertex-free expression_ is an expression with no free vertex variables. A _vertex expression_ is an expression with exactly one free vertex variable \(x\) (so \(\varphi=\varphi(x)\) itself is a vertex expression), and an _edge expression_ is an expression with two free vertex variables. This terminology is justified by the observation that edge expressions must be guarded, so the two free variables must be interpreted by the endpoints of an edge. The most important edge formulas are the formulas \(E(x,x^{\prime})\wedge\psi\) appearing within the neighbourhood terms.

To simplify the presentation, let us a fix a graph \(G\) of order \(n\) and a signal \(\omega:V(G)\to\{0,\ldots,n-1\}^{p}\) in the following. Of course the GNN we shall define will not depend on \(G\) or \(\omega\); it will only depend on \(n\) and \(\varphi\).

Recall that all subterms of \(\varphi\) take values polynomially bounded in \(n\). Let \(m\in\mathbb{N}\) be polynomial in \(n\) such that all subterms of \(\psi\) take values strictly smaller than \(m\). Let \(M\coloneqq\{0,\ldots,m-1\}\). When evaluating \(\varphi\), we only need to consider assignments that map number variables \(y_{1},\ldots,y_{\ell}\) appearing in \(\varphi\) to values in \(M\). Then every vertex-free formula \(\psi\) defines a relation \(S_{\psi}\subseteq M^{\ell+1}\) consisting of all tuples \((b,a_{1},\ldots,a_{\ell})\in M^{\ell+1}\) such that \((G,\omega)\vDash\psi\) for all assignments \(\omega\) over \(G\) with \(\omega(y_{j})=a_{j}\) for all \(j\in[\ell]\). (The first coordinate \(b\) is just a dummy coordinate that will allow us to work with \(\ell+1\)-ary relations throughout.) Every vertex-free term \(\theta\) defines a relation \(S_{\theta}\subseteq M^{\ell+1}\) consisting of all tuples \((b,a_{1},\ldots,a_{\ell})\in M^{\ell+1}\) such that \([\![\theta]\!]^{(G,\alpha)}=b\) for all assignments \(\omega\) over \(G\) with \(\omega(y_{j})=a_{j}\) for all \(j\in[\ell]\). Similarly, every vertex expression \(\xi\) defines a relation \(S_{\xi}(v)\subseteq M^{\ell+1}\) for every \(v\in V(G)\) and every edge expression \(\xi\) defines a relation \(S_{\xi}(v,v^{\prime})\subseteq M^{\ell+1}\) for every pair \((v,v^{\prime})\in V(G)^{2}\).
Let \(\widetilde{m}\coloneqq m^{\ell+1}\) and \(\widetilde{M}\coloneqq\{0,\ldots,\widetilde{m}-1\}\). Let \(\langle\cdot\rangle:M^{\ell+1}\to\widetilde{M}\) be the bijection defined by \(\left((a_{0},\ldots,a_{\ell})\right)=\sum_{i=0}^{\ell}a_{i}m^{i}\). Note that \(\langle\cdot\rangle\) maps relations \(R\subseteq M^{\ell+1}\) to subsets \(\langle R\rangle\subseteq\widetilde{M}\), which we may also view as vectors in \(\{0,1\}^{\widetilde{m}}\): for \(i\in\widetilde{m}\) we let \(\langle R\rangle_{i}=1\) if \(i\in\langle R\rangle\) and \(\langle R\rangle_{i}=0\) otherwise.

Thus every vertex-free expression \(\xi\) defines a vector \(\mathbf{x}_{\xi}\coloneqq\langle S_{\xi}\rangle\in\{0,1\}^{\widetilde{m}}\). Every vertex expression defines an \(\widetilde{m}\)-ary Boolean signal \(\mathbf{x}_{\xi}\) defined by

\[\mathbf{x}_{\xi}(v)\coloneqq\left\langle S_{\xi}(v)\right\rangle.\]

Every edge expression \(\xi\) defines an "edge signal" \(\mathbf{y}_{\xi}\) defined by

\[\mathbf{y}_{\xi}(v,v^{\prime})\coloneqq\left\langle S_{\xi}(v,v^{\prime})\right\rangle.\]

Note that, formally, \(\mathbf{y}_{\xi}(v,v^{\prime})\) is defined for all pairs \(v,v^{\prime}\) and not only for the pairs of endpoints of edges.

Assume that the vertex expressions are \(\xi^{(1)},\ldots,\xi^{(d)}\), ordered in such a way that if \(\xi^{(i)}\) is a subexpression of \(\xi^{(j)}\) then \(i<j\). Then \(\xi^{(d)}=\varphi\). Our GNN \(\mathfrak{N}\) will have \(d+1\) layers \(\mathfrak{L}^{(1)},\ldots,\mathfrak{L}^{(d+1)}\). For \(t\in[d]\), the layer \(\mathfrak{L}^{(t)}\) will have input dimension \(p_{t-1}\coloneqq p+(t-1)\widetilde{m}\) and output dimension \(p_{t}\coloneqq p+t\widetilde{m}\). Layer \(\mathfrak{L}^{(d+1)}\) will have input dimension \(p_{d}\) and output dimension \(p_{d+1}\coloneqq 1\). Note that \(p_{0}=p\). So our GNN \(\mathfrak{N}=(\mathfrak{L}^{(1)},\ldots,\mathfrak{L}^{(d+1)})\) will have input dimension \(p\) and output dimension \(1\), which is exactly what we need.

Let \(\mathbf{x}^{(0)}\coloneqq\mathbf{\iota}\) be the input signal, and for every \(t\in[d+1]\), let \(\mathbf{x}^{(t)}\coloneqq\widetilde{\mathfrak{L}}^{(t)}(G,\mathbf{x}^{(t-1)})\). We shall define the layers in such a way that for all \(t\in[d]\), all \(v\in V(G)\) we have

\[\mathbf{x}^{(t)}(v)=\mathbf{\iota}(v)\mathbf{x}_{\xi^{(1)}}(v)\ldots\mathbf{x}_{\xi^{(t)}}(v). \tag{7.D}\]

Furthermore,

\[\mathbf{x}^{(d+1)}(v)=\begin{cases}1&\text{if }G\coloneqq\varphi(v),\\ 0&\text{otherwise.}\end{cases} \tag{7.E}\]

So the GNN will take care of the vertex expressions. We also need to take care of the vertex-free expressions and the edge expressions. For vertex-free expressions, this is easy. Observe that a vertex-free expression contains no vertex variables at all, free or bound, because once we introduce a vertex variable the only way to bind it is by a neighbourhood term, and such a term always leaves one vertex variable free. Thus the value of a vertex-free expression does not depend on the input graph \(G\), but only on its order \(n\), the built-in numerical relations, and the integer arithmetic that is part of the logic. This means that for a vertex-free expression \(\xi\) the relation \(S_{\xi}\) is "constant" and can be treated like a built-in numerical relation that can be hardwired into the GNN. Dealing with edge expressions is more difficult. We will handle them when dealing with the neighbourhood terms.

Let us turn to the vertex expressions. Let \(t\in[d]\), and let \(\xi\coloneqq\xi^{(t)}\). We distinguish between several cases depending on the shape of \(\xi\).

_Case 1:_\(\xi=U_{i}(x)\) for some \(i\in[p]\).
Observe that for \(v\in V(G)\) we have

\[S_{\xi}(v)=\{\boldsymbol{\iota}(v)_{i}\}\times M^{\ell}.\]

Thus \(\boldsymbol{x}_{\xi}(v)_{k}=1\) if \(k=\boldsymbol{\iota}(v)_{i}+\sum_{j=1}^{\ell}a_{j}m^{j}\) for some \((a_{1},\ldots,a_{\ell})\in M^{\ell}\) and \(\boldsymbol{x}_{\xi}(v)_{k}=0\) otherwise. Using Lemma 7.2 in the special case \(m=1\), we can design an FNN \(\mathfrak{F}_{1}\) of input dimension \(1\) and output dimension \(m\) that maps \(\boldsymbol{\iota}(v)_{i}\) to \((0,\ldots,0,1,0,\ldots,0)\in\{0,1\}^{m}\) with a \(1\) at index \(\boldsymbol{\iota}(v)_{i}\). We put another layer of \(\widetilde{m}\) output nodes on top of this and connect the node \(k=\sum_{j=0}^{\ell}a_{j}m^{j}\) on this layer to the output node of \(\mathfrak{F}_{1}\) with index \(a_{0}\) by an edge of weight \(1\). All nodes have bias \(0\) and use lsig activations. The resulting FNN \(\mathfrak{F}_{2}\) has input dimension \(1\) and output dimension \(\widetilde{m}\), and it maps an input \(x\) to the vector \(\left\langle\{x\}\times M^{\ell}\right\rangle\in\{0,1\}^{\widetilde{m}}\). Thus in particular, it maps \(\boldsymbol{\iota}(v)_{i}\) to \(\left\langle S_{\xi}(v)\right\rangle\). Padding this FNN with additional input and output nodes, we obtain an FNN \(\mathfrak{F}_{3}\) of input dimension \(p_{t-1}+1\) and output dimension \(p_{t}=p_{t-1}+\widetilde{m}\) that map \((x_{1},\ldots,x_{p_{t-1}+1})\) to \((x_{1},\ldots,x_{p_{t-1}},\mathfrak{F}_{2}(x_{i}))\). We use \(\mathfrak{F}_{3}\) to define the combination functions \(\mathsf{comb}^{(t)}:\mathbb{R}^{p_{t-1}+1}\to\mathbb{R}^{p_{t}}\) of the layer \(\mathfrak{L}^{(t)}\). We define the message function by \(\mathsf{msg}^{(t)}:\mathbb{R}^{2p_{t-1}}\to\mathbb{R}\) by \(\mathsf{msg}^{(t)}(\boldsymbol{x})\succeq 0\) for all \(\boldsymbol{x}\), and we use sum aggregation. Then clearly, \(\mathfrak{L}^{(t)}\) computes the transformation \((G,\boldsymbol{x}^{(t-1)})\mapsto(G,\boldsymbol{x}^{(t)})\) satisfying (7.D).

_Case 2: \(\xi=\xi^{\prime}\ast\xi^{\prime\prime}\), where \(\xi^{\prime},\xi^{\prime\prime}\) are vertex expressions and either \(\ast\in\{+,\cdot,\leq\}\) and \(\xi,\xi^{\prime},\xi^{\prime}\) are terms or \(\ast=\wedge\) and \(\xi,\xi^{\prime},\xi^{\prime\prime}\) are formulas._

We could easily (though tediously) handle this case by explicitly constructing the appropriate FNNs, as we did in Case 1. However, we will give a general argument that will help us through the following cases as well. As the encoding \(R\subseteq M^{\ell+1}\mapsto\langle R\rangle\subseteq\widetilde{M}\) and the decoding \(\langle R\rangle\subseteq\widetilde{M}\mapsto R\subseteq M^{\ell+1}\) are easily definable by arithmetical \(\mathsf{FO}+\mathsf{C}\)-formulas, the transformation

\[\left(\langle S_{\xi^{\prime}}(v)\rangle,\langle S_{\xi^{\prime\prime}}(v) \rangle\right)\mapsto\langle S_{\xi^{\prime\ast}\xi^{\prime\prime}}(v)\rangle,\]

which can be decomposed as

\[\left(\langle S_{\xi^{\prime}}(v)\rangle,\langle S_{\xi^{\prime\prime}}(v) \rangle\right)\mapsto\left(S_{\xi^{\prime}}(v),S_{\xi^{\prime}}(v)\right) \mapsto S_{\xi\ast\xi^{\prime\prime}}(v)\mapsto\langle S_{\xi^{\prime}\ast\xi^ {\prime\prime}}(v)\rangle,\]

is also definable by an arithmetical \(\mathsf{FO}+\mathsf{C}\)-formula, using Lemmas 3.9 and 3.14 for the main step \(\left(S_{\xi^{\prime}}(v),S_{\xi^{\prime}}(v)\right)\mapsto S_{\xi\ast\xi^{ \prime\prime}}(v)\). Hence by Corollary 3.5, it is computable by a threshold circuit \(\mathfrak{C}^{\ast}\) of bounded depth (only depending on \(\ast\)) and polynomial size. Hence by Lemma 2.6 it is computable by an FNN \(\mathfrak{F}^{\ast}\) of bounded depth and polynomial size. Note that for every vertex \(v\), this FNN \(\mathfrak{F}^{\ast}\) maps \(\left(\boldsymbol{x}_{\xi^{\prime}}(v),\boldsymbol{x}_{\xi^{\prime}}(v)\right)\) to \(\boldsymbol{x}_{\xi}(v)\). We have \(\xi^{\prime}=\xi^{(t^{\prime})}\) and \(\xi^{\prime\prime}=\xi^{(t^{\prime\prime})}\) for some \(t^{\prime},t^{\prime\prime}<t\). Based on \(\mathfrak{F}^{\ast}\) we construct an FNN \(\mathfrak{F}\) of input dimension \(p_{t-1}+1\) and output dimension \(p_{t}=p_{t-1}+\widetilde{m}\) such that for \(\boldsymbol{u}\in\{0,\ldots,n-1\}^{p}\) and \(\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{t-1}\in\mathbb{R}^{\widetilde{m}}\), \(x\in\mathbb{R}\)

\[\mathfrak{F}(\boldsymbol{u}\boldsymbol{x}_{1}\ldots\boldsymbol{x}_{t-1}x)= \boldsymbol{u}\boldsymbol{x}_{1}\ldots\boldsymbol{x}_{t-1}\mathfrak{F}^{\ast} (\boldsymbol{x}_{t^{\prime}},\boldsymbol{x}_{t^{\prime\prime}}).\]
Continuing as in Case 1, we use \(\mathfrak{F}\) to define the combination functions \(\mathsf{comb}^{(t)}\) of the layer \(\mathfrak{L}^{(t)}\), and again we use a trivial message function.

_Case 3:_: \(\xi=\neg\xi^{\prime}\), where \(\xi^{\prime}\) is vertex formula.

This case can be handled as Case 2.

_Case 4:_: \(\xi=\xi^{\prime}\star\xi^{\prime\prime}\), where \(\xi^{\prime}\) is a vertex expression, \(\xi^{\prime\prime}\) is a vertex-free expression, and either \(\star\in\{+,\cdot,=,\leq\}\) and \(\xi,\xi^{\prime},\xi^{\prime}\) are terms or \(\star\in\wedge\) and \(\xi,\xi^{\prime},\xi^{\prime\prime}\) are formulas.

As in Case 2, we construct a threshold circuit \(\mathfrak{C}^{*}\) of bound depth and polynomial size that computes the mapping \(\big{(}\langle S_{\xi^{\prime}}(v)\rangle,\langle S_{\xi^{\prime\prime}} \rangle\big{)}\mapsto\langle S_{\xi^{\prime}\ast\xi^{\prime\prime}}(v)\rangle\). As we argued above, the relation \(S_{\xi}^{\prime\prime}\) for the vertex-free expression \(\xi^{\prime\prime}\) and hence the vector \(\big{\langle}S_{\xi^{\prime\prime}}\big{\rangle}\) do not depend on the input graph. Hence we can simply hardwire the \(\big{\langle}S_{\xi^{\prime\prime}}\big{\rangle}\) into \(\mathfrak{C}^{*}\), which gives us a circuit that computes the mapping \(\langle S_{\xi^{\prime}}(v)\rangle\mapsto\langle S_{\xi^{\prime}\ast\xi^{ \prime\prime}}(v)\rangle\). From this circuit we obtain an FNN \(\mathfrak{F}^{*}\) that computes the same mapping, and we can continue as in Case 2.

_Case 5:_: \(\xi=\#(y^{1}<\theta,\ldots,y^{k}<\theta).\psi\), where \(\psi\) is a vertex formula.

We argue as in Cases 2-4. We construct a threshold circuit that computes the mapping

\[\langle S_{\psi}(v)\rangle\mapsto\langle S_{\xi}(v)\rangle.\]

Turning this circuit into an FNN \(\mathfrak{F}^{*}\) that computes the same mapping, we can continue as in Case 2.

_Case 6:_: \(\xi=\#(x^{\prime},y^{1}<\theta,\ldots,y^{k}<\theta).\big{(}E(x,x^{\prime}) \wedge\psi\big{)}\), where \(\psi\) is an edge formula or a vertex formula.

Without loss of generality, we assume that \(\psi\) is an edge formula. If it is not, instead of \(\psi\) we take the conjunction of \(\psi\) with \(U_{1}(x)+U_{1}(x^{\prime})\geq 0\), which is always true, instead. Moreover, we may assume that \(\psi\) contains no equality atoms \(x=x^{\prime}\), because the guard \(E(x,x^{\prime})\) forces \(x\) and \(x^{\prime}\) to be distinct. Thus \(\psi\) is constructed from vertex expressions and vertex-free expressions using \(+,\cdot,\leq\) to combine terms, \(\neg,\wedge\) to combine formulas, and counting terms \(\#((y^{\prime})^{1}<\theta,\ldots,(y^{\prime})^{k^{\prime}}<\theta).\psi^{\prime}\). Let \(\xi^{(t_{1})},\ldots,\xi^{(t_{s})}\) be the maximal (with respect to the inclusion order on expressions) vertex expressions occurring in \(\psi\). Assume that \(x\) is the free vertex variable of \(\xi^{(t_{1})},\ldots,\xi^{(t_{r})}\) and \(x^{\prime}\) is the free vertex variable of \(\xi^{(t_{r+1})},\ldots,\xi^{(t_{s})}\). Arguing as in Case 2 and Case 5, we can construct a threshold circuit \(\mathfrak{C}\) of bounded depth and polynomial size that computes the mapping

\[\big{(}\langle S_{\xi^{(t_{1})}}(v)\rangle,\ldots,\langle S_{\xi^{(t_{r})}}(v )\rangle,\langle S_{\xi^{(t_{r+1})}}(v^{\prime})\rangle,\ldots,\langle S_{\xi _{i_{s}}}(v^{\prime})\rangle\big{)}\mapsto\langle S_{\psi}(v,v^{\prime})\rangle\]

To simplify the notation, let us assume that \(y^{i}=y_{i}\). Thus the free variables of \(\xi\) are among \(x,y_{k+1},\ldots,y_{\ell}\), and we may write \(\xi(x,y_{k+1},\ldots,y_{\ell})\). Let

\[\zeta(x,x^{\prime},y_{k+1},\ldots,y_{\ell})\coloneqq\#(y_{1}<\theta,\ldots,y_ {k}<\theta).\psi.\]
and observe that for all \(v\in V(G)\) and \(a_{k+1},\ldots,a_{\ell}\in M\) we have

\[\llbracket\xi\rrbracket^{G}\left(v,a_{k+1},\ldots,a_{\ell}\right)=\sum_{v^{\prime }\in N(v)}\llbracket\zeta\rrbracket^{G}\left(v,v^{\prime},a_{k+1},\ldots,a_{ \ell}\right).\]

For \(v,v^{\prime}\in V(G)\), let

\[R_{\zeta}(v,v)\coloneqq\{(a_{0},\ldots,a_{\ell-k-1},b)\;\big{|}\;b<\llbracket \zeta\rrbracket^{G}\left(v,v^{\prime},a_{0},\ldots,a_{\ell-k-1}\right)\} \subseteq M^{\ell-k+1},\]

and slightly abusing notation, let

\[\big{\langle}R_{\zeta}(v,v)\big{\rangle}=\left\{\sum_{i=0}^{\ell-k}a_{i}m^{i} \;\middle|\;(a_{0},\ldots,a_{\ell-k})\in R_{\zeta}(v)\right\}\subseteq\big{\{} 0,\ldots,m^{\ell-k+1}-1\big{\}}\]

which we may also view as a vector in \(\{0,1\}^{m^{\ell-k+1}}\). Again arguing via \(\mathsf{FO}+\mathsf{C}\), we can construct a threshold circuit \(\mathfrak{C}^{\prime}\) that computes the transformation

\[\big{(}\langle S_{\xi^{(t_{1})}}(v)\rangle,\ldots,\langle S_{\xi^{(tr)}}(v) \rangle,\langle S_{\xi^{(tr+1)}}(v^{\prime})\rangle,\ldots,\langle S_{\xi_{i_{ s}}}(v^{\prime})\rangle\big{)}\mapsto\langle R_{\zeta}(v,v^{\prime})\rangle.\]

Using Lemma 2.6, we can turn \(\mathfrak{C}^{\prime}\) into an FNN \(\mathfrak{F}^{\prime}\) that computes the same Boolean function. Let \(\boldsymbol{c}(v,v^{\prime})=(c_{0},\ldots,c_{m^{\ell-k}-1})\in M^{m^{\ell-k}}\) be the vector defined by as follows: for \((a_{0},\ldots,a_{\ell-k-1})\in M^{\ell-k}\) and \(j=\sum_{i=0}^{\ell-k-1}a_{i}m^{i}\) we let

\[c_{j}\coloneqq\llbracket\zeta\rrbracket^{G}\left(v,v^{\prime},a_{0},\ldots,a_ {\ell-k-1}\right).\]

Then

\[c_{j} =\big{|}\big{\{}b\;\big{|}\;b<\llbracket\zeta\rrbracket^{G} \left(v,v^{\prime},a_{0},\ldots,a_{\ell-k-1}\right)\big{\}}\big{|}\] \[=\big{|}\big{\{}b\;\big{|}\;(a_{0},\ldots,a_{\ell-k-1},b)\in R_{ \zeta}(v,v^{\prime})\big{\}}\big{|}\] \[=\sum_{beM}\langle R_{\zeta}(v,v^{\prime})\rangle_{j+bm^{\ell-k}}.\]

Thus an FNN of depth \(1\) with input dimension \(m^{\ell-k+1}\) and output dimension \(m^{\ell-k}\) can transform \(\langle R_{\theta}(v,v^{\prime})\rangle\) into \(\frac{1}{m}\boldsymbol{c}(v,v^{\prime})\). We take the factor \(\frac{1}{m}\) because \(0\leq c_{j}<m\) and thus \(\frac{1}{m}\boldsymbol{c}(v,v^{\prime})\in[0,1]^{m^{\ell-k}}\), and we can safely use lsig-activation. We add this FNN on top of the \(\mathfrak{F}^{\prime}\) and obtain an FNN \(\mathfrak{F}^{\prime\prime}\) that computes the transformation

\[\big{(}\langle S_{\xi^{(t_{1})}}(v)\rangle,\ldots,\langle S_{\xi^{(t_{r})}}(v )\rangle,\langle S_{\xi^{(t_{r+1})}}(v^{\prime})\rangle,\ldots,\langle S_{ \xi_{i_{s}}}(v^{\prime})\rangle\big{)}\mapsto\frac{1}{m}\boldsymbol{c}(v,v^{ \prime}).\]

The message function \(\mathsf{msg}^{(t)}\) of the GNN layer \(\mathfrak{L}^{(t)}\) computes the function

\[\big{(}x^{(t-1)}(v),x^{(t-1)}(v^{\prime})\big{)}\rightarrow\frac{1}{m} \boldsymbol{c}(v,v^{\prime})\]

which we can easily implement by an FNN based on \(\mathfrak{F}^{\prime\prime}\). Aggregating, we obtain the signal \(\varkappa\) such that

\[\varkappa(v)=\sum_{v^{\prime}\in N(v)}\mathsf{msg}^{(t)}(v,v^{\prime})=\frac{1 }{m}\sum_{v^{\prime}\in N(v)}\boldsymbol{c}(v,v^{\prime})\]
Thus for \((a_{0},\ldots,a_{\ell-k-1})\in M^{\ell-k}\) and \(j=\sum_{i=0}^{\ell-k-1}a_{i}m^{i}\) we have

\[\begin{split}\varkappa(v)_{j}&=\frac{1}{m}\sum_{v^{ \prime}\in N(v)}\mathbf{c}(v,v^{\prime})\\ &=\frac{1}{m}\sum_{v^{\prime}\in N(v)}\left[\zeta\right]^{G}(v,v ^{\prime},a_{0},\ldots,a_{\ell-k-1})\\ &=\frac{1}{m}\left[\xi\right]^{G}(v,a_{0},\ldots,a_{\ell-k-1}). \end{split}\]

Our final task will be to transform the vector \(\varkappa(v)\in[0,1]^{m^{\ell-k}}\) to the vector \(\left\langle S_{\xi}\right\rangle\in\{0,1\}^{\widebar{m}}\). In a first step, we transform \(\varkappa(v)\) into a vector \(\varkappa^{\prime}(v)\in M^{m^{\ell}}\) with entries

\[\varkappa^{\prime}(v)_{j}=\left[\xi\right]^{G}(v,a_{k+1},\ldots,a_{\ell})\]

for \((a_{1},\ldots,a_{\ell})\in M^{\ell}\) and \(j=\sum_{i=0}^{\ell-1}a_{i+1}m^{i}\). We need to transform \(\varkappa^{\prime}(v)\) into \(\left\langle S_{\xi}(v)\right\rangle\in\{0,1\}^{\widebar{m}}\), which for every \((a_{1},\ldots,a_{\ell})\in M^{\ell}\) has a single \(1\)-entry in position \(j=\sum_{i=0}^{\ell}a_{i}m^{i}\) for \(a_{0}=\left[\xi\right]^{G}(v,a_{k+1},\ldots,a_{\ell})\) and \(0\)-entries in all positions \(j^{\prime}=\sum_{i=0}^{\ell}a_{i}m^{i}\) for \(a_{0}\neq\left[\xi\right]^{G}(v,a_{k+1},\ldots,a_{\ell})\). We can use Lemma 7.2 for this transformation.

Thus we can construct an FNN \(\mathfrak{F}^{\prime\prime}\) that transforms the output \(\mathbf{z}(v)\) of the aggregation into \(\left\langle S_{\xi}(v)\right\rangle\). We define the combination function \(\mathsf{comb}^{(t)}:\mathbb{R}^{p_{t-1}+m^{\ell-k}}\to\mathbb{R}^{p_{t}}\) by \(\mathsf{comb}^{(t)}(\mathbf{x},\mathbf{z}):=(\mathbf{x},\mathfrak{F}^{\prime\prime}(\mathbf{z })).\) Then

\[\mathsf{comb}^{(t)}(\mathbf{x}(v),\varkappa(v)):=(\mathbf{x},\varkappa_{\xi}(v)).\]

Thus the layer \(\mathfrak{L}^{(t)}\) with message function \(\mathsf{msg}^{(t)}\) and combination function \(\mathsf{comb}^{(t)}\) satisfies (7.D).

All that remains is to define the last layer \(\mathfrak{L}^{(d+1)}\) satisfying (7.E). Since \(\xi^{(d)}=\varphi\), by (7.D) with \(t=d\), the vector \(\varkappa_{\varphi}(v)=\left\langle S_{\varphi}(v)\right\rangle\) is the projection of \(\mathbf{x}^{(d)}(v)\) on the last \(\widewidetilde{m}\) entries. As \(\varphi\) has no free number variables, we have

\[\left\langle S_{\varphi}(v)\right\rangle=\begin{cases}\mathbf{1}&\text{if }G\vDash \varphi(v),\\ \mathbf{0}&\text{if }G\vDash\varphi(v).\end{cases}\]

In particular, the last entry of \(\left\langle S_{\varphi}(v)\right\rangle\) and hence of \(\mathbf{x}^{(d)}(v)\) is \(1\) if \(G\vDash\varphi(v)\) and \(0\) otherwise. Thus all we need to do on the last layer is project the output on the last entry.

This completes the construction. 

The following theorem directly implies Theorem 1.1 stated in the introduction.

**Theorem 7.4**.: _Let \(\mathbb{G}\) be a unary query on \(\mathscr{GG}_{p}^{\mathrm{bool}}\). Then the following are equivalent:_

1. \(\mathbb{G}\) _is definable in_ \(\mathsf{GFO}+\mathsf{C}_{\mathrm{nu}}\)_._
_._
2. _There is a polynomial-size bounded-depth family of rational piecewise-linear GNNs using only_ \(\operatorname{\mathrm{isig}}\)_-activations and_ \(\mathsf{SUM}\)_-aggregation that computes_ \(\mathcal{G}\)_._
3. _There is a rpl-approximable polynomial weight bounded-depth family of GNNs that computes_ \(\mathcal{G}\)_._

Proof.: The implication \((1)\Longrightarrow(2)\) follows from Lemma 7.1 in the special case that the \(U_{i}\) are Boolean, that is, only take values in \(\{0,1\}\). We can then replace them by the unary relations \(P_{i}\) that we usually use to represent Boolean signals.

The implication \((2)\Longrightarrow(3)\) is trivial.

The implication \((3)\Longrightarrow(1)\) is Corollary 6.3. 

_Remark 7.5_.: As the earlier results, Lemma 7.1 and Theorem 7.4 also hold for GNNs with global readout and the logic \(\mathsf{GFO}+\mathsf{C}_{\mathrm{nu}}^{\mathrm{gc}}\). The proofs can easily be adapted.

_Remark 7.6_.: Let us finally address a question which we we already raised at the end of Section 5. Is every unary query definable in \(\mathsf{GFO}+\mathsf{C}\) computable by single rational piecewise linear or at least by an rpl approximable GNN? In other words: do we really need families of GNNs in Theorem 7.4, or could we just use a single GNN?

It has been observed in [26] that the answer to this question is no. Intuitively, the reason is that GNNs cannot express "alternating" queries like nodes having an even degree. To prove this, we analyse the behaviour of GNNs on stars \(S_{n}\) with \(n\) leaves, for increasing \(n\). The signal at the root node that the GNN computes is approximately piecewise polynomial as a function of \(n\). However, a function that is \(1\) for all even natural numbers \(n\) and \(0\) for all odd numbers is very far from polynomial.

## 8 Random Initialisation

A GNN with random initialisation receives finitely many random features together with its \(p\)-dimensional input signal. We assume that the random features are chosen independently uniformly from the interval \([0,1]\). We could consider other distributions, like the normal distribution \(N(0,1)\), but in terms of expressiveness this makes no difference, and the uniform distribution is easiest to analyse. The random features at different vertices are chosen, independently. As in [2], we always assume that GNNs with random initialisation have global readout.7

Footnote 7: There is no deeper reason for this choice, it is just that the results get cleaner this way. This is the same reason as in [2].

We denote the uniform distribution on \([0,1]\) by \(\mathcal{U}_{[0,1]}\), and for a graph \(G\) we write \(\boldsymbol{r}\sim\mathcal{U}_{[0,1]}^{r\times V(G)}\) to denote that \(\boldsymbol{r}\in\mathcal{S}_{r}(G)\) is obtained by picking the features \(\boldsymbol{r}(v)_{i}\) independently from \(\mathcal{U}_{[0,1]}\). Moreover, for a signal \(\boldsymbol{x}\in\mathcal{S}_{p}(G)\), by \(\boldsymbol{x}\boldsymbol{r}\) we denote the \((p+r)\)-dimensional signal with \(\boldsymbol{x}\boldsymbol{r}(v)=\boldsymbol{x}(v)\boldsymbol{r}(v)\). Formally, a \((p,q,r)\)_-dimensional GNN with \(\mathfrak{N}\)_ with global readout of input dimension \(p+r\) and output dimension \(q\). It computes a random variable mapping pairs \((G,\boldsymbol{x})\in\mathcal{F}\mathcal{S}_{p}\) to the space \(\mathcal{S}_{q}(G)\), which we view as
a product measurable space \(\mathbb{R}^{q\times V(G)}\) equipped with a Borel \(\sigma\)-algebra (or Lebesgue \(\sigma\)-algebra, this does not matter here). Abusing (but hopefully also simplifying) notation, we use \(\mathfrak{R}\) to denote a GNN that we interpret as a GNN with random initialisation, and we use \(\widetilde{\mathfrak{R}}\) to denote the random variable. Sometimes it is also convenient to write \(\mathfrak{R}(G,\mathpzc{x}))\coloneqq(G,\widetilde{\mathfrak{R}}(x))\). It is not hard to show that the mapping \(\widetilde{\mathfrak{R}}\) is measurable with respect to the Borel \(\sigma\)-algebras on the product spaces \(\mathcal{S}_{p}(G)=\mathbb{R}^{p+r\times V(G)}\) and \(\mathcal{S}_{q}(G)=\mathbb{R}^{q\times V(G)}\). Here we use that the activation functions of \(\mathfrak{R}\) are continuous. To define the probability distribution of the random variable \(\widetilde{\mathfrak{R}}\), for all \((G,\mathpzc{x})\in\mathscr{S}_{p}\) and all events (that is, measurable sets) \(\mathcal{Y}\subseteq\mathcal{S}_{q}(G)\) we let

\[\Pr\big{(}\widetilde{\mathfrak{R}}(G,\mathpzc{x})\in\mathcal{Y}\big{)} \coloneqq\Pr_{r\sim\mathcal{Y}_{[0,1]}^{r\times V(G)}}\big{(}\widetilde{ \mathfrak{R}}(G,\mathpzc{x}\mathpzc{r})\in\mathcal{Y}\big{)}, \tag{8.A}\]

where on the left-hand side we interpret \(\mathfrak{R}\) as a \((p,q,r)\)-dimensional GNN with ri and on the right-hand side just as an ordinary GNN of input dimension \((p+r)\) and output dimension \(q\).

For a \((p,q,r)\)-dimensional GNN with ri we call \(p\) the _input dimension_, \(q\) the _output dimension_, and \(r\) the _randomness dimension_.

Let \(\mathscr{G}\) be a unary query on \(\mathscr{S}\mathcal{G}_{p}^{\mathrm{bool}}\). We say that a GNN with ri \(\mathfrak{R}\) of input dimension \(p\) and output dimension \(1\)_computes_\(\mathscr{G}\) if for all \((G,\mathscr{G})\in\mathscr{S}\mathcal{G}_{p}^{\mathrm{bool}}\) and all \(v\in V(G)\) it holds that

\[\begin{cases}\Pr\big{(}\widetilde{\mathfrak{R}}(G,\mathscr{G})\geq\frac{2}{3 }\big{)}\geq\frac{2}{3}&\text{if }\mathscr{G}(G,\mathscr{G})=1,\\ \Pr\big{(}\widetilde{\mathfrak{R}}(G,\mathscr{G})\leq\frac{1}{3}\big{)}\geq \frac{2}{3}&\text{if }\mathscr{G}(G,\mathscr{G})=0.\end{cases}\]

We can easily extend this definition to families \(\mathscr{R}=(\mathfrak{R}^{(n)})_{n\in\mathbb{N}}\) of GNNs with ri.

By a fairly standard probability amplification result, we can make the error probabilities exponentially small.

**Lemma 8.1**.: _Let \(\mathscr{G}\) be a unary query over \(\mathscr{S}\mathcal{S}_{p}\) that is computable by family \(\mathscr{R}=(\mathfrak{R}^{(n)})_{n\in\mathbb{N}}\) of GNNs with ri, and let \(\pi(X)\) be a polynomial. Then there is a family \(\mathscr{R}^{\prime}=(\mathfrak{R}^{\prime})^{(n)})_{n\in\mathbb{N}}\) of GNNs with ri such that the following holds._

1. \(\mathscr{R}^{\prime}\) _computes_\(\mathscr{G}\)_, and for every_ \(n\)_, every_ \((G,\mathscr{\ell})\in\mathscr{S}\mathcal{G}_{p}^{\mathrm{bool}}\) _of order_ \(n\)_, and every_ \(v\in V(G)\)_,_ \[\begin{cases}\Pr\big{(}\widetilde{\mathscr{R}}^{\prime}(G,\mathscr{\ell})=1 \big{)}\geq 1-2^{-\pi(n)}&\text{if }\mathscr{G}(G,\mathscr{\ell})=1,\\ \Pr\big{(}\widetilde{\mathscr{R}}^{\prime}(G,\mathscr{\ell})=0\big{)}\geq 1-2^{-\pi(n)}& \text{if }\mathscr{G}(G,\mathscr{\ell})=0.\end{cases}\] (8.B)
2. _The weight of_ \((\mathfrak{R}^{\prime})^{(n)}\) _is polynomially bounded in the weight of_ \(\mathfrak{R}^{(n)}\) _and_ \(n\)_._
3. _The depth of_ \((\mathfrak{R}^{\prime})^{(n)}\) _is at most the depth of_ \(\mathfrak{R}^{(n)}\) _plus_ \(2\)_._
4. _The randomness dimension of_ \((\mathfrak{R}^{\prime})^{(n)}\) _is polynomially bounded in_ \(n\) _and the randomness dimension of_ \(\mathfrak{R}^{(n)}\)_._
Proof.: We just run sufficiently many copies of \(\mathscr{R}\) in parallel (polynomially many suffice) and then take a majority vote in the end, which is responsible for one of the additional layers. This way we obtain a family \(\mathscr{R}^{\prime\prime}\) that achieves

\[\begin{cases}\Pr\left(\mathscr{R}^{\prime\prime}(G,\mathscr{O})\geq\frac{2}{3} \right)\geq 1-2^{-\pi(n)}&\text{if }\mathscr{O}(G,\mathscr{O})=1,\\ \Pr\left(\mathscr{R}^{\prime}(G,\mathscr{O})\leq\frac{1}{3}\right)\geq 1-2^{-\pi(n)}& \text{if }\mathscr{O}(G,\mathscr{O})=0.\end{cases}\]

To get the desired \(0,1\)-outputs, we apply the transformation \(\operatorname{lsig}(3x-1)\) to the output on an additional layer. 

**Lemma 8.2**.: _Let \(\mathscr{O}\) be a unary query over \(\mathscr{SG}_{p}\) that is computable by an rpl-approximable polynomial-weight, bounded-depth family \(\mathscr{R}\) of GNNs with ri. Then there is an order-invariant \(\mathsf{GFO}+\mathsf{C}_{\operatorname{nn}}^{\operatorname{gc}}\)-formula that defines \(\mathscr{O}\)._

Proof.: Suppose that \(\mathscr{R}=(\mathfrak{R}^{(n)})_{n\in\mathbb{N}}\), and for every \(n\), let \(r^{(n)}\) be the randomness dimension of \(\mathfrak{R}^{(n)}\). Viewed as a standard GNN, \(\mathfrak{R}^{(n)}\) has input dimension \(p+r^{(n)}\) and output dimension \(1\). By the previous lemma, we may assume that our family satisfies (8.B) for \(\pi(X)=3pX^{3}\).

Our first step is to observe that we can safely truncate the random numbers, which we assume to be randomly chosen from \([0,1]\), to \(O(n)\) bits. This follows easily from Corollary 6.13. Truncating the numbers to \(cn\) bits means that we replace the random signal \(\boldsymbol{\nu}\) by a \(\boldsymbol{\nu}^{\prime}\) such that \(\left\|\boldsymbol{\nu}-\boldsymbol{\nu}^{\prime}\right\|_{\infty}\leq 2^{-cn}\), and the corollary implies that if we choose \(c\) sufficiently large, we will approximate the original GNN up to an additive error of \(\frac{1}{10}\). Thus for some constant \(c\) we may assume that the random strings are not drawn uniformly from \([0,1]\), but from the set

\[U_{n}\coloneqq\left\{\left.\sum_{i=0}^{cn-1}a_{i}2^{-i-1}\ \right|\ a_{0},\ldots,a_{cn-1}\in\{0,1\}\right\}.\]

Let us denote the uniform distribution on this set by \(\mathscr{U}_{n}\) Hence for for every \(n\), every \((G,\mathscr{O})\in\mathscr{SG}_{p}^{\operatorname{bool}}\) of order \(n\), and every \(v\in V(G)\),

\[\begin{cases}\Pr_{\boldsymbol{\nu}\sim\mathscr{U}_{n}^{r(n)}\times V(G)}\left( \mathscr{R}(G,\mathscr{O})\geq\frac{9}{10}\right)\geq 1-2^{-\pi(n)}&\text{if } \mathscr{O}(G,\mathscr{O})=1,\\ \Pr_{\boldsymbol{\nu}\sim\mathscr{U}_{n}^{r(n)}\times V(G)}\left(\mathscr{R} (G,\mathscr{O})\leq\frac{1}{10}\right)\geq 1-2^{-\pi(n)}&\text{if } \mathscr{O}(G,\mathscr{O})=0.\end{cases} \tag{8.C}\]

Next, we want to apply Theorem 6.2 in the version for GNNs with global readout and the logic \(\mathsf{GFO}+\mathsf{C}_{\operatorname{nn}}^{\operatorname{gc}}\). Let \(\boldsymbol{X}_{r}\) be an r-schema of type \(\mathsf{v}\mathsf{n}\to\mathsf{r}\). Suppose that \((G,\mathscr{O})\in\mathscr{SG}_{p}^{\operatorname{bool}}\), and let \(\mathscr{L}\) be an assignment over \(G\). We view \((G,\mathscr{O})\) as a \(p\)-labelled graph here, that is, as an \(\{E,P_{1},\ldots,P_{p}\}\)-structure. So \(((G,\mathscr{O}),\mathscr{L})\) is the pair consisting of this structure together with the assignment \(\mathscr{L}\). Let

\[\boldsymbol{\nu}(v)\coloneqq\Big{(}\langle\!\langle\boldsymbol{X}_{r}\rangle\! \rangle^{((G,\mathscr{O}),\mathscr{L})}\,(v,0),\ldots,\langle\!\langle \boldsymbol{X}_{r}\rangle\!\rangle^{((G,\mathscr{O}),\mathscr{L})}\,(v,r^{(n) }-1)\Big{)}. \tag{8.D}\]

In the following we will write \((G,\mathscr{O},\boldsymbol{\nu})\) instead of \(((G,\mathscr{O}),\mathscr{L})\) if \(\boldsymbol{\nu}\) is obtained from some assignment \(\mathscr{L}\) via (8.D), always assuming that \(\langle\!\langle\boldsymbol{X}_{r}\rangle\!\rangle^{((G,\mathscr{O}),\mathscr{ L})}\,(v,k)=0\) for \(k\geq r^{(n)}\).
Then if \(\varphi(x)\) is a formula whose only free variables are among \(x\) and the relation and function variables appearing in \(\mathbf{X}_{r}\), the value \(\llbracket\varphi\rrbracket^{((G,\mathscr{G}),\mathpzc{a})}\) only depends on \((G,\mathscr{E})\), \(\mathpzc{r}\) and \(v\coloneqq\mathpzc{a}(x)\), and we may ignore the rest of \(\mathpzc{a}\). In particular, we may write \((G,\mathscr{E},\mathpzc{r})\vDash\varphi(v)\) instead of \(\llbracket\varphi\rrbracket^{(G,\mathscr{G},\mathpzc{a})}=1\).

With is notation at hand, let us apply Theorem 6.2. Replacing \(W\) by the constant \(1\) and \(W^{\prime}\) by \(10\), we obtain an r-expression \(\mathbf{\rho}\) in \(\mathsf{GFO}+\mathsf{C}^{\mathrm{gc}}\) such that for all \((G,\mathscr{E})\in\mathscr{GF}_{p}\) of order \(n\) and \(\mathpzc{r}\in\mathscr{S}_{r^{(n)}}(G)\) defined as in (8.D), if \(\left\lVert\mathpzc{r}\right\rVert_{\infty}\leq 1\), then for all \(v\in V(G)\) we have

\[\left\lvert\tilde{\mathscr{R}}(G,\mathscr{E},\mathpzc{r})(v)-\left\langle \!\langle\mathbf{\rho}\rangle\!\right\rangle^{(G,\mathscr{E},\mathpzc{r})}(v) \right\rvert\leq\frac{1}{10}.\]

Note that if \(\mathpzc{r}(v)_{i}\in U_{n}\) for \(0\leq i<r^{(n)}\), then we have \(\left\lVert\mathpzc{r}\right\rVert_{\infty}\leq 1\). Hence,

\[\begin{cases}\Pr_{\mathpzc{r}\sim\mathpzc{M}_{n}^{r^{(n)}\times V(G)}}\big{(} \left\langle\!\langle\mathbf{\rho}\rangle\!\right\rangle^{(G,\mathscr{E},\mathpzc{ r})}(v)\geq\frac{8}{10}\big{)}\geq 1-2^{-\pi(n)}&\text{if }\mathscr{G}(G,\mathscr{E})=1,\\ \Pr_{\mathpzc{r}\sim\mathpzc{M}_{n}^{r^{(n)}\times V(G)}}\big{(}\left\langle \!\langle\mathbf{\rho}\rangle\!\right\rangle^{(G,\mathscr{E},\mathpzc{r})}(v)\leq \frac{2}{10}\big{)}\geq 1-2^{-\pi(n)}&\text{if }\mathscr{G}(G,\mathscr{E})=0.\end{cases}\]

From \(\mathbf{\rho}\) we easily obtain a formula \(\varphi(x)\), just saying \(\mathbf{\rho}\geq\frac{1}{2}\), such that

\[\Pr_{\mathpzc{r}\sim\mathpzc{M}_{n}^{r^{(n)}\times V(G)}}\big{(}\left\llbracket \varphi\rrbracket^{(G,\mathscr{E},\mathpzc{r})}(v)=\mathscr{G}(G,\mathscr{E} )(v)\right)\geq 1-2^{-\pi(n)}. \tag{8.E}\]

Our next step will be to simplify the representation of the random features in the signal \(\mathpzc{r}\in U_{n}^{r^{(n)}\times V(G)}\). Each number in \(U_{n}\) can be described by a subset of \(\{0,\ldots,cn-1\}\): the set \(S\) represents the number \(\sum_{s\in S}2^{-s-1}\). Thus we can represent \(\mathpzc{r}\in U_{n}^{r^{(n)}\times V(G)}\) by a relation \(R\subseteq V(G)\times\{0,\ldots,r^{(n)}-1\}\times\{0,\ldots,cn-1\}\), and we can transform \(\varphi(x)\) into a formula \(\varphi^{\prime}(x)\) that uses a relation variable \(X_{r}\) of type \(\{\texttt{vnn}\}\) instead of the r-schema \(\mathbf{X}_{r}\) such that

\[\Pr_{R}\Big{(}\left\llbracket\varphi^{\prime}\right\rrbracket^{(G,\mathscr{E},R)}(v)=\mathscr{G}(G,\mathscr{E})(v)\Big{)}\geq 1-2^{-\pi(n)}, \tag{8.F}\]

where the probability is over all \(R\) chosen uniformly at random from \(V(G)\times\{0,\ldots,r^{(n)}-1\}\times\{0,\ldots,cn-1\}\) and \((G,\mathscr{E},R)\) is \(\big{(}(G,\mathscr{E}),\mathpzc{a}\big{)}\) for some assignment \(\mathpzc{a}\) with \(\mathpzc{a}(X_{r})=R\).

The next step is to introduce a linear order and move towards an order invariant formula. We replace the relation variable \(X_{r}\) of type \(\{\texttt{vnn}\}\) by a relation variable \(Y_{r}\) of type \(\{\texttt{nnn}\}\), and we introduce a linear order \(\leqslant\) on \(V(G)\). Then we replace atomic subformulas \(X_{r}(x^{\prime},y,y^{\prime})\) of \(\varphi^{\prime}(x)\) by

\[\exists y^{\prime\prime}\leq\mathsf{ord}\big{(}\#x.x\leqslant x^{\prime}=\# y^{\prime\prime\prime}\leq\mathsf{ord}.y^{\prime\prime\prime}\leq y^{\prime \prime}\wedge Y_{r}(y^{\prime\prime},y,y^{\prime})\big{)}\]

The equation \(\#x.x\leqslant x^{\prime}=\#y^{\prime\prime\prime}\leq\mathsf{ord}.y^{\prime \prime\prime}\leq y^{\prime\prime}\) just says that \(x^{\prime}\) has the same position in the linear order \(\leqslant\) on \(V(G)\) as \(y^{\prime\prime}\) has in the natural linear order \(\leqslant\). So basically, we store the random features for the \(i\)th vertex in the linear order \(\leqslant\) in the \(i\)-entry of \(Y_{r}\). We obtain a new formula \(\varphi^{\prime\prime}(x)\) satisfying

\[\Pr_{R}\Big{(}\left\llbracket\varphi^{\prime\prime}\right\rrbracket^{(G, \mathscr{E},\leqslant,R)}(v)=\mathscr{G}(G,\mathscr{E})(v)\Big{)}\geq 1-2^{-\pi(n)}, \tag{8.G}\]
where the probability is over all \(R\) chosen uniformly at random from \(\{0,\ldots,n-1\}\times\{0,\ldots,r^{(n)}-1\}\times\{0,\ldots,cn-1\}\). Importantly, this holds for all linear orders \(\leqslant\) on \(V(G)\). Thus in some sense, the formula is order invariant, because \(\mathscr{O}(G,\mathscr{O})(v)\) does not depend on the order. However, the set of \(R\subseteq\{0,\ldots,n-1\}\times\{0,\ldots,r^{(n)}-1\}\times\{0,\ldots,cn-1\}\) for which we have \(\llbracket\varphi^{\prime\prime}\rrbracket^{(G,\mathscr{O},\leqslant,R)}(v)= \mathscr{O}(G,\mathscr{O})(v)\) may depend on \(\leqslant\); (8.G) just says that this set contains all \(R\) except for an exponentially small fraction.

In the final step, we apply a standard construction to turn randomness into non-uniformity, which is known as the "Adleman trick". It will be convenient to let

\[\Omega\coloneqq 2^{\{0,\ldots,n-1\}\times\{0,\ldots,r^{(n)}-1\}\times\{0, \ldots,cn-1\}}.\]

This is the sample space from which we draw the relations \(R\) uniformly at random. By (8.G), for each triple \((G,\mathscr{O},\leqslant)\) consisting of a graph \(G\) of order \(n\), a signal \(\mathscr{O}\in\mathcal{S}_{p}^{\mathrm{bool}}(G)\), and a linear order \(\leqslant\) on \(V(G)\) there is a "bad" set \(B(G,\mathscr{O},\leqslant)\subseteq\Omega\) such that

\[\forall R\not\in B(G,\mathscr{O},\leqslant):\,\llbracket\varphi^{\prime\prime }\rrbracket^{(G,\mathscr{O},\leqslant,R)}(v)=\mathscr{O}(G,\mathscr{O})(v) \tag{8.H}\]

and

\[\frac{\big{|}B(G,\mathscr{O},\leqslant)\big{|}}{|\Omega|}\leq 2^{-\pi(n)}. \tag{8.I}\]

Observe that the number of triples \((G,\mathscr{O},\leqslant)\) is bounded from above by \(2^{n^{2}+pn+n\log n}\) and that \(n^{2}+pn+n\log n<\pi(n)\). Thus we have

\[\frac{\big{|}\bigcup_{(G,\mathscr{O},\leqslant)}B(G,\mathscr{O},\leqslant) \big{|}}{|\Omega|}\leq\sum_{(G,\mathscr{O},\leqslant)}\frac{\big{|}B(G, \mathscr{O},\leqslant)\big{|}}{|\Omega|}\leq 2^{n^{2}+pn+n\log n}\cdot 2^{-\pi(n)}<1.\]

This means that there is a \(R^{(n)}\in\Omega\setminus\bigcup_{(G,\mathscr{O},\leqslant)}B(G,\mathscr{O}, \leqslant)\), and by (8.H) we have

\[\llbracket\varphi^{\prime\prime}\rrbracket^{(G,\mathscr{O},\leqslant,R^{(n)}) }(v)=\mathscr{O}(G,\mathscr{O})(v) \tag{8.J}\]

for all graphs \(G\) of order \(n\), signals \(\mathscr{O}\in\mathcal{S}_{p}^{\mathrm{bool}}(G)\), and linear orders \(\leqslant\) on \(V(G)\). Let \(R^{*}\coloneqq\bigcup_{n\in\mathbb{N}}\{n\}\times R^{(n)}\subseteq\mathbb{N}^ {4}\). We use \(R^{*}\) as built-in numerical relation (in addition to the numerical relations already in \(\varphi^{\prime\prime}\)). In \(\varphi^{\prime\prime}(x)\) we replace atomic subformulas \(Y_{r}(y,y^{\prime},y^{\prime\prime})\) by \(R^{*}(\mathsf{ord},y,y^{\prime},y^{\prime\prime})\). Then it follows from (8.J) that the resulting formula is order-invariant and defines \(\mathscr{O}\). 

Before we prove the converse of the previous lemma, we need another small technical lemma about FNNs.

**Lemma 8.3**.: _Let \(k,n\in\mathbb{N}_{>0}\). There is a rational piecewise linear FNN \(\mathfrak{F}\) of input and output dimension \(1\), size polynomial in \(n\) and \(k\) such that_

\[\Pr_{r\sim\mathcal{U}}\big{(}\mathfrak{F}(r)\in\{0,\ldots,n-1\}\big{)}\geq 1-2 ^{-k},\]
_and for all \(i\in\{0,\ldots,n-1\}\),_

\[\frac{1-2^{-k}}{n}\leq\Pr_{r\sim\mathcal{U}}\left(\mathfrak{F}(r)=i\right)\leq \frac{1}{n}.\]

_Furthermore, \(\mathfrak{F}\) only uses \(\operatorname{relu}\)-activations._

Proof.: For all \(a\in\mathbb{R}\) and \(\varepsilon>0\), let \(f_{\varepsilon,a}:\mathbb{R}\to\mathbb{R}\) be defined by \(f_{\varepsilon,a}(x)\coloneqq\operatorname{lsig}\left(\frac{1}{\varepsilon}x -\frac{a}{\varepsilon}\right)\). Then

\[\begin{cases}f_{\varepsilon,a}(x)=0&\text{if }x\leq a,\\ 0\leq f_{\varepsilon,a}(x)\leq 1&\text{if }a\leq x\leq a+\varepsilon,\\ f_{\varepsilon,a}(x)=1&\text{if }a+\varepsilon\leq x.\end{cases}\]

Furthermore, for \(a,b\in\mathbb{R}\) with \(a+2\varepsilon\leq b\) and \(g_{\varepsilon,a,b}\coloneqq f_{\varepsilon,a}-f_{\varepsilon,b-\varepsilon}\) we have

\[\begin{cases}g_{\varepsilon,a,b}(x)=0&\text{if }x\leq a,\\ 0\leq g_{\varepsilon,a,b}(x)=1&\text{if }a\leq x\leq a+\varepsilon,\\ g_{\varepsilon,a,b}(x)=1&\text{if }a+\varepsilon\leq x\leq b-\varepsilon,\\ 0\leq g_{\varepsilon,a,b}(x)=1&\text{if }b-\varepsilon\leq x\leq b,\\ g_{\varepsilon,a,b}(x)=0&\text{if }b\leq x.\end{cases}\]

Let \(\ell\coloneqq\lceil\log n\rceil\) and \(\varepsilon\coloneqq 2^{-k-\ell}\). In the following, let For \(0\leq i\leq n\), let \(a_{i}\coloneqq\frac{i}{n}\). Let \(a_{i}^{-}\coloneqq\left\lfloor 2^{k+\ell+2}\frac{i}{n}\right\rfloor 2^{-k-\ell-2}\) and \(a_{i}^{+}\coloneqq\left\lceil 2^{k+\ell+2}\frac{i}{n}\right\rceil 2^{-k-\ell-2}\). Then \(a_{i}^{-}\leq a_{i}\leq a_{i}^{+}\) and \(a_{i}-a_{i}^{-}\leq\frac{\varepsilon}{4}\), \(a_{i}^{+}-a_{i}\leq\frac{\varepsilon}{4}\). Moreover, \(a_{i}^{-}-a_{i-1}^{+}\geq\frac{1}{n}-\varepsilon 2\geq\frac{\varepsilon}{2}\). Let \(a_{i}^{++}\coloneqq a_{i}^{+}+\frac{\varepsilon}{4}\) and \(a_{i}^{--}\coloneqq a_{i}^{-}-\frac{\varepsilon}{4}\). Then \(a_{i}^{-}\leq a_{i}^{-}\leq a_{i}\leq a_{i}^{+}\leq a_{i}^{++}\) and \(a_{i}-a_{i}^{-}\leq\frac{\varepsilon}{2}\), \(a_{i}^{++}-a_{i}\leq\frac{\varepsilon}{2}\).

For \(1\leq i\leq n\), let \(I_{i}\coloneqq[a_{i-1},a_{i}]\) and \(J_{i}\coloneqq[a_{i-1}^{++},a_{i}^{--}]\). Then \(J_{i}\subseteq I_{i}\). The length of \(I_{i}\) is \(\frac{1}{n}\), and the length of \(J_{i}\) is at least \(\frac{1}{n}-\varepsilon\geq\frac{1}{n}(1-2^{-k})\). Thus the probability that a randomly chosen \(r\in[0,1]\) ends up in one of the intervals \(J_{i}\) is at least \(1-2^{-k}\). Moreover, for every \(i\) we have

\[\frac{1-2^{-k}}{n}\leq\frac{1}{n}-\varepsilon\leq\Pr_{r\sim\mathcal{U}}(r\in J _{i})\leq\frac{1}{n}.\]

Let \(g_{i}\coloneqq g_{\frac{\varepsilon}{4},a_{i-1}^{+},a_{i}^{-}}\). Then \(g(r)=1\) for \(r\in J_{i}\), \(g_{i}(r)=0\) for \(r\not\in I_{i}\), and \(0\leq g_{i}(r)\leq 1\) for \(r\in I_{i}\setminus J_{i}\).

We use the first two layers of our FNN \(\mathfrak{F}\) to compute \(g_{i}\) of the input for all \(i\). That is, on the second level, \(\mathfrak{F}\) has \(n\) nodes \(v_{1},\ldots,v_{n}\), and \(f_{\mathfrak{F},v_{i}}(r)=g_{i}(r)\). As the intervals \(I_{i}\) are disjoint, at most one \(v_{i}\) computes a nonzero value, and with probability at least \(2-2^{-k}\), at least one of the nodes takes value \(1\). On the last level, for each \(i\) there is an edge of weight \(i-1\) from \(v_{i}\) to the output node. Then if \(r\in J_{i}\) the output is \(i-1\), and the assertion follows. 

**Lemma 8.4**.: _Let \(\mathbb{G}\) be a unary query over \(\mathcal{G}\mathcal{S}_{p}^{\mathrm{bool}}\) that is definable by an order-invariant \(\mathsf{GFO}+\mathsf{C}_{\mathrm{nu}}^{\mathrm{gc}}\)-formula. Then there is a polynomial-size bounded-depth family \(\mathcal{R}\) of rational piecewise-linear GNNs with \(\operatorname{ri}\) that computes \(\mathbb{G}\)._

_Furthermore, the GNNs in \(\mathcal{R}\) only use \(\operatorname{relu}\)-activations and \(\mathsf{SUM}\)-aggregation._
Proof.: Let \(\varphi(x)\) be an order-invariant \(\mathsf{GFO}+\mathsf{C}_{\mathrm{nu}}^{\mathcal{E}_{\mathrm{nu}}}\)-formula that defines \(\mathscr{O}\). This means that for all \(p\)-labelled graphs \((G,\mathscr{O})\in\mathscr{E}\mathcal{S}_{p}^{\mathrm{bool}}\), all linear orders \(\leqslant\) on \(V(G)\), and all \(v\in V(G)\), we have

\[(G,\mathscr{O},\leqslant)\vDash\varphi(v)\iff\mathscr{O}(G,\mathscr{O})(v)=1.\]

We want to exploit that if we choose the random features for each vertex, independently for all vertices, then with high probability, they are all distinct and thus they give us a linear order on the vertices.

However, in order to be able to apply Lemma 7.1, we need to carefully limit the randomness. Let \(U_{1},U_{2},U_{3}\) be function variables of type \(\mathtt{v}\to\mathtt{n}\). We let \(\varphi^{\prime}(x)\) be the formula obtained from \(\varphi\) by replacing each atomic subformula \(x\leqslant x^{\prime}\) by the formula

\[U_{1}(x)<U_{1}(x^{\prime})\vee\big{(}U_{1}(x)=U_{1}(x^{\prime}) \wedge U_{2}(x)<U_{2}(x^{\prime})\big{)}\] \[\vee\big{(}U_{1}(x)=U_{1}(x^{\prime})\wedge U_{2}(x)=U_{2}(x^{ \prime})\wedge U_{3}(x)\leq U_{3}(x^{\prime})\big{)}.\]

That is, we order the vertices lexicographically by their \(U_{i}\)-values. If no two vertices have identical \(U_{i}\)-values for \(i=1,2,3\), then this yields a linear order.

Let \((G,\mathscr{O})\in\mathscr{E}\mathcal{S}_{p}^{\mathrm{bool}}\) of order \(n\coloneqq|G|\). For functions \(F_{1},F_{2},F_{3}:V(G)\to\{0,\ldots,n-1\}\) and \(v\in V(G)\), we write \((G,\mathscr{O},F_{1},F_{2},F)\vDash\varphi^{\prime}(v)\) instead of \(((G,\mathscr{O}),\mathscr{O})\vDash\varphi^{\prime}\) for some and hence every assignment \(\mathscr{O}\) with \(\mathscr{O}(U_{i})=F_{i}\) and \(\mathscr{O}(x)=v\). Let us call \(F_{1},F_{2},F_{3}:V(G)\to\{0,\ldots,n-1\}\)_bad_ if there are distinct \(v,w\in V(G)\) such that \(F_{i}(v)=F_{i}(w)\) for \(i=1,2,3\), and call them _good_ otherwise. Observe that for randomly chosen \(F_{1},F_{2},F_{3}\), the probability that they are bad is at most \(\frac{1}{n}\).

By the construction of \(\varphi^{\prime}\) from \(\varphi\), if \(F_{1},F_{2},F_{3}\) are good then for all \(v\in V(G)\) we have

\[(G,\mathscr{O},F_{1},F_{2},F_{3})\vDash\varphi^{\prime}(v)\iff\mathscr{O}(G, \mathscr{O})(v)=1.\]

By Lemma 7.1 in its version for GNNs with global readout and the logic \(\mathsf{FO}^{2}+\mathsf{C}_{\mathrm{nu}}\), there is a polynomial-size bounded-depth family \(\mathscr{N}=(\mathfrak{N}^{(n)})_{n\in\mathbb{N}}\) of rational piecewise-linear GNNs of input dimension \(p+3\) such that for all \((G,\mathscr{O})\in\mathscr{E}\mathcal{S}_{p}\) of order \(n\) and all functions \(F_{1},F_{2},F_{3}:V(G)\to\{0,\ldots,n-1\}\) the following holds. Let \(\mathscr{U}\in\mathcal{S}_{3}(G)\) be the signal defined by \(\mathscr{U}(v)=\big{(}F_{1}(v),F_{2}(v),F_{3}(v)\big{)}\). Then for all \(v\in V(G)\) we have \(\widetilde{\mathscr{N}}(G,\mathscr{O}\mathscr{U})\in\{0,1\}\) and

\[\widetilde{\mathscr{N}}(G,\mathscr{O}\mathscr{U})=1\iff(G,\mathscr{O},F_{1}, F_{2},F_{3})\vDash\varphi^{\prime}(v).\]

Thus if \(F_{1},F_{2},F_{3}\) are good,

\[\widetilde{\mathscr{N}}(G,\mathscr{O}\mathscr{U})=\mathscr{O}(G,\mathscr{O}).\]

Thus all we need to do is use the random features to generate three random functions from \(V(G)\to\{0,\ldots,n-1\}\). At first sight this seems easy, because the random features from \([0,1]\) contain "more randomness" than the functions. However, in fact it is not possible, essentially because we cannot map the interval \([0,1]\) to a discrete subset of the reals of more than one element by a continuous function. But it is good enough to do this approximately, and for this we can use Lemma 8.3. We use this lemma to create a GNN layer \(\mathfrak{L}^{(n)}\) that takes a random random signal \(\mathscr{U}\in\mathcal{S}_{3}(G)\) and computes a signal \(\mathscr{U}\in\mathcal{S}_{3}(G)\) such that with high probability, \(\mathscr{U}(v)_{i}\in\{0,\ldots,n-1\}\), and \(\mathscr{U}(v)_{i}\) is almost
uniformly distributed in \(\{0,\ldots,n-1\}\), for all \(i,v\). This is good enough to guarantee that with high probability the functions \(F_{1},F_{2},F_{3}\) defined by \(\upomega\) are good. Thus if we combined \(\mathfrak{L}^{(n)}\) with \(\mathfrak{N}^{(n)}\) for all \(n\), we obtain a family of GNNs with ri that computes \(\mathcal{G}\). 

_Remark 8.5_.: With a little additional technical work, we could also prove a version of the lemma where the GNNs only use lsig-activations. But it is not clear that this is worth the effort, because actually relu activations are more important (in practice) anyway. Recall that lsig can be simulated with relu, but not the other way around.

Finally, we can prove the following Theorem, which implies Theorem 1.3 stated in the introduction.

**Theorem 8.6**.: _Let \(\mathcal{G}\) be a unary query on \(\mathcal{FG}_{p}^{\mathrm{bool}}\). Then the following are equivalent:_

1. \(\mathcal{G}\) _is definable in order-invariant_ \(\mathsf{GFO}+\mathsf{C}_{\mathrm{nu}}^{\mathrm{gc}}\)_._
2. _There is a polynomial-size bounded-depth family of rational piecewise-linear GNNs with ri, using only_ \(\mathrm{relu}\)_-activations and_ \(\mathsf{SUM}\)_-aggregation, that computes_ \(\mathcal{G}\)_._
3. _There is a rpl-approximable polynomial-weight bounded-depth family of GNNs with ri that computes_ \(\mathcal{G}\)_._
4. \(\mathcal{G}\) _is in_ \(\mathsf{TC}^{0}\)_._

Proof.: The implication \((1)\Longrightarrow(2)\) is Lemma 8.4. The implication \((2)\Longrightarrow(3)\) is trivial. The implication \((3)\Longrightarrow(1)\) is Lemma 8.4. And finally, the equivalence \((1)\Longleftrightarrow(4)\) is Corollary 3.31. 

## 9 Conclusions

We characterise the expressiveness of graph neural networks in terms of logic and Boolean circuit complexity. While this forces us to develop substantial technical machinery with many tedious details, the final results, as stated in the introduction, are surprisingly simple and clean: GNNs correspond to the guarded fragment of first-order logic with counting, and with random initialisation they exactly characterise \(\mathsf{TC}^{0}\). One reason I find this surprising is that GNNs carry out real number computations, whereas the logics and circuits are discrete Boolean models of computation.

We make some advances on the logical side that may be of independent interest. This includes our treatment of rational arithmetic, non-uniformity and built-in relations on unordered structures, and unbounded aggregation (both sum and max). The latter may also shed new light on the relation between first-order logic with counting and the recently introduced weight aggregation logics [6] that deserves further study.

Our results are also interesting from a (theoretical) machine-learning-on-graphs perspective. Most importantly, we are the first to show limitations of GNNs with random initialisation. Previously, it was only known that exponentially large GNNs with ri can
approximate all functions on graphs of order \(n\)[2], but no upper bounds were known for the more realistic model with a polynomial-size restriction. Another interesting consequence of our results (Theorems 7.4 and 8.6) is that arbitrary GNNs can be simulated by GNNs using only \(\mathsf{SUM}\)-aggregation and relu-activations with only a polynomial overhead in size. This was partially known [32] for GNNs distinguishing two graphs, but even for this weaker result the proof of [32] requires exponentially large GNNs because it is based on the universal approximation theorem for feedforward neural networks [32]. It has recently been shown that such a simulation of arbitrary GNNs by GNNs with \(\mathsf{SUM}\)-aggregation is not possible in a uniform setting [26].

We leave it open to extend our results to higher-order GNNs [24] and the bounded-variable fragments of \(\mathsf{FO}+\mathsf{C}\). It might also be possible to extend the results to larger classes of activation functions, for example, those approximable by piecewise-polynomial functions (which are no longer Lipschitz continuous). Another question that remains open is whether the "Uniform Theorem", Theorem 5.1 or Corollary 5.3, has a converse in terms of some form of uniform families of GNNs. It is not even clear, however, what a suitable uniformify notion for families of GNNs would be.

The most interesting extensions of our results, however, would be to GNNs of unbounded depth. Does the correspondence between circuits and GNNs still hold if we drop or relax the depth restriction? And, thinking about uniform models, is there a descriptive complexity theoretic characterisation of recurrent GNNs?
