Lipschitz-bounded 1D convolutional neural networks using the Cayley
  transform and the controllability Gramian
We establish a layer-wise parameterization for 1D convolutional neural
networks (CNNs) with built-in end-to-end robustness guarantees. In doing so, we
use the Lipschitz constant of the input-output mapping characterized by a CNN
as a robustness measure. We base our parameterization on the Cayley transform
that parameterizes orthogonal matrices and the controllability Gramian of the
state space representation of the convolutional layers. The proposed
parameterization by design fulfills linear matrix inequalities that are
sufficient for Lipschitz continuity of the CNN, which further enables
unconstrained training of Lipschitz-bounded 1D CNNs. Finally, we train
Lipschitz-bounded 1D CNNs for the classification of heart arrythmia data and
show their improved robustness.
Patricia Pauli, Ruigang Wang, Ian R. Manchester, Frank Allgöwer
# Lipschitz-bounded 1D convolutional neural networks

###### Abstract

We establish a layer-wise parameterization for 1D convolutional neural networks (CNNs) with built-in end-to-end robustness guarantees. Herein, we use the Lipschitz constant of the input-output mapping characterized by a CNN as a robustness measure. We base our parameterization on the Cayley transform that parameterizes orthogonal matrices and the controllability Gramian for the state space representation of the convolutional layers. The proposed parameterization by design fulfils linear matrix inequalities that are sufficient for Lipschitz continuity of the CNN, which further enables unconstrained training of Lipschitz-bounded 1D CNNs. Finally, we train Lipschitz-bounded 1D CNNs for the classification of heart arrythmia data and show their improved robustness.

## I Introduction

Robustness of neural networks (NNs) has lately been a topic of increasing importance, for which the Lipschitz constant of the NN's input-output mapping has become a common metric [1]. Finding an accurate upper bound on an NN's Lipschitz constant has broadly been tackled, e.g. using relaxations by quadratic constraints [2, 3], average operators [4] and polynomial optimization [5]. In addition, the training of provably Lipschitz-bounded NNs was proposed by including constraints [6, 7] and regularization techniques [8]. While effective, one drawback of these techniques is the computational overhead coming from constraints and projections in the optimization problem [7].

To overcome this, [9, 10, 11] suggest direct parameterizations for equilibrium networks, recurrent equilibrium networks, and feedforward neural networks, respectively, with guaranteed Lipschitz bounds. From a set of unconstrained variables [9, 10, 11] formulate the NNs in such a way that they by design satisfy linear matrix inequalities (LMIs). These LMIs in turn are sufficient conditions for Lipschitz continuity such that, this way, one can parameterize the class of Lipschitz-bounded NNs with a Lipschitz upper bound predefined by the user. The underlying training problem boils down to an unconstrained optimization problem that can be solved using gradient methods. In this work, we take the same approach as in [9, 10, 11] to parameterize Lipschitz-bounded 1D convolutional neural networks (CNNs).

CNNs have been tremendously successful in image and audio processing tasks and they are the state of the art in these applications [12, 13, 14]. In this paper, we focus on 1D CNNs as a stepping stone to the more relevant, yet more involved case of 2D CNNs, which is beyond the scope of this paper. However, an extension based on a 2D systems representation [15] is possible and left as future work. CNNs typically consist of convolutions, nonlinear activation functions, pooling layers, and linear layers that are concatenated in a feedforward structure. While numerous methods exist for enforcing Lipschitz continuity and orthogonality in fully connected layers [16], the design of Lipschitz bounded convolutional layers and CNNs is less studied and often restricted to special convolutions [17]. Recently, this has been approached via parameterization of convolutional layers in the Fourier domain, however this requires a computationally expensive inverse that depends on the input size [11, 17]. We instead formulate convolutions in state space independent of the input dimension [3], providing us with a compact and nonrepetitive description thereof. This leads to a simple and structurally very similar parameterization to the one for fully connected layers. Another feature of our approach is that we impose Lipschitz continuity onto the input-output mapping only rather than on the individual layers, like it is done in many other works [18], using that the product of the Lipschitz bounds of the layers yields a Lipschitz bound for the overall NN. On the contrary, our approach imposes more general dissipativity properties onto the individual layers [3], yielding a _layer-wise_ parameterization with _end-to-end_ robustness guarantees. This leads to reduced conservatism in the compliance of the Lipschitz bound, i. e., higher expressivity for the same Lipschitz bound. In addition our approach accounts for standard pooling layers, which were not addressed in other recent Lipschitz-bounded parameterizations of CNNs [11].

Our main contribution is a direct, scalable, and expressive layer-wise parameterization for Lipschitz-bounded 1D CNNs that makes use of the Cayley transform to parameterize orthogonal matrices. Beside the Cayley transform, a tool that was used for NN parameterization before, we newly propose to utilize the controllability Gramian in the context of parameterizing convolutional layers of Lipschitz-bounded CNNs. In particular, we reformulate parts of the underlying LMI, that enforces dissipativity onto convolutional layers, as a Lyapunov equation whose unique analytical solution is the controllability Gramian. Using our parameterization, we then train Lipschitz-bounded 1D CNNs solving an unconstrained optimization problem.
The remainder of the paper is structured as follows: Section II first introduces 1D CNNs and formally states the training problem. In Section III, we discuss preliminaries, including the state space represenation for 1D convolutions and the Lipschitz constant estimation for 1D CNNs. In Section IV, we present the direct parameterization for Lipschitz-bounded 1D CNNs and in Section V, we train Lipschitz-bounded 1D CNNs on the MIT-BIH arrhythmia database [19], a well-known benchmark dataset for 1D CNNs. Finally, in Section VI, we conclude the paper.

**Notation:** By \(\mathbb{D}^{n}\) (\(\mathbb{D}_{+}^{n}\)) and \(\mathbb{S}^{n}\) (\(\mathbb{S}_{+}^{n}\)), we denote the set of \(n\)-dimensional (positive definite) diagonal and symmetric matrices, respectively, and by \(\mathbb{N}_{+}\) the natural numbers without zero. \(\mathcal{I}\) is a set of indices with elements \(i\in\mathbb{N}_{+}\), and \(|\mathcal{I}|\) gives the number of elements in the index set \(\mathcal{I}\).

## II Problem statement

We consider 1D CNNs that are a concatenation of convolutional layers \(\mathcal{C}_{i}:\mathbb{R}^{c_{i-1}\times N_{i-1}}\rightarrow\mathbb{R}^{c_{ i}\times N_{i}}\) with indices \(i\in\mathcal{C}_{i}\), and fully connected layers \(\mathcal{C}_{i}:\mathbb{R}^{n_{i-1}}\rightarrow\mathbb{R}^{n_{i}}\) with indices \(i\in\mathcal{I}_{F}\)

\[\text{CNN}_{\theta}=\mathcal{L}_{l}\circ\ldots\circ\mathcal{L}_{p+1}\circ F \circ\mathcal{C}_{p}\circ\ldots\circ\mathcal{C}_{1}, \tag{1}\]

adding up to a total number of \(l=|\mathcal{A}_{C}|+|\mathcal{I}_{F}|\) layers. Herein, \(N_{i}\) denotes the signal length, \(c_{i}\) the channel size, and \(n_{i}\) the layer dimension of the respective \(i\)-th layer. At the intersection of the fully connected part and the fully convolutional part of the CNN, there is a flattening operation \(F:\mathbb{R}^{c_{p}\times N_{p}}\rightarrow\mathbb{R}^{n_{p}}\) of the output of the \(p\)-th (last) convolutional layer with \(n_{p}=c_{p}N_{p}\).

A _convolutional_ layer consists of two to three stages, a convolution operation, a nonlinear activation, and possibly a pooling operation. The first two stages are

\[\widetilde{\mathcal{C}_{i}}:w_{k}^{i}=\phi_{i}\left(b_{i}+\sum_{j=0}^{\ell_{i }-1}K_{j}^{i}w_{k-j}^{i-1}\right),\quad k=0,\ldots,N_{i}-1\ \forall i\in\mathcal{I}_{C}, \tag{2}\]

with convolution kernel \(K_{j}^{i}\in\mathbb{R}^{c_{i}\times c_{i-1}}\), \(j=0,\ldots,\ell_{i}-1\), kernel size \(\ell_{i}\), and bias \(b_{i}\in\mathbb{R}^{c_{i}}\). First, a convolution on the signal \(w^{i-1}\in\mathbb{R}^{c_{i-1}\times N_{i-1}}\) is applied and subsequently, the nonlinear activation function \(\phi_{i}:\mathbb{R}^{c_{i}}\rightarrow\mathbb{R}^{c_{i}}\) is evaluated elementwise to obtain the output \(w^{i}\in\mathbb{R}^{c_{i}\times N_{i-1}}\). Oftentimes, a convolutional layer additionally contains pooling layers \(\mathcal{P}_{i}:\mathbb{R}^{c_{i}\times N_{i-1}}\rightarrow\mathbb{R}^{c_{i} \times N_{i}}\) to downsample the signal \(w^{i}\). We consider maximum pooling

\[\mathcal{P}_{i}^{\text{max}}:\tilde{w}_{k}^{i}=\max_{j=1,\ldots,\ell_{i}}w_{ \ell_{i}(k-1)+j}^{i},\ k=0,\ldots,N_{i}-1,\forall i\in\mathcal{I}_{P}^{\text {max}},\]

and average pooling

\[\mathcal{P}_{i}^{\text{av}}:\tilde{w}_{k}^{i}=\frac{1}{\ell_{i}}\sum_{j=1}^{ \ell_{i}}w_{\ell_{i}(k-1)+j}^{i},\ k=0,\ldots,N_{i}-1,\forall i\in\mathcal{I} _{P}^{\text{av}},\]

where \(\mathcal{I}_{P}^{\text{av}}\cup\mathcal{I}_{P}^{\text{max}}\subseteq\mathcal{I }_{C}\). As a result, the convolutional layer becomes \(\mathcal{C}_{i}=\mathcal{P}_{i}\circ\mathcal{C}_{i}\) in case a pooling layer is added or \(\mathcal{C}_{i}=\widetilde{\mathcal{C}_{i}}\) otherwise. Finally, a CNN typically holds _fully connected_ layers, which we define as mappings

\[\begin{split}&\mathcal{L}_{l}:\ w^{i}=\phi_{i}(W_{l}w^{i-1}+b_{i}) \quad\forall i\in\mathcal{I}_{F}\backslash\{l\},\\ &\mathcal{L}_{l}:\ w^{l}=W_{l}w^{l-1}+b_{l}\end{split} \tag{3}\]

with weights \(W_{l}\in\mathbb{R}^{n_{l}\times n_{l-1}}\), biases \(b_{i}\in\mathbb{R}^{n_{l}}\) and activation functions \(\phi_{i}:\mathbb{R}^{n_{l}}\rightarrow\mathbb{R}^{n_{l}}\) that are applied elementwise.

The 1D CNN \(f_{\theta}(w^{0})=w^{l}\) is hence characterized by \(\theta=\{(K^{i},b_{i})_{i=1}^{p},(W_{i},b_{i})_{i=p+1}^{i}\}\) and the chosen activation and pooling operations. In this work, we present a direct parameterization for Lipschitz-bounded 1D CNNs (1).

**Problem 1:** Find a parameterization \(\kappa\mapsto\theta\) of \(f_{\theta}\) for a predefined Lipschitz bound \(\rho>0\) such that all 1D CNNs parameterized by \(\kappa\) are \(\rho\)-Lipschitz continuous with respect to the \(\ell_{2}\) norm, i. e., they satisfy

\[\|f_{\theta}(x)-f_{\theta}(y)\|_{2}\leq\rho\|x-y\|_{2}\quad\forall x,y\in \mathbb{R}^{n}. \tag{4}\]

In the case of multiple channels \(c\), \(n=cN\) denotes the stacked up version of the input. Note that \(\|\cdot\|_{2}\) in (4) can either be interpreted as the Euclidean norm of a vector-valued input \(x\) or as the \(\ell_{2}\) norm of a signal \(x\).

To train a Lipschitz-bounded CNN, we minimize a learning objective \(\mathcal{L}(\theta)\), e. g., the mean squared error, the cross-entropy loss or, to encourage robustness through the learning objective a tailored loss, e.g. the hinge loss [20], while at the same time enforcing Lipschitz-boundedness onto the CNN. Rather than solving a training problem subject to a Lipschitz constraint, i. e.,

\[\min_{\theta}\ \mathcal{L}(\theta)\quad\text{s.\ t.}\quad f_{\theta}\text{ is Lipschitz-bounded},\]

the suggested parameterization \(\kappa\mapsto\theta\) allows to solve an unconstrained training problem over \(\kappa\)

\[\min_{\kappa}\ \mathcal{L}(\theta(\kappa)).\]

## III Preliminaries

Before we state the parameterization of Lipschitz-bounded 1D CNNs in Section IV, we introduce a compact formulation of convolutions in state space and state LMI conditions that certify Lipschitz boundedness and that can be used to estimate the Lipschitz constant for 1D CNNs [3]. In addition, we introduce the Cayley transform used to parameterize orthogonal matrices.

### _State space representation for convolutions_

To formulate LMI conditions for convolutional layers, we can either reformulate the convolutional operation as a fully connected layer characterized by a sparse and redundant Toeplitz matrix [7] that scales with the input dimension or, as suggested in [3], we can compactly state the convolution, i. e., a finite impulse response (FIR) filter, in state space, completely independent of the input signal length. A possible discrete-time state space representation of the \(i\)-th convolutional layer (2) with state \(x_{k}^{i}\in\mathbb{R}^{n_{l_{i}}}\) and state dimension \(n_{x_{i}}=(\ell_{i}-1)c_{i-1}\) is

\[\begin{split} x_{k+1}^{i}&=A_{i}x_{k}^{i}+B_{i}w_{k}^{i -1},\\  y_{k}^{i}&=C_{i}x_{k}^{i}+D_{i}w_{k}^{i-1}+b_{i}, \\ w_{k}^{i}&=\phi(y_{k}^{i}),\end{split} \tag{5}\]
where

\[A_{i} =\begin{bmatrix}0&I&&0\\ 0&0&\ddots\\ \vdots&&\ddots&I\\ 0&\ldots&&0\end{bmatrix}, B_{i} =\begin{bmatrix}0\\ \vdots\\ 0\\ I\end{bmatrix}, \tag{6a}\] \[C_{i} =\begin{bmatrix}K_{\ell_{i}-1}^{i}&\ldots&K_{1}^{i}\end{bmatrix}, D_{i} =K_{0}^{i}. \tag{6b}\]

Note that 2D convolutions also admit a state space realization, namely as a 2D system [15], based on which our parameterization can potentially be extended to end-to-end Lipschitz-bounded 2D CNNs.

**Remark 2**: **The evaluation of a convolution via a fast Fourier transform necessitates the entire signal, whereas our causal representation of convolutional layers allows their use in real time.**

### _Lipschitz constant estimation_

The Lipschitz constant is a sensitivity measure to changes in the input, which is commonly used to verify robustness for NNs [1]. Since, however, the calculation of the true Lipschitz constant is an NP-hard problem, an accurate upper bound is sought instead. For this purpose, we over-approximate the nonlinear activation functions by their slope-restriction cone [2, 6]. Commonly used nonlinear activation functions \(\varphi:\mathbb{R}\to\mathbb{R}\), such as ReLU and tanh, are slope-restricted in \([0,1]\), i. e.,

\[0\leq\frac{\varphi(x)-\varphi(y)}{x-y}\leq 1\quad\forall x,y\in\mathbb{R}^{n}.\]

Based on this property, we formulate an incremental quadratic constraint

\[\begin{bmatrix}\phi(x)-\phi(y)\\ x-y\end{bmatrix}^{\top}\begin{bmatrix}-2\Lambda&\Lambda\\ \Lambda&0\end{bmatrix}\begin{bmatrix}\phi(x)-\phi(y)\\ x-y\end{bmatrix}\geq 0\ \forall x,y\in\mathbb{R}^{n}. \tag{7}\]

with multipliers \(\Lambda\in\mathbb{D}_{+}^{n}\), yielding a suitable over-approximation of the NN. The following theorem states a set of \(l\) LMI conditions that serve as a sufficient condition for Lipschitz continuity for 1D CNNs based on the relaxation (7) [3].

**Theorem 3** ([3]): _Let \(\mathrm{CNN}_{\theta}\) and \(\rho>0\) be given and let all activation functions be slope-restricted in \([0,1]\). If there exist_

1. \(Q_{i}\in\mathbb{S}^{c_{i}}\) _(_\(Q_{i}\in\mathbb{D}^{c_{i}}\)_ if a convolutional layer contains a maximum pooling layer),_ \(P_{i}\in\mathbb{S}_{+}^{n_{\mu_{i}}}\)_, and_ \(\Lambda_{i}\in\mathbb{D}_{+}^{c_{i}}\) _such that_ \(\forall i\in\mathcal{I}_{C}\)__ \[\left[\begin{array}{cc|c}P_{i}-A_{i}^{\top}P_{i}A_{i}&-A_{i}^{\top}P_{i}B_{ i}&-C_{i}^{\top}\Lambda_{i}\\ -B_{i}^{\top}P_{i}A_{i}&Q_{i-1}-B_{i}^{\top}P_{i}B_{i}&-D_{i}^{\top}\Lambda_{i} \\ \hline-\Lambda_{i}C_{i}&-\Lambda_{i}D_{i}&2\Lambda_{i}-Q_{i}\end{array}\right]\succeq 0,\] (8) _where_ \(Q_{0}=\tilde{\rho}^{2}I\)_,_
2. \(Q_{i}\in\mathbb{S}^{n_{i}}\) _and_ \(\Lambda_{i}\in\mathbb{D}_{+}^{n_{i}}\) _such that_ \(\forall i=\mathcal{I}_{F}\backslash\{I\}\)__ \[\begin{bmatrix}Q_{i-1}&-W_{i}^{\top}\Lambda_{i}\\ -\Lambda_{i}W_{i}&2\Lambda_{i}-Q_{i}\end{bmatrix}\succeq 0\text{ and }\begin{bmatrix}Q_{l-1}&-W_{l}^{\top}\\ -W_{l}&I\end{bmatrix}\succeq 0,\] (9) _where_ \(Q_{p}:=I_{N_{p}}\otimes Q_{p}\)_,_

_then the \(\mathrm{CNN}_{\theta}\) is \(\rho\)-Lipschitz continuous with \(\rho=\tilde{\rho}\prod_{S\in\mathcal{I}_{F}^{\mu}}\mu_{s}\), where \(\mu_{s}\) are the Lipschitz constants of the average pooling layers._

_The underlying idea in Theorem 3 is to enforce dissipativity onto all individual layers that are connected in a feedforward fashion. Thus the matrix \(Q_{i-1}\) links the \(i\)-th layer to the previous layer and by this interconnection we can finally analyse Lipschitz continuity of the input-output mapping \(w^{l}=\mathrm{CNN}_{\theta}(w^{0})\)[3]._

_Based on Theorem 3, we can determine an upper bound on the Lipschitz constant for a given CNN solving a semidefinite program_

\[\min_{\rho^{2},\Lambda,P,Q}\ \rho^{2}\quad\text{s.\,t.}\quad(\ref{eq:1}),(\ref{eq:2}), \tag{10}\]

_where \(\Lambda=\{\Lambda_{i}\}_{i\in\mathcal{I}_{C}\cup\mathcal{I}_{F}\backslash\{I\}}\), \(Q=\{Q_{i}\}_{i\in\mathcal{I}_{C}\cup\mathcal{I}_{F}\backslash\{I\}}\), \(P=\{P_{i}\}_{i\in\mathcal{I}_{C}}\) serve as decision variables together with \(\rho^{2}\)._

### _Cayley transform_

Typically, the Cayley transform maps skew-symmetric matrices to orthogonal matrices and its extended version parameterizes the Stiefel manifold from non-square matrices, which can be useful in designing NNs [11, 17, 21].

**Lemma 4** (Cayley transform [22]): _For all \(Y\in\mathbb{R}^{n\times n}\) and \(Z\in\mathbb{R}^{m\times n}\) the Cayley transform_

\[\mathrm{Cayley}\left(\begin{bmatrix}Y\\ Z\end{bmatrix}\right)=\begin{bmatrix}U\\ V\end{bmatrix}=\begin{bmatrix}(I+M)^{-1}(I-M)\\ 2Z(I+M)^{-1}\end{bmatrix},\]

_where \(M=Y-Y^{\top}+Z^{\top}Z\), yields matrices \(U\in\mathbb{R}^{n\times n}\) and \(V\in\mathbb{R}^{m\times n}\) that satisfy \(U^{\top}U+V^{\top}V=I\)._

_Note that \(I+M\) is nonsingular since \(1\leq\lambda_{\min}(I+Z^{\top}Z)\leq Re(\lambda_{\min}(I+M))\)[23]._

## IV Direct parameterization

While (10) analyses Lipschitz continuity for given 1D CNNs, it might also be desirable to train robust CNNs, i. e., \(\rho\)-Lipschitz bounded CNNs where the robustness level \(\rho\) is chosen by the user. In this section, we introduce a direct layer-wise parameterization for 1D CNNs (1) that renders the input-output mapping Lipschitz continuous. We first discuss a parameterization for fully connected layers that satisfy (9) by design, using a similar construction to [11]. Our key contribution then is the parameterization of convolutional layers, which is carried out in two steps. In a first step, we establish a parameterization of \(P_{i}\) that renders the left upper block in (8) positive definite using the controllability Gramian and afterwards, we introduce the parameterization for convolutional layers that by design satisfy (8).

### _Fully connected layers_

In the following, we present a mapping \(\kappa_{i}\mapsto(W_{i},b_{i})\) from unconstrained variables \(\kappa_{i}\) that renders (9) feasible by design.

**Theorem 5**: _Fully connected layers (3) parameterized by_

\[W_{i} =\sqrt{2}\Gamma_{i}^{-1}V_{i}^{\top}L_{i-1}, b_{i}\in\mathbb{R}^{c_{i}}, i\in\mathcal{I}_{F}\backslash\{I\}, \tag{11a}\] \[W_{i} =V_{i}^{\top}L_{i-1}, b_{i}\in\mathbb{R}^{c_{i}}, \tag{11b}\]
satisfy (9). Herein,

\[\Gamma_{i}=\text{diag}(\gamma_{i}),\ L_{i}=\sqrt{2}U_{i}\Gamma_{i},\ \ \begin{bmatrix}U_{i}\\ V_{i}\end{bmatrix}=\text{Cayley}\left(\begin{bmatrix}Y_{i}\\ Z_{i}\end{bmatrix}\right)\]

with free variables \(Y_{i}\in\mathbb{R}^{n_{i}\times n_{i}}\), \(Z_{i}\in\mathbb{R}^{n_{i-1}\times n_{i}}\), \(b_{i}\in\mathbb{R}^{n_{i}}\), \(i\in\mathcal{I}_{\mathcal{F}},\ \gamma_{i}\in\mathbb{R}^{n_{i}},\ i\in \mathcal{I}_{\mathcal{F}}\backslash\{I\}\). This yields the mappings \((Y_{i},Z_{i},\gamma_{i},b_{i})\mapsto(W_{i},b_{i})\), \(i\in\mathcal{I}_{\mathcal{F}}\backslash\{I\}\), and \((Y_{i},Z_{i},b_{i})\mapsto(W_{i},b_{i})\), respectively.

According to Lemma 4, \(U_{i}\) and \(V_{i}\) satisfy \(U_{i}^{\top}U_{i}+V_{i}^{\top}V_{i}=I\). Now inserting the parameterization (11a), we obtain

\[\frac{1}{2}(\Gamma_{i}^{\top}L_{i}^{\top}L_{i}\Gamma_{i}^{-1}+\Gamma_{i}W_{i} ^{\top}L_{i-1}^{-1}L_{i-1}^{-\top}W_{i}\Gamma_{i})=I.\]

With \(Q_{i}=L_{i}^{\top}L_{i}\) and \(\Lambda_{i}=\Gamma_{i}^{\top}\Gamma_{i}\), we further obtain

\[Q_{i}+\Lambda_{i}W_{i}Q_{i-1}^{-1}W_{i}^{\top}\Lambda_{i}=2\Lambda_{i},\]

which implies \(2\Lambda_{i}-Q_{i}-\Lambda_{i}W_{i}Q_{i-1}^{-1}W_{i}^{\top}\Lambda_{i}\succeq 0\). Next, we apply the Schur complement, which yields the left inequality in (9). The last fully connected layer is a special case that does not contain an activation function. Inserting the parameterization (11b) gives

\[U_{i}^{\top}U_{l}+V_{l}^{\top}V_{l}=U_{l}^{\top}U_{l}+W_{l}Q_{l-1}^{-1}W_{l}^ {\top}=I,\]

which implies \(I-W_{l}Q_{l-1}^{-1}W_{l}^{\top}=U_{l}^{\top}U_{l}\succeq 0\), which by the application of the Schur complement satisfies the right inequality in (9).

Note that the connection between the auxiliary matrices \(L_{i}\) in (11) and \(Q_{i}\) in (9) is \(L_{i}^{\top}L_{i}=Q_{i}\) and the relation between the multiplier matrices \(\Lambda_{i}\) in (7) / (9) and \(\Gamma_{i}\) in (11) is \(\Gamma_{i}^{\top}\Gamma_{i}=\Lambda_{i}\).

**Remark 6**: _Throughout the paper, we assume that \(\Gamma_{i}\) and \(L_{i}\) are nonsingular. In our experiments, this was always the case. However, there also are tricks to enforce this property, e.g., by choosing \(\Gamma_{i}=\text{diag}(e^{\gamma_{i}})\)[11]._

**Remark 7**: _Our parameterization (11) is equivalent to the one established in [11], where they show that it is necessary and sufficient, i. e., the fully connected layers (3) satisfy (9) if and only if the weights can be parameterized by (11)._

### _Parameterization by the controllability Gramian_

In this section, we make use of the controllability Gramian of (5) to parameterize convolutional layers, which to the best knowledge of the authors, has thus far not appeared in the context of parameterizing NNs. For that purpose, we introduce

\[F_{i}:=\left[\begin{array}{cc}P_{i}-A_{i}^{\top}P_{i}A_{i}&-A_{i}^{\top}P_{ i}B_{i}\\ -B_{i}^{\top}P_{i}A_{i}&Q_{i-1}-B_{i}^{\top}P_{i}B_{i}\end{array}\right]\succ 0, \tag{12}\]

which is the left upper block in (2) and further, we introduce \(\widehat{C}_{i}:=\begin{bmatrix}C_{i}&D_{i}\end{bmatrix}\), which simplifies the notation of (8) to

\[\begin{bmatrix}F_{i}&-\widehat{C}_{i}^{\top}\Lambda_{i}\\ -\Lambda_{i}\widehat{C}_{i}&2\Lambda_{i}-Q_{i}\end{bmatrix}\succeq 0. \tag{13}\]

We note that the LMI (13) and the left LMI in (9) share a similar structure. The right lower block is the same in both LMIs. In addition to the bias terms \(b_{i}\), the parameters to be trained in the CNN layers are collected in \(\widehat{C}_{i}\), cmp. (6), whereas the parameters \(W_{i}\) characterize the fully connected layers. In the off-diagonal blocks of the respective LMIs (13) and (9) \(\widehat{C}_{i}\) and \(W_{i}\) appear respectively multiplied by \(\Lambda_{i}\). The only difference is in the left upper blocks of the LMIs. While in LMI (9) for fully connected layers, we here have \(Q_{i-1}=L_{i-1}^{\top}L_{i-1}\succ 0\) for nonsingular \(L_{i-1}\), LMI (13) for convolutional layers here contains \(F_{i}\), that depends on \(Q_{i-1}\). To render \(F_{i}\) positive definite, we parameterize \(P_{i}\) as follows, using the controllability Gramian.

**Lemma 8**: _For some \(\varepsilon>0\) and all \(H_{i}\in\mathbb{R}^{n_{i}\times n_{i}}\), the matrix \(P_{i}=X_{i}^{-1}\) with_

\[X_{i}=\sum_{k=0}^{n_{i}-c_{i-1}}A_{i}^{k}(B_{i}Q_{i-1}^{-1}B_{i}^{\top}+H_{i}^ {\top}H_{i}+\varepsilon I)(A_{i}^{\top})^{k}, \tag{14}\]

_renders (12) feasible._

The matrix \(A_{i}\) is a nilpotent matrix, i. e., \(A^{n_{i}-c_{i-1}+k}=0\ \forall k\geq 1\), such that

\[X_{i}=\sum_{k=0}^{\infty}A_{i}^{k}(B_{i}Q_{i-1}^{-1}B_{i}^{\top}+H_{i}^{\top}H_ {i}+\varepsilon I)(A_{i}^{\top})^{k}\]

corresponds to the controllability Gramian of the linear time-invariant system characterized by \((A_{i},B_{i})\) as defined in (6a), i. e., the unique solution \(X_{i}\succ 0\) to the Lyapunov equation

\[X_{i}-A_{i}X_{i}A_{i}^{\top}-B_{i}Q_{i-1}^{-1}B_{i}^{\top}=H_{i}^{\top}H_{i}+ \varepsilon I\succ 0. \tag{15}\]

Note that \(X_{i}\) is positive definite by design, given that \(Q_{i-1}=L_{i-1}^{\top}L_{i-1}\succ 0\) such that \(B_{i}Q_{i-1}^{-1}B_{i}^{\top}+H_{i}^{\top}H_{i}+\varepsilon I\) is positive definite. Next, we apply the Schur complement to (15) to obtain

\[\begin{bmatrix}X_{i}^{-1}&0&A_{i}^{\top}\\ 0&Q_{i-1}&B_{i}^{\top}\\ A_{i}&B_{i}&X_{i}\end{bmatrix}\succ 0,\]

Now inserting \(P_{i}=X_{i}^{-1}\) and again applying the Schur complement yields (12).

### _Convolutional layers_

In this subsection, we present a direct parameterization for convolutional layers such that they satisfy (13) by design. Our parameterization of convolution kernels \(K^{i}\in\mathbb{R}^{c_{i}\times c_{i-1}\times c_{i}}\), or equivalently \(\widehat{C}_{i}\in\mathbb{R}^{c_{i}\times c_{i}c_{i-1}}\), is independent of the input dimension \(N_{i}\) whereas other approaches design a parameterization for Lipschitz-bounded convolutions and CNNs in the Fourier domain which involves the costly inversion of \(N_{i}\) matrices [17, 11].

**Theorem 9**: _Covolutional layers (2) that contain an average pooling layer or no pooling layer parameterized by_

\[\widehat{C}_{i}=\sqrt{2}\Gamma_{i}^{-1}V_{i}^{\top}L_{i}^{F},\ b_{i}\in\mathbb{R }^{c_{i}},\ \forall i\in\mathcal{I}_{\mathcal{C}}\backslash\mathcal{I}_{\mathcal{C}}^{ \text{max}} \tag{16}\]

_satisfy (8). Herein,_

\[\Gamma_{i}=\text{diag}(\gamma_{i}),\ \begin{bmatrix}U_{i}\\ V_{i}\end{bmatrix}=\text{Cayley}\left(\begin{bmatrix}Y_{i}\\ Z_{i}\end{bmatrix}\right),\ L_{i}^{F}=\text{chol}(F_{i}),\]

\(\text{chol}(\cdot)\) _denoting the Cholesky decomposition, \(Q_{i}=L_{i}^{\top}L_{i},\ L_{0}=\rho I,\ L_{i}=\sqrt{2}U_{i}\Gamma_{i}\), where \(F_{i}\) is given by (12) with \(P_{i}\) parameterized from \(Q_{i-1}\) and \(H_{i}\) using (14). The free variables beside \(b_{i}\) are \(Y_{i}\in\mathbb{R}^{c_{i}\times c_{i}}\), \(Z_{i}\in\mathbb{R}^{c_{i-1}\times c_{i}}\), \(H_{i}\in\mathbb{R}^{c_{i}\times c_{i}}\), where \(c_{i}\) is the number of
\(\mathbb{R}^{n_{ij}\times n_{ij}}\), \(\gamma_{i}\in\mathbb{R}^{c_{i}}\), \(i\in\mathcal{I}_{C}\mathcal{I}_{C}^{\max}\), which yields the mapping \((Y_{i},Z_{i},H_{i},\gamma_{i},b_{i})\mapsto(K^{i},b_{i})\)._

Proof:: The matrices \(U_{i}\) and \(V_{i}\) satisfy \(U_{i}^{\top}U_{i}+V_{i}^{\top}V_{i}=I\). Now inserting the parameterization (16), we obtain

\[\frac{1}{2}(\Gamma_{i}^{-1}L_{i}^{\top}L_{i}\Gamma_{i}^{-1}+\Gamma_{i}W_{i}^{ \top}L_{i-1}^{\top}L_{i-1}^{\top}{}^{-\top}W_{i}\Gamma_{i})=I.\]

Lemma 8 ensures positive definiteness of \(F_{i}\), i. e., its Cholesky decomposition exists, and we insert \(Q_{i}=L_{i}^{\top}L_{i}\), \(F_{i}=L_{i}^{F\top}L_{i}^{F}\), \(\Lambda_{i}=\Gamma_{i}^{\top}\Gamma_{i}\), to further obtain

\[Q_{i}+\Lambda\widehat{C}_{i}F_{i}^{-1}\widehat{C}_{i}^{\top}\Lambda_{i}=2 \Lambda_{i},\]

which implies \(2\Lambda_{i}-Q_{i}-\Lambda_{i}\widehat{C}_{i}F_{i}^{-1}\widehat{C}_{i}^{\top} \Lambda_{i}\succeq 0\). Next, we apply the Schur complement and obtain (13) which corresponds to (8). 

To account for average pooling layers present in the CNN, we rescale the Lipschitz bound with the product of the Lipschitz bounds of the average pooling layers, i. e., \(\tilde{\rho}=\rho/\Pi_{s\in\mathcal{I}_{\mathcal{I}_{\mathcal{I}_{\mathcal{I }_{\mathcal{I}_{\mathcal{I}_{\mathcal{I}_{\mathcal{I}_{\mathcal{I}_{\mathcal{I }_{\mathcal{I}_{\mathcal{I}_{\mathcal{I}_{\mathcal{I}_{I}_{\mathcal{I}_{I}_{I }_{I}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}\)\)\)\)\)\)\)   \) \)
robustness, yet lower test accuracies and vice versa. We note that the vanilla CNN has significantly larger upper Lipschitz bounds than LipCNN and further, LipCNN maintains high test accuracies as the Lipschitz bound decreases in Fig. 3. With even larger weighting parameters \(\gamma\) in L2 regularized training the training failed, whereas LipCNNs allows for training with very low Lipschitz bounds.

## VI Conclusion

In this paper, we introduced a parameterization for Lipschitz-bounded 1D CNNs using Cayley transforms and controllability Gramians. Using our parameterization we can train Lipschitz-bounded 1D CNNs in an unconstrained training problem which we illustrated in the classification of ECG data from the MIT-BIH database. Future research includes the extension of our parameterization to 2D CNNs using a 2D systems approach as suggested in [15].
