# Kernel-based Impulse Response Identification with Side-Information on
Steady-State Gain

Mohammad Khosravi  Roy S. Smith

###### Abstract

In this paper, we consider the problem of system identification when side-
information is available on the steady-state (or DC) gain of the system. We
formulate a general nonparametric identification method as an infinite-
dimensional constrained convex program over the reproducing kernel Hilbert
space (RKHS) of stable impulse responses. The objective function of this
optimization problem is the empirical loss regularized with the norm of RKHS,
and the constraint is considered for enforcing the integration of the steady-
state gain side-information. The proposed formulation addresses both the
discrete-time and continuous-time cases. We show that this program has a
unique solution obtained by solving an equivalent finite-dimensional convex
optimization. This solution has a closed-form when the empirical loss and
regularization functions are quadratic and exact side-information is
considered. We perform extensive numerical comparisons to verify the
efficiency of the proposed identification methodology.

##  I Introduction

System identification is a well-established research area on the theory and
techniques of creating appropriate mathematical abstractions for the dynamical
systems using their measurement data [1]. According to the importance and
numerous applications of system identification in different fields of science
and technology, it has received a significant deal of attention [2]. In
various situations, identifying a dynamical system can be beyond a mere model
fitting to the data, and additionally, we may need to include particular known
features and attributes of the system into the model. More precisely, together
with the measurement data, we might be provided with certain so-called side-
information, which is indeed a specific qualitative or quantitative knowledge
to be incorporated in the identified model of the system. This side-
information can originate from various sources, e.g., a general understanding
of the intrinsic physical nature of the system, or from the observed behaviors
in experimental or historical data [3]. Integrating side-information can
improve the identification performance by rejecting spurious model candidates,
which are common when the measurement data is scarce, highly noise-
contaminated, or generated by insufficient excitation [4].

Various forms of side-information such as stability, dissipativity, region of
attraction, and many others are considered in identifying nonlinear dynamical
systems [5, 6, 7, 8]. On the other hand, the key role of linear systems in
practice has led to increased research on how to integrate different sorts of
side-information in their identification [9]. For example, the incorporation
of structural side-information, that is the available knowledge on the
configuration and types of the subsystems, has been studied by imposing
specific model structural constraints such as being circulant [10]. Due to the
importance of frequency domain analysis in controller design, a variety of
relevant properties are included in the identification procedure, e.g., the
location of poles of the system [11], phase constraints [12], the peak in
frequency response [13], moments and derivatives of the transfer function
[14], positive-realness [15], and the passivity of the system [16].
Identification with side-information such as positivity or being compartmental
are considered in [17, 18]. The side-information on the low internal
complexity of the system is included by means of sparsity promoting
regularizations such as the rank and the nuclear norm of associated Hankel
matrices [19, 20], or by employing atomic norm regularization applied on
specific atomic linear representations of the system [21]. The stability side-
information is integrated into the subspace identification method by ensuring
that the poles of the identified models are inside the unit disc [22, 23]. The
kernel-based system identification approach [24], besides addressing model
order selection, robustness, and bias-variance trade-off issues, opened new
avenues for the integration of various types of side-information [25, 26, 27],
including stability, dissipativity, resonant frequencies, smoothness of the
impulse response, oscillatory behaviors, relative degree, exponential decay of
the impulse response, structural properties, and the presence of fast and slow
poles [28, 29, 30, 31, 32, 33, 34, 35, 36]. Moreover, the incorporation of
positivity and internal low-complexity are revisited in this framework [37,
38, 39, 40, 41, 42].

The steady-state gain information has special importance from the control
system perspective, e.g., in closed-loop design and model predictive control
[43, 44, 45]. Hence, it is of particular interest to integrate this
information into the identified model. To this end, various heuristics are
introduced based on the subspace identification approach [4, 46, 47, 48, 49].
Indeed, to identify a finite impulse response (FIR) model for the system, the
subspace method can be employed in the multi-step ahead prediction form [4,
46]. Following this and using a Bayesian approach, the steady-state gain side-
information can be encoded in the covariance of the prior distribution [4]. On
the other hand, a frequentist framework is employed in [46, 47, 48, 49], where
the steady-state gain side-information is incorporated by imposing linear
constraints. Moreover, to leverage the previously mentioned advantages of the
kernel-based approach, Bayesian FIR estimation methods are proposed in [50,
51], where kernel-based priors are employed and the steady-state gain side-
information is integrated into the resulting estimation problem. The
identification scheme in [50] first estimates the step response of the system,
and then, the impulse response is obtained via a naïve discrete derivative
calculation, which is prone to numerical imprecision and instability. One the
other hand, while the method introduced in [51] improves the estimation
performance approach in [50], the proposed formulation is incapable of
including deterministic information on the steady-state gain of the system.
The identification approaches discussed are only applicable when a large set
of high-quality data is available. Furthermore, they are limited to relatively
short FIR estimation and fast decaying dynamics. Therefore, these estimation
methodologies are not suitable for infinite impulse responses (IIR) and
continuous-time systems, particularly when the dynamics have a very slowly
decaying impulse response and considerably long memory.

In this paper, we develop a nonparametric identification approach where the
side-information on the steady-state gain of the system is integrated into the
proposed scheme. To leverage the powerful framework of kernel-based
identification, we employ RKHS of stable impulse responses as the hypothesis
space [26, 52], enabling the formulation of the problem for the continuous-
time and discrete-time cases together. The identification problem is expressed
as a constrained optimization where a generic regularized empirical loss is
minimized subject to a suitably designed constraint encoding the available
side-information on the steady-state gain of the system. According to the
frequentist approach employed, the resulting formulation is flexible, e.g.,
one can address the issue of outliers by defining the empirical loss based on
the Huber function and its variants. We show that the estimation problem,
initially formulated as an infinite-dimensional optimization, is equivalent to
a finite-dimensional convex program with a unique solution. This solution has
a closed-form when exact side-information is considered, and the empirical
loss and regularization functions used are quadratic. We perform extensive
numerical simulations confirming the efficacy of the proposed identification
method.

##  II Notations and Preliminaries

The set of natural numbers, the set of non-negative integers, the set of real
numbers, the set of non-negative real numbers, the $n$-dimensional Euclidean
space and the space of $n$ by $m$ real matrices are denoted by ${\mathbb{N}}$,
${\mathbb{Z}}_{+}$, ${\mathbb{R}}$, ${\mathbb{R}}_{+}$, ${\mathbb{R}}^{n}$ and
${\mathbb{R}}^{n\times m}$, respectively. The $i^{\text{\tiny{th}}}$ entry of
vector ${\mathrm{a}}$ is denoted by $[{\mathrm{a}}]_{(i)}$, and the entry of
matrix ${\mathrm{A}}$ at the $i^{\text{\tiny{th}}}$ row and the
$j^{\text{\tiny{th}}}$ column is denoted by $[{\mathrm{A}}]_{(i,j)}$. To
handle discrete and continuous time in the same formulation, ${\mathbb{T}}$
denotes either ${\mathbb{Z}}_{+}$ or ${\mathbb{R}}_{+}$, and
${\mathbb{T}}_{\pm}$ is the set of scalars $t$ where either $t\in{\mathbb{T}}$
or $-t\in{\mathbb{T}}$. Given measure space ${\mathscr{X}}$, the space of
measurable functions $g:{\mathscr{X}}\to{\mathbb{R}}$ is denoted by
${\mathbb{R}}^{{\mathscr{X}}}$. The element
${\mathrm{u}}\in{\mathbb{R}}^{{\mathscr{X}}}$ is shown entry-wise as
${\mathrm{u}}=(u_{x})_{x\in{\mathscr{X}}}$, or equivalently as
${\mathrm{u}}=\big{(}u(x)\big{)}_{x\in{\mathscr{X}}}$. Depending on the
context of discussion, ${\mathscr{L}}^{\infty}$ refers either to
$\ell^{\infty}({\mathbb{Z}})$ or $L^{\infty}({\mathbb{R}})$. Similarly,
${\mathscr{L}}^{1}$ is either $\ell^{1}({\mathbb{Z}}_{+})$ or
$L^{1}({\mathbb{R}}_{+})$. For $p\in\\{1,\infty\\}$, the norm in
${\mathscr{L}}^{p}$ is denoted by $\|\cdot\|_{p}$. Given
${\mathscr{V}}\subseteq{\mathbb{X}}$, the linear span of ${\mathscr{V}}$,
denoted by $\mathrm{span}{\mathscr{V}}$, is a linear subspace of
${\mathbb{X}}$ containing linear combination of the elements of
${\mathscr{V}}$. Let ${\mathcal{Y}}$ be a set and
${\mathcal{C}}\subseteq{\mathcal{Y}}$. We define the function
$\delta_{{\mathcal{C}}}$ as $\delta_{{\mathcal{C}}}(y)=0$, if
$y\in{\mathcal{C}}$, and $\delta_{{\mathcal{C}}}(y)=\infty$, otherwise.
Similarly, function $\mathbf{1}_{{\mathcal{C}}}$ is defined as
$\mathbf{1}_{{\mathcal{C}}}(y)=1$, if $y\in{\mathcal{C}}$ and
$\mathbf{1}_{{\mathcal{C}}}(y)=0$, otherwise. With respect to each bounded
signal
${\mathrm{u}}=(u_{s})_{s\in{\mathbb{T}}_{\pm}}\in{\mathscr{L}}^{\infty}$ and
each $t\in{\mathbb{T}}_{\pm}$, the linear map
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}:{\mathscr{L}}^{1}\to{\mathbb{R}}$ is
defined as
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}):=\sum_{s\in{\mathbb{Z}}_{+}}g_{s}u_{t-s}$,
when ${\mathbb{T}}={\mathbb{Z}}_{+}$, and
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}):=\int_{{\mathbb{R}}_{+}}\\!\\!g_{s}u_{t-s}\
\\!\mathrm{d}s$, when ${\mathbb{T}}={\mathbb{Z}}_{+}$.

##  III Identification with Steady-State Gain Side-Information

Let ${\mathcal{S}}$ be a stable LTI system with impulse response
${\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}}:=({g}^{{\scalebox{0.5}{$({\mathcal{S}})$}}}_{t})_{t\in{\mathbb{T}}}\in{\mathbb{R}}^{\mathbb{T}}$,
where ${\mathbb{T}}:={\mathbb{Z}}_{+}$, for the case of discrete-time, and,
${\mathbb{T}}:={\mathbb{R}}_{+}$, for the case of continuous-time. The
_steady-state gain_ system ${\mathcal{S}}$ is equal to
$\ell_{\text{{\scalebox{0.75}{$0$}}}}({\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}})$,
where $\ell_{0}$ is a real-valued linear operator defined on the space of
stable impulse responses as following

$\ell_{\text{{\scalebox{0.75}{$0$}}}}({\mathrm{g}}):=\begin{cases}\sum_{t\in{\mathbb{Z}}_{+}}g_{t},&\text{
if }{\mathbb{T}}={\mathbb{Z}}_{+},\\\ \int_{{\mathbb{R}}_{+}}\\!\\!g_{t}\
\\!\mathrm{d}t,&\text{ if }{\mathbb{T}}={\mathbb{R}}_{+},\\\ \end{cases}$ (1)

for any ${\mathrm{g}}=(g_{t})_{t\in{\mathbb{T}}}\in{\mathscr{L}}^{1}$.

Let ${\mathrm{u}}=(u_{t})_{t\in{\mathbb{T}}}$ be a bounded signal applied to
the input of system ${\mathcal{S}}$, and the corresponding output be measured
at time instants

${\mathscr{T}}:=\\{t_{i}\ \\!|\\!\
i=1,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\\},$ (2)

where $n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\in{\mathbb{N}}$ denotes the
number of measurement samples. From the definition of
${\mathrm{L}}^{\\!{\mathrm{u}}}_{t}$, the measured output of the system at
time instant $t\in{\mathscr{T}}$, denoted by $y_{t}$, is

$y_{t}:={\mathrm{L}}^{\\!{\mathrm{u}}}_{t}({\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}})+w_{t},\qquad
t\in{\mathscr{T}},$ (3)

where $\\{w_{t}|t\in{\mathscr{T}}\\}$ are the measurement uncertainty.
Consequently, we are provided with the set of input-output data
${\mathscr{D}}$ defined as

${\mathscr{D}}=\\{(u_{t},y_{t})\ |\ t\in{\mathscr{T}}\\}.$ (4)

In addition to ${\mathscr{D}}$, suppose that we know the steady-state gain of
the system. Accordingly, one may ask whether the given steady-state gain side-
information is naturally preserved and encoded in the identification of system
${\mathcal{S}}$. We elucidate this issue in the following numerical example.

######  Example.

Consider continuous-time system ${\mathcal{S}}$ described by the following
transfer function

$G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}(s)=\frac{s+2}{s^{2}+s+2},$ (5)

with the step response denoted by
${\mathrm{s}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}}$. The system is initially
at rest, and then actuate it by a random switching pulse signal in the time
interval $[0,100]$. To obtain the set of data ${\mathscr{D}}$ in (4), the
output of system is uniformly measured with the sampling frequency of $2$ Hz
and the signal-to-noise ratio (SNR) of $20$ dB. Furthermore, let the steady-
state gain of the system be given, i.e., we know that
$G^{{\scalebox{0.5}{$({\mathcal{S}})$}}}(0)=1$. The impulse response of the
system can be estimated using _direct_ and _indirect_ methods [53]. In the
direct approach, we use the tfsrivc function provided by CONTSID Toolbox [54]
with the known order of the system. Let $\hat{{\mathrm{g}}}_{1}$ and
$\hat{{\mathrm{s}}}_{1}$ respectively denote the impulse response and the step
response of the resulting estimated system. Also, we identify the system
indirectly by employing the n4sid function available in Matlab’s System
Identification Toolbox [55] to estimate a discrete-time model, and
subsequently, the continuous-time impulse response is obtained from a linear
interpolation of the discrete-time estimate. Let the resulting impulse and
step responses be denoted by $\hat{{\mathrm{g}}}_{2}$ and
$\hat{{\mathrm{s}}}_{2}$, respectively. As shown in Figure 1, the steady-state
values for $\hat{{\mathrm{s}}}_{1}$ and $\hat{{\mathrm{s}}}_{2}$ are
respectively $0.85$ and $1.22$, meaning that the steady-state gains have a
$15\%$ and a $22\%$ error. Consequently, one can observe that the estimated
models do not take into account the steady-state gain side-information.

Figure 1: The step responses for system ${\mathcal{S}}$ and the estimated
models.

Motivated by this example, the main problem discussed in this paper is the
identification with side-information on the steady-state gain of the system.
More precisely, we address the identification problem introduced below.

######  Problem 1.

Given the set of data ${\mathscr{D}}$, estimate the impulse response of stable
system ${\mathcal{S}}$ satisfying the side-information
$\ell_{\text{{\scalebox{0.75}{$0$}}}}({\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}})\in[\underline{\delta},\overline{\delta}]$,
where $\underline{\delta}$ and $\overline{\delta}$ are given bounds for the
steady-state gain of system ${\mathcal{S}}$.

Compared to the above example, Problem 1 addresses the more common scenario in
which the available side-information on the steady-state gain is imprecise and
provided in the form of interval $[\underline{\delta},\overline{\delta}]$.
When the steady-state gain of the system is known to be exactly equal to
$\delta$, we set $\underline{\delta}=\overline{\delta}=\delta$. The precise
formulation of Problem 1 is discussed in the next section.

##  IV The Estimation Problem: Existence and Uniqueness of the Solution

In this section, we formulate a constrained regularized empirical loss
minimization to address estimation Problem 1. For this purpose, in addition to
an appropriate objective function, we introduce a suitable hypothesis space
characterizing the feasible set of the optimization problem and a constraint
encoding the side-information about the steady-state gain of the system.
Furthermore, we study the existence and uniqueness property for the solution
of the resulting problem.

###  IV-A Stable Reproducing Kernel Hilbert Spaces

The hypothesis space taken for the estimation problem is a reproducing kernel
Hilbert space (RKHS) [56], which contains stable impulse responses. Based on
the structure of RKHS, we can investigate the problem and obtain a tractable
approach for solving the estimation problem. These features are provided by
the _kernel function_ , which characterizes the RKHS uniquely and completely.

######  Definition 1 ([56]).

Let $\mathbbm{k}:{\mathbb{T}}\times{\mathbb{T}}\to{\mathbb{R}}$ be a non-zero
symmetric function, which is assumed to be continuous when
${\mathbb{T}}={\mathbb{R}}_{+}$. Then, $\mathbbm{k}$ is called a Mercer kernel
if we have

$\sum_{i=1}^{m}\sum_{j=1}^{m}a_{i}\mathbbm{k}(t_{i},t_{j})a_{j}\geq 0,$ (6)

for any $m\in{\mathbb{N}}$, $t_{1},\ldots,t_{m}\in{\mathbb{T}}$ and
$a_{1},\ldots,a_{m}\in{\mathbb{R}}$. Moreover, the section of kernel
$\mathbbm{k}$ at $t\in{\mathbb{T}}$, denoted by $\mathbbm{k}_{t}$, is the
function defined as $\mathbbm{k}(t,\cdot):{\mathbb{T}}\to{\mathbb{R}}$.

######  Theorem 1 ([56]).

Given Mercer kernel
$\mathbbm{k}:{\mathbb{T}}\times{\mathbb{T}}\to{\mathbb{R}}$, there exists a
unique Hilbert space
${\mathscr{H}}_{\mathbbm{k}}\subseteq{\mathbb{R}}^{{\mathbb{T}}}$ equipped
with inner product
${\langle{\cdot,\cdot}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}$, called a _RKHS
with kernel_ $\mathbbm{k}$, such that, for any $t\in{\mathbb{T}}$, we have

  * i)

$\mathbbm{k}_{t}\in{\mathscr{H}}_{\mathbbm{k}}$, and

  * ii)

${\langle{{\mathrm{g}},\mathbbm{k}_{t}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}=g_{t}$,
for all
${\mathrm{g}}=(g_{t})_{t\in{\mathbb{T}}}\in{\mathscr{H}}_{\mathbbm{k}}$.

The second feature is called the reproducing property.

According to Theorem 1, a RKHS is completely characterized by the
corresponding kernel. As we are interested in the stable impulse responses in
the bounded-input-bounded-output (BIBO) sense, we need to employ a kernel such
that we have ${\mathscr{H}}_{\mathbbm{k}}\subseteq{\mathscr{L}}^{1}$. The
following theorem provides a necessary and sufficient condition for this
feature.

######  Theorem 2 ([26, 57]).

Let $\mathbbm{k}:{\mathbb{T}}\times{\mathbb{T}}\to{\mathbb{R}}$ be a Mercer
kernel. Then, $\mathbbm{k}$ is stable if and only if, for any
${\mathrm{u}}=(u_{s})_{s\in{\mathbb{T}}}\in{\mathscr{L}}^{\infty}$, we have

$\sum_{t\in{\mathbb{Z}}_{+}}\bigg{|}\sum_{s\in{\mathbb{Z}}_{+}}u_{s}\mathbbm{k}(t,s)\bigg{|}<\infty,$
(7)

when ${\mathbb{T}}={\mathbb{Z}}_{+}$, and,

$\int_{{\mathbb{R}}_{+}}\bigg{|}\int_{{\mathbb{R}}_{+}}u_{s}\mathbbm{k}(t,s)\mathrm{d}s\bigg{|}\mathrm{d}t<\infty,$
(8)

when ${\mathbb{T}}={\mathbb{R}}_{+}$. The kernel $\mathbbm{k}$ is said to be
_stable_ when it satisfies this property.

The following stable kernels are frequently used in the literature [26]:

  * •

_diagonally/correlated_ (DC) kernel:

$\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}(s,t)=\alpha^{\max(s,t)}\gamma^{|s-t|},$
(9)

  * •

_tuned/correlated_ (TC) kernel:

$\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}(s,t)=\alpha^{\max(s,t)},$
(10)

  * •

_stable spline_ (SS) kernel:

$\mathbbm{k}_{\SS}(s,t)=\alpha^{\max(s,t)+s+t}-\frac{1}{3}\alpha^{3\max(s,t)},$
(11)

where $\alpha\in(0,1)$,
$\gamma\in(-\alpha^{-\frac{1}{2}},\alpha^{-\frac{1}{2}})$, if
${\mathbb{T}}={\mathbb{Z}}_{+}$, and, $\gamma\in(0,\alpha^{-\frac{1}{2}})$, if
${\mathbb{T}}={\mathbb{R}}_{+}$.

###  IV-B Empirical Loss and Regularization Function

Given the set of data ${\mathscr{D}}$ and the hypothesis space
${\mathscr{H}}_{\mathbbm{k}}$, the _empirical loss function_ ,
${\mathcal{E}}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}_{+}$, can be defined
as the sum of squared errors, i.e., for each
${\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}$, we have

${\mathcal{E}}({\mathrm{g}}):=\sum_{i=1}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}\big{(}{\mathrm{L}}^{\\!{\mathrm{u}}}_{t_{i}}({\mathrm{g}})-y_{t_{i}}\big{)}^{2}.$
(12)

We can consider a more general form for the empirical loss function. More
precisely, let
${\mathcal{I}}:=\\{i_{k}|k=1,\ldots,n_{{\scalebox{0.5}{${\mathcal{I}}$}}}\\}$
be a subset of $\\{1,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\\}$,
${\mathrm{y}}_{{\mathcal{I}}}$ be the vector defined as
${\mathrm{y}}_{{\mathcal{I}}}=[y_{t_{i}}]_{i\in{\mathcal{I}}}$, and
$\ell:{\mathbb{R}}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}\times{\mathbb{R}}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}\to{\mathbb{R}}_{+}$
be a given convex function. Accordingly, we define the generalized _loss
function_ ,
${\mathcal{E}}_{\ell}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}_{+}$ as
follows

${\mathcal{E}}_{\ell}({\mathrm{g}}):=\ell([{\mathrm{L}}^{\\!{\mathrm{u}}}_{t_{i}}({\mathrm{g}})]_{i\in{\mathcal{I}}},{\mathrm{y}}_{{\mathcal{I}}}),\qquad\forall{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}},$
(13)

where subscript $\ell$ is considered to highlight the role of function $\ell$
in the definition of ${\mathcal{E}}_{\ell}$. When
${\mathcal{I}}=\\{1,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\\}$ and
function
$\ell:{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}\times{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}\to{\mathbb{R}}_{+}$
is defined as
$\ell({\mathrm{v}}_{1},{\mathrm{v}}_{2})=\|{\mathrm{v}}_{1}-{\mathrm{v}}_{2}\|^{2}$,
for any
${\mathrm{v}}_{1},{\mathrm{v}}_{2}\in{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}$,
the empirical loss ${\mathcal{E}}_{\ell}$ reduces to the special case
introduced in (12). Also, to be robust with respect to outliers, one may take
function
$\ell:{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}\times{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}\to{\mathbb{R}}_{+}$
as

$\ell({\mathrm{v}}_{1},{\mathrm{v}}_{2})=\sum_{i=1}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}L_{\sigma}(\left[{\mathrm{v}}_{1}\right]_{(i)}-\left[{\mathrm{v}}_{2}\right]_{(i)}),\quad\forall{\mathrm{v}}_{1},{\mathrm{v}}_{2}\in{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}},$
(14)

where, for given $\sigma\in{\mathbb{R}}_{+}$, function
$L_{\sigma}:{\mathbb{R}}\to{\mathbb{R}}_{+}$ is the _Huber loss_ defined as
follows

$L_{\sigma}(e)=\begin{cases}\frac{1}{2}e^{2},&\text{ if }|e|\leq\sigma,\\\
\sigma(|e|-\frac{1}{2}\sigma),&\text{ otherwise, }\\\ \end{cases}$ (15)

or, it can be the smoothed version of (15), known as the _pseudo-Huber_
function [58], which is

$L_{\sigma}(e)=(e^{2}+\sigma^{2})^{\frac{1}{2}}-\sigma^{2},\quad\forall
e\in{\mathbb{R}}.$ (16)

The resulting empirical loss function ${\mathcal{E}}_{\ell}$ is more suitable
to the cases where output measurements are subject to noise disturbances with
large outliers.

Since the hypothesis space is a RKHS endowed with kernel $\mathbbm{k}$, we
define the regularization term based on the norm in
${\mathscr{H}}_{\mathbbm{k}}$, i.e.,
$\|\cdot\|_{{\mathscr{H}}_{\mathbbm{k}}}$. More precisely, let
$\rho:{\mathbb{R}}_{+}\to{\mathbb{R}}_{+}$ be a strictly increasing convex
function. Then, the regularization function,
${\mathcal{R}}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}_{+}$, is defined as
${\mathcal{R}}({\mathrm{g}})=\rho(\|{\mathrm{g}}\|_{{\mathscr{H}}_{\mathbbm{k}}})$,
for any ${\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}$. Accordingly, the
objective function for the estimation problem,
${\mathcal{J}}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}_{+}$, is defined as
following

${\mathcal{J}}({\mathrm{g}}):={\mathcal{E}}_{\ell}({\mathrm{g}})+\lambda{\mathcal{R}}({\mathrm{g}}),\qquad\forall{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}},$
(17)

where $\lambda>0$ is the regularization weight.

###  IV-C Steady-State Gain Side-Information

Define the set
${\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])\subset{\mathscr{H}}_{\mathbbm{k}}$
as follows

${\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}]):=\Big{\\{}{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}\
\\!\Big{|}\
\\!\ell_{\text{{\scalebox{0.75}{$0$}}}}({\mathrm{g}})\in[\underline{\delta},\overline{\delta}]\Big{\\}}.$
(18)

The elements of
${\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])$ are
exactly the ones satisfying the side-information on the steady-state gain of
the system. Therefore, the estimation Problem 1 is formulated as the following
optimization problem

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}}&\
{\mathcal{E}}_{\ell}({\mathrm{g}})+\lambda{\mathcal{R}}({\mathrm{g}})\\\
\mathrm{s.t.}&\
{\mathrm{g}}\in{\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}]).\end{array}$
(19)

The existence and uniqueness of the solution of optimization problem (19)
depends on the topological properties of set
${\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])$ which is
characterized by operator
$\ell_{\text{{\scalebox{0.75}{$0$}}}}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}$.
To study these properties, we need the notion of integrable kernels [26].

######  Definition 2.

The Mercer kernel $\mathbbm{k}:{\mathbb{T}}\times{\mathbb{T}}\to{\mathbb{R}}$
is said to be _integrable_ if

$\int_{{\mathbb{R}}_{+}}\int_{{\mathbb{R}}_{+}}|\mathbbm{k}(s,t)|\
\\!\mathrm{d}s\mathrm{d}t<\infty,$ (20)

when ${\mathbb{T}}={\mathbb{R}}_{+}$, or, if

$\sum_{s\in{\mathbb{Z}}_{+}}\sum_{t\in{\mathbb{Z}}_{+}}|\mathbbm{k}(s,t)|<\infty,$
(21)

when ${\mathbb{T}}={\mathbb{Z}}_{+}$.

One can easily see that each of the kernels
$\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$,
$\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$ and
$\mathbbm{k}_{\SS}$ is integrable (see Appendix A-A). Moreover, for any
integrable kernel $\mathbbm{k}$ and any
${\mathrm{u}}=(u_{s})_{s\in{\mathbb{T}}}\in{\mathscr{L}}^{\infty}$, we have

$\sum_{t\in{\mathbb{Z}}_{+}}\Big{|}\sum_{s\in{\mathbb{Z}}_{+}}u_{s}\mathbbm{k}(t,s)\Big{|}\leq\|{\mathrm{u}}\|_{\infty}\sum_{t\in{\mathbb{Z}}_{+}}\sum_{s\in{\mathbb{Z}}_{+}}|\mathbbm{k}(t,s)|<\infty,$

when ${\mathbb{T}}={\mathbb{Z}}_{+}$, and,

$\int_{{\mathbb{R}}_{+}}\\!\bigg{|}\int_{{\mathbb{R}}_{+}}\\!u_{s}\mathbbm{k}(t,s)\mathrm{d}s\bigg{|}\mathrm{d}t\leq\|{\mathrm{u}}\|_{\infty}\int_{{\mathbb{R}}_{+}}\\!\int_{{\mathbb{R}}_{+}}\\!|\mathbbm{k}(t,s)|\mathrm{d}s\mathrm{d}t<\infty,$

when ${\mathbb{T}}={\mathbb{R}}_{+}$. Accordingly, the integrable kernels are
stable. The main importance of integrable kernels in this paper is highlighted
by the following theorems.

######  Theorem 3.

Let $\mathbbm{k}:{\mathbb{Z}}_{+}\times{\mathbb{Z}}_{+}\to{\mathbb{R}}$ be an
integrable Mercer kernel, and
$\varphi_{0}=(\varphi_{0,t})_{t\in{\mathbb{Z}}_{+}}$ be defined as following

$\varphi_{0,t}=\sum_{s\in{\mathbb{Z}}_{+}}\mathbbm{k}(t,s),\quad\forall\\!\
t\in{\mathbb{Z}}_{+}.$ (22)

Then, $\varphi_{0}$ is well-defined and
$\varphi_{0}\in{\mathscr{H}}_{\mathbbm{k}}$. Moreover, we have

$\ell_{\text{{\scalebox{0.75}{$0$}}}}({\mathrm{g}})={\langle{\varphi_{0},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}},\qquad\forall{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}.$
(23)

Furthermore, one can see

$\|\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}={\langle{\varphi_{0},\varphi_{0}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}=\sum_{s,t\in{\mathbb{Z}}_{+}}\mathbbm{k}(t,s).$
(24)

###### Proof.

For each $n\in{\mathbb{Z}}_{+}$, define
${\mathrm{f}}_{n}=(f_{n,s})_{s\in{\mathbb{Z}}_{+}}$ as
${\mathrm{f}}_{n}=\sum_{t=0}^{n}\mathbbm{k}_{t}$. Since, for each
$t\in{\mathbb{Z}}_{+}$, one has
$\mathbbm{k}_{t}\in{\mathscr{H}}_{\mathbbm{k}}$, we know that
${\mathrm{f}}_{n}\in{\mathscr{H}}_{\mathbbm{k}}$. Furthermore, from the
reproducing property, one can see that

$\|{\mathrm{f}}_{n}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}=\sum_{s=0}^{n}\sum_{t=0}^{n}\mathbbm{k}(s,t).$
(25)

Since $\mathbbm{k}$ is an integrable kernel, we know that $\sum_{s,t\geq
0}|\mathbbm{k}(s,t)|<\infty$. Accordingly, for any positive real scalar
$\varepsilon$, there exists $N\in{\mathbb{Z}}_{+}$ such that $\sum_{s,t\geq
N}|\mathbbm{k}(s,t)|\leq\varepsilon^{2}$. Let $N_{\varepsilon}$ denote
smallest non-negative integer with this property. For any
$n,m\in{\mathbb{Z}}_{+}$ such that $n>m\geq N_{\varepsilon}$, we have
${\mathrm{f}}_{n}-{\mathrm{f}}_{m}=\sum_{t=m+1}^{n}\\!\mathbbm{k}_{t}$.
Accordingly, from the reproducing property, one can see that

$\|{\mathrm{f}}_{n}-{\mathrm{f}}_{m}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}=\sum_{s=m+1}^{n}\sum_{t=m+1}^{n}\mathbbm{k}(s,t).$
(26)

Subsequently, from $n,m\geq N_{\varepsilon}$, the triangle inequality and the
definition of $N_{\varepsilon}$, it follows that
$\|{\mathrm{f}}_{n}-{\mathrm{f}}_{m}\|_{{\mathscr{H}}_{\mathbbm{k}}}\leq\varepsilon$.
Therefore, $\\{{\mathrm{f}}_{n}\\}_{n\geq 0}$ is a Cauchy sequence in Hilbert
space ${\mathscr{H}}_{\mathbbm{k}}$, and consequently, we know that there
exists
${\mathrm{f}}=(f_{s})_{s\in{\mathbb{Z}}_{+}}\in{\mathscr{H}}_{\mathbbm{k}}$
such that
$\lim_{n\to\infty}\|{\mathrm{f}}_{n}-{\mathrm{f}}\|_{{\mathscr{H}}_{\mathbbm{k}}}=0$.
For any $s\in{\mathbb{Z}}_{+}$, from the reproducing property, we know that
$f_{s}-f_{n,s}={\langle{{\mathrm{f}}-{\mathrm{f}}_{n},\mathbbm{k}_{s}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}$.
Accordingly, due to the Cauchy-Schwartz inequality, we have

$\lim_{n\to\infty}|f_{s}-f_{n,s}|\leq\lim_{n\to\infty}\|{\mathrm{f}}-{\mathrm{f}}_{n}\|_{{\mathscr{H}}_{\mathbbm{k}}}\\!\
\|\mathbbm{k}_{s}\|_{{\mathscr{H}}_{\mathbbm{k}}}=0,$ (27)

i.e., one has $\lim_{n\to\infty}f_{n,s}=f_{s}$, for any
$s\in{\mathbb{Z}}_{+}$, which implies that
${\mathrm{f}}=\sum_{t\in{\mathbb{Z}}_{+}}\mathbbm{k}_{t}$. Therefore,
${\mathrm{f}}$ coincides with $\varphi_{0}$ defined by (22). Moreover, from
$\lim_{n\to\infty}{\mathrm{f}}_{n}={\mathrm{f}}$, the dominated convergence
theorem and being $\mathbbm{k}$ an integrable kernel, we have

$\begin{split}\\!\\!\big{\|}{\mathrm{f}}\big{\|}_{{\mathscr{H}}_{\mathbbm{k}}}^{2}=\\!\lim_{n\to\infty}\\!\big{\|}{\mathrm{f}}_{n}\big{\|}_{{\mathscr{H}}_{\mathbbm{k}}}^{2}=\lim_{n\to\infty}\\!\sum_{0\leq
s,t\leq
n}\\!\\!\\!\mathbbm{k}(s,t)=\\!\\!\\!\sum_{s,t\in{\mathbb{Z}}_{+}}\\!\\!\mathbbm{k}(s,t),\end{split}$

which implies (24). For any
${\mathrm{g}}=(g_{t})_{t\in{\mathbb{Z}}_{+}}\in{\mathscr{H}}_{\mathbbm{k}}$,
we know that ${\mathrm{g}}$ is integrable, i.e.,
$\sum_{t\in{\mathbb{Z}}_{+}}|g_{t}|<\infty$. Therefore, from
$\lim_{n\to\infty}{\mathrm{f}}_{n}={\mathrm{f}}$ and the reproducing property,
it follows that

$\begin{split}\sum_{t\in{\mathbb{Z}}_{+}}g_{t}&=\lim_{n\to\infty}\sum_{0\leq
t\leq n}g_{t}\\\ &=\lim_{n\to\infty}\Big{\langle}\sum_{0\leq t\leq
n}\\!\\!\mathbbm{k}(\cdot,t),{\mathrm{g}}\Big{\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&=\lim_{n\to\infty}{\langle{{\mathrm{f}}_{n},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}={\langle{{\mathrm{f}},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}},\end{split}$
(28)

which implies (23), and concludes the proof. ∎

The following theorem is the continuous-time version of Theorem 3 which
highlights the importance of integrable kernels for the case
${\mathbb{T}}={\mathbb{R}}_{+}$.

######  Theorem 4.

Let $\mathbbm{k}:{\mathbb{R}}_{+}\times{\mathbb{R}}_{+}\to{\mathbb{R}}$ be an
integrable Mercer kernel, and
$\varphi_{0}=(\varphi_{0,t})_{t\in{\mathbb{R}}_{+}}$ be defined as following

$\varphi_{0,t}=\int_{{\mathbb{R}}_{+}}\mathbbm{k}(t,s)\mathrm{d}s,\quad\forall\\!\
t\in{\mathbb{R}}_{+}.$ (29)

Then, $\varphi_{0}$ is well-defined and
$\varphi_{0}\in{\mathscr{H}}_{\mathbbm{k}}$. Moreover, we have

$\ell_{\text{{\scalebox{0.75}{$0$}}}}({\mathrm{g}})={\langle{\varphi_{0},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}},\qquad\forall{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}.$
(30)

Furthermore, one can see

$\big{\|}\varphi_{0}\big{\|}_{{\mathscr{H}}_{\mathbbm{k}}}^{2}={\langle{\varphi_{0},\varphi_{0}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}=\int_{{\mathbb{R}}_{+}\times{\mathbb{R}}_{+}}\mathbbm{k}(t,s)\mathrm{d}s\mathrm{d}t.$
(31)

###### Proof.

See Appendix A-B. ∎

Theorem 3 and Theorem 4 say that integrability of kernel $\mathbbm{k}$ implies
that the steady-state operator
$\ell_{0}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}$ is a linear continuous
functional. Accordingly, throughout this paper, we assume $\mathbbm{k}$ is an
integrable kernel. More precisely, we make the following assumption.

######  Assumption 1.

The kernel $\mathbbm{k}$ is an integrable Mercer kernel, for which there
exists $\tau\in{\mathbb{T}}$ such that
$\sum_{s\in{\mathbb{Z}}_{+}}\mathbbm{k}(\tau,s)\neq 0$, when
${\mathbb{T}}={\mathbb{Z}}_{+}$, or,
$\int_{{\mathbb{R}}_{+}}\mathbbm{k}(\tau,s)\mathrm{d}s\neq 0$, when
${\mathbb{T}}={\mathbb{R}}_{+}$.

The next theorem describe topological properties of set
${\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])$.

######  Theorem 5.

Let Assumption 1 hold. Then, for any $\underline{\delta}$ and
$\overline{\delta}$ such that
$-\infty\leq\underline{\delta}\leq\overline{\delta}\leq\infty$, the set
${\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])$ is a
non-empty, closed and convex subset of ${\mathscr{H}}_{\mathbbm{k}}$.

###### Proof.

Let $\delta$ be a real scalar such that
$\underline{\delta}\leq\delta\leq\overline{\delta}$. We know that
$\ell_{\text{{\scalebox{0.75}{$0$}}}}(\mathbbm{k}_{\tau})=\int_{{\mathbb{R}}_{+}}\mathbbm{k}(\tau,t)\mathrm{d}t$
is non-zero. Define ${\mathrm{h}}$ as

${\mathrm{h}}=\frac{\delta}{\ell_{\text{{\scalebox{0.75}{$0$}}}}(\mathbbm{k}_{\tau})}\mathbbm{k}_{\tau}\in{\mathscr{H}}_{\mathbbm{k}}.$
(32)

One can see that
$\ell_{\text{{\scalebox{0.75}{$0$}}}}({\mathrm{h}})=\delta\in[\underline{\delta},\overline{\delta}]$,
and hence,
${\mathrm{h}}\in{\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])$.
Thus, ${\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])$ is
non-empty. From Theorem 3 and Theorem 4, we have

$\begin{split}{\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])&=\big{\\{}{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}\big{|}{\langle{\varphi_{0},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\geq\underline{\delta}\big{\\}}\\\
&\qquad\qquad\scalebox{1.5}{$\cap$}\\!\
\big{\\{}{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}\big{|}{\langle{\varphi_{0},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\leq\overline{\delta}\big{\\}}.\end{split}$
(33)

In the right-hand side of (33), each of the sets is convex and closed.
Therefore,
${\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])$ is a
convex and closed subset of ${\mathscr{H}}_{\mathbbm{k}}$ as well. This
concludes the proof. ∎

###  IV-D From Infinite to Finite Dimension

The optimization problem (19) is defined over the infinite-dimensional Hilbert
space ${\mathscr{H}}_{\mathbbm{k}}$. In the following, we show that (19)
admits a unique solution in ${\mathscr{H}}_{\mathbbm{k}}$. Furthermore, we
introduce an equivalent convex finite-dimensional program which provides a
tractable approach to address (19). To this end, we need to introduce
additional definitions and mathematical properties for (19). Theorem 5 has
provided suitable properties for the feasible set of (19). The next assumption
and lemma provide foundations to show that the objective function in (19) has
desired features which are latter employed in the main theorem of this paper
to show the existence and uniqueness for the solution of (19).

######  Assumption 2.

The operator
${\mathrm{L}}^{\\!{\mathrm{u}}}_{\tau}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}$
is continuous for each $\tau\in{\mathscr{T}}$.

When ${\mathbb{T}}={\mathbb{R}}_{+}$ and ${\mathrm{u}}$ is a step function as
in (59), one can show the continuity of
${\mathrm{L}}^{\\!{\mathrm{u}}}_{\tau}$, for any $\tau\in{\mathbb{R}}_{+}$,
based on an argument similar to the proof of Theorem 4. Also, for the case of
${\mathbb{T}}={\mathbb{Z}}_{+}$, one can easily see that Assumption 2 holds if
the system is initially at rest, or more generally, when $(u_{t})_{t\leq
t_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}}$ is finitely non-zero. Given
this assumption, we have the following theorem.

######  Lemma 6.

Let Assumption 2 hold. Then, for each $\tau\in{\mathscr{T}}$, there exists
$\varphi_{\tau}^{\text{\rm{(u)}}}=(\varphi_{\tau,t}^{\text{\rm{(u)}}})_{t\in{\mathbb{T}}}\in{\mathscr{H}}_{\mathbbm{k}}$
such that

${\mathrm{L}}^{\\!{\mathrm{u}}}_{\tau}({\mathrm{g}})={\langle{\varphi_{\tau}^{\text{\rm{(u)}}},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}},\quad\forall{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}.$
(34)

Furthermore, for any $t\in{\mathbb{T}}$, we have

$\varphi_{\tau,t}^{\text{\rm{(u)}}}=\begin{cases}\int_{{\mathbb{R}}_{+}}\mathbbm{k}(t,s)u_{\tau-s}\mathrm{d}s,&\text{
if }{\mathbb{T}}={\mathbb{R}}_{+},\\\
\sum_{s\in{\mathbb{Z}}_{+}}\mathbbm{k}(t,s)u_{\tau-s},&\text{ if
}{\mathbb{T}}={\mathbb{Z}}_{+}.\end{cases}$ (35)

###### Proof.

See Appendix A-C. ∎

Recall the index set
${\mathcal{I}}=\\{i_{k}|k=1,\ldots,n_{{\scalebox{0.5}{${\mathcal{I}}$}}}\\}$
introduced in Section IV-B. For
$k=1,\ldots,n_{{\scalebox{0.5}{${\mathcal{I}}$}}}$, let $\varphi_{k}$ be
defined as $\varphi_{t_{i}}^{\text{\rm{(u)}}}$ with $i=i_{k}$. Accordingly, we
define matrices $\Phi$ and ${\mathrm{A}}$ respectively as

$\Phi:=\big{[}{\langle{\varphi_{i},\varphi_{j}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\big{]}_{i=0,j=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}},n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}\in{\mathbb{R}}^{(n_{{\scalebox{0.5}{${\mathcal{I}}$}}}+1)\times(n_{{\scalebox{0.5}{${\mathcal{I}}$}}}+1)},$
(36)

and

${\mathrm{A}}:=\big{[}{\langle{\varphi_{i},\varphi_{j}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\big{]}_{i=1,j=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}},n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}\in{\mathbb{R}}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}\times(n_{{\scalebox{0.5}{${\mathcal{I}}$}}}+1)}.$
(37)

Note that ${\mathrm{A}}$ is a sub-matrix of $\Phi$ which contains the rows
corresponding to the index set ${\mathcal{I}}$. Denote the columns of $\Phi$
by
${\mathrm{a}}_{0},\ldots,{\mathrm{a}}_{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}\in{\mathbb{R}}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}+1}$,
i.e., we have
$\Phi=[{\mathrm{a}}_{0},\ldots,{\mathrm{a}}_{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}]$.
Since $\Phi$ is a symmetric matrix, one can see that
${\mathrm{A}}=[{\mathrm{a}}_{1},\ldots,{\mathrm{a}}_{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}]^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!\\!$.

######  Theorem 7 (Representer Theorem, [59]).

Let ${\mathscr{H}}$ be a Hilbert space endowed with inner product
${\langle{\cdot,\cdot}\rangle}_{{\mathscr{H}}}$ and
$r:{\mathbb{R}}_{+}\to{\mathbb{R}}$ be an increasing function. Consider the
following optimization problem

$\min_{{\mathrm{w}}\in{\mathscr{H}}}\
e({\langle{{\mathrm{w}}_{1},{\mathrm{w}}}\rangle}_{{\mathscr{H}}},\ldots,{\langle{{\mathrm{w}}_{m},{\mathrm{w}}}\rangle}_{{\mathscr{H}}})+r\big{(}\big{\|}{\mathrm{w}}\big{\|}_{{\mathscr{H}}}\big{)},$
(38)

where ${\mathrm{w}}_{1},\ldots,{\mathrm{w}}_{m}$ are vectors in
${\mathscr{H}}$ and $e:{\mathbb{R}}^{m}\to{\mathbb{R}}\cup\\{+\infty\\}$ is a
given function. Then, (38) has a solution in
${\mathscr{W}}:=\mathrm{span}\\{{\mathrm{w}}_{i}\\}_{i=1}^{m}$, when it admits
a solution.

Given the above definitions and theorems, we can present our main theorem.

######  Theorem 8.

Let Assumption 1 and Assumption 2 hold. Then, the optimization problem (19)
admits a _unique_ solution ${\mathrm{g}}^{\star}$. Moreover, there exists
${\mathrm{x}}^{\star}=[x_{0}^{\star},\ldots,x_{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}^{\star}]^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!\in{\mathbb{R}}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}+1}$ such that

${\mathrm{g}}^{\star}=x_{0}^{\star}\varphi_{0}+\ldots+x_{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}^{\star}\varphi_{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}=\sum_{i=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}x_{i}^{\star}\varphi_{i}.$
(39)

Furthermore, ${\mathrm{x}}^{\star}$ is the solution of following convex
program

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{x}}\in{\mathbb{R}}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}+1}}&\
\ell({\mathrm{A}}{\mathrm{x}},{\mathrm{y}}_{{\mathcal{I}}})+\lambda{\mathcal{R}}\big{(}({\mathrm{x}}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!\Phi{\mathrm{x}})^{\frac{1}{2}}\big{)}\\\ \mathrm{s.t.}&\
{\mathrm{a}}_{0}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!{\mathrm{x}}\in[\underline{\delta},\overline{\delta}].\end{array}$ (40)

###### Proof.

Let
${\mathcal{J}}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}\cup\\{+\infty\\}$ be
defined as

${\mathcal{J}}({\mathrm{g}}):={\mathcal{E}}_{\ell}({\mathrm{g}})+\lambda{\mathcal{R}}({\mathrm{g}})+\delta_{{\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])}({\mathrm{g}}),\quad\forall{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}.$
(41)

One can see that $\min_{g\in{\mathscr{H}}_{\mathbbm{k}}}{\mathcal{J}}(g)$ is
equivalent to (19). For ${\mathrm{h}}$ introduced in (32), we know
${\mathrm{h}}\in{\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])$.
Therefore, one can see that
$\delta_{{\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])}({\mathrm{h}})=0$,
and subsequently, we have

$0\leq{\mathcal{J}}({\mathrm{h}})={\mathcal{E}}_{\ell}({\mathrm{h}})+\lambda\rho(\big{\|}{\mathrm{h}}\big{\|}_{{\mathscr{H}}_{\mathbbm{k}}})<\infty,$
(42)

i.e., function ${\mathcal{J}}$ is proper. Due to Theorem 5,
${\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])$ is a
convex and closed subset of ${\mathscr{H}}_{\mathbbm{k}}$. Accordingly,
$\delta_{{\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}\cup\\{+\infty\\}$
is a proper lower semi-continuous convex function [60]. Moreover, from Lemma
6, and also the continuity and the convexity of the function $\ell$, it
follows that
${\mathcal{E}}_{\ell}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}_{+}$ is
convex and continuous. The convexity of function
$\rho:{\mathbb{R}}_{+}\to{\mathbb{R}}_{+}$ implies its continuity.
Furthermore, since $\rho$ is strictly increasing, we know that
${\mathcal{R}}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}_{+}$ is a strictly
convex continuous function. Thus,
${\mathcal{J}}:{\mathscr{H}}_{\mathbbm{k}}\to{\mathbb{R}}\cup\\{+\infty\\}$ is
a proper lower semi-continuous strictly convex function, and subsequently, the
optimization problem $\min_{g\in{\mathscr{H}}_{\mathbbm{k}}}{\mathcal{J}}(g)$
admits a unique finite solution [60], denoted by ${\mathrm{g}}^{\star}$. From
the definition of
${\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])$, Theorem
3 and Theorem 4, one has

$\delta_{{\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])}({\mathrm{g}})=\delta_{[\underline{\delta},\overline{\delta}]}({\langle{\varphi_{0},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}),\qquad\forall{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}.$
(43)

Also, according to Lemma 6, we know that

${\mathcal{E}}_{\ell}({\mathrm{g}})=\ell([{\langle{\varphi_{i},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}]_{i=1}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}},{\mathrm{y}}_{{\mathcal{I}}}),\qquad\forall{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}.$
(44)

Let function
$e:{\mathbb{R}}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}+1}\to{\mathbb{R}}_{+}\cup\\{+\infty\\}$
be defined as

$e(z_{0},\ldots,z_{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}})=\ell([z_{i}]_{i=1}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}},{\mathrm{y}}_{{\mathcal{I}}})+\lambda\delta_{[\underline{\delta},\overline{\delta}]}(z_{0}),$
(45)

for any
$[z_{0},\ldots,z_{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}]^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!\in{\mathbb{R}}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}+1}$. Then, due to
(43) and (44), one can see that

$e({\langle{\varphi_{0},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}},\ldots,{\langle{\varphi_{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}})={\mathcal{E}}_{\ell}({\mathrm{g}})+\lambda\delta_{{\mathscr{G}}_{\mathbbm{k}}([\underline{\delta},\overline{\delta}])}({\mathrm{g}}),$
(46)

for any ${\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}$. Hence, from Theorem 7
and since $\min_{g\in{\mathscr{H}}_{\mathbbm{k}}}{\mathcal{J}}(g)$ admits a
solution, it has a solution in
$\mathrm{span}\\{\varphi_{i}\\}_{i=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}$
as well. According to the uniqueness of this solution, we know that
${\mathrm{g}}^{\star}$ belongs to
$\mathrm{span}\\{\varphi_{i}\\}_{i=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}$,
i.e., ${\mathrm{g}}^{\star}$ has the parametric form given in (40). For each
${\mathrm{g}}\in\mathrm{span}\\{\varphi_{i}\\}_{i=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}$,
we know that there exists
${\mathrm{x}}=[x_{0},\ldots,x_{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}]^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!\in{\mathbb{R}}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}+1}$ such that
${\mathrm{g}}=\sum_{j=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}x_{j}\varphi_{j}$.
This implies that

$\\!\\!\\!\ell_{\text{{\scalebox{0.75}{$0$}}}}({\mathrm{g}})=\Big{\langle}\varphi_{0},\sum_{j=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}x_{j}\varphi_{j}\Big{\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\!=\sum_{j=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}{\langle{\varphi_{0},\varphi_{j}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}x_{j}={\mathrm{a}}_{0}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!{\mathrm{x}}.$ (47)

Similarly, we have

$\begin{split}\big{[}{\mathrm{L}}^{\\!{\mathrm{u}}}_{t_{i}}({\mathrm{g}})\big{]}_{i\in{\mathcal{I}}}&=\Big{[}\Big{\langle}\varphi_{i},\sum_{j=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}x_{j}\varphi_{j}\Big{\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\Big{]}_{i=1}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}\\\
&=\Big{[}\sum_{j=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}{\langle{\varphi_{i},\varphi_{j}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}x_{j}\Big{]}_{i=1}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}={\mathrm{A}}{\mathrm{x}}.\end{split}$
(48)

Moreover, from the definition of matrix $\Phi$, one can see that

$\begin{split}\|{\mathrm{g}}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}&=\Big{\langle}\sum_{i=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}x_{i}\varphi_{i},\sum_{j=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}x_{j}\varphi_{j}\Big{\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&=\sum_{i,j=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}x_{i}{\langle{\varphi_{i},\varphi_{j}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}x_{j}={\mathrm{x}}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!\Phi{\mathrm{x}}.\end{split}$ (49)

Accordingly, due to (47), (48) and (49), we obtain convex program (40) by
replacing ${\mathrm{g}}$ in (19) with its parametric form. This concludes the
proof. ∎

In the literature, it is common to employ the empirical loss (12) and the
regularization function
${\mathcal{R}}({\mathrm{g}})=\|{\mathrm{g}}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}$.
When the steady-state gain of the system is known to be
$\delta\in{\mathbb{R}}$, the resulting impulse response estimation problem is
as following

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}}&\
\operatornamewithlimits{\sum}\limits_{i=1}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}\big{(}{\mathrm{L}}^{\\!{\mathrm{u}}}_{t_{i}}({\mathrm{g}})-y_{t_{i}}\big{)}^{2}+\lambda\big{\|}{\mathrm{g}}\big{\|}_{{\mathscr{H}}_{\mathbbm{k}}}^{2}\\\
\mathrm{s.t.}&\
\ell_{\text{{\scalebox{0.75}{$0$}}}}({\mathrm{g}})=\delta.\end{array}$ (50)

The next corollary provides a closed-form solution for this optimization
problem.

######  Corollary 9.

Under the assumptions of Theorem 8, the convex program (50) has a _unique_
solution ${\mathrm{g}}^{\star}$. Moreover, there exist
${\mathrm{x}}^{\star}=[x_{0}^{\star},\ldots,x_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}^{\star}]^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!\in{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+1}$ and
$\lambda\in{\mathbb{R}}$ such that ${\mathrm{g}}^{\star}$ has the parametric
form (39) and $[{\mathrm{x}}^{\star}{}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!,\gamma^{\star}]^{{\scalebox{0.63}{$\mathsf{T}$ }}}\\!$ is a solution of
the following system of linear equations

$\begin{bmatrix}{\mathrm{Q}}&{\mathrm{a}}_{0}\\\
{\mathrm{a}}_{0}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!&0\end{bmatrix}\begin{bmatrix}{\mathrm{x}}\\\
\gamma\end{bmatrix}=\begin{bmatrix}{\mathrm{A}}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!{\mathrm{y}}\\\ \delta\end{bmatrix},$ (51)

where ${\mathrm{Q}}={\mathrm{A}}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!{\mathrm{A}}+\lambda\Phi$ and
${\mathrm{y}}=[y_{t_{i}}]_{i=1}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}\in{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}$.
Furthermore, when
$\varphi_{0},\ldots,\varphi_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}$ are
linearly independent, we have

${\mathrm{x}}^{\star}={\mathrm{Q}}^{-1}{\mathrm{A}}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!{\mathrm{y}}+\frac{\delta-{\mathrm{a}}_{0}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!{\mathrm{Q}}^{-1}{\mathrm{A}}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!{\mathrm{y}}}{{\mathrm{a}}_{0}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!{\mathrm{Q}}^{-1}{\mathrm{a}}_{0}}{\mathrm{Q}}{\mathrm{a}}_{0}.$ (52)

###### Proof.

The convex program (50) is a special case of (19) where
${\mathcal{I}}=\\{1,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\\}$,
function
$\ell:{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}\times{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}\to{\mathbb{R}}_{+}$
is defined as
$\ell({\mathrm{v}}_{1},{\mathrm{v}}_{2})=\|{\mathrm{v}}_{1}-{\mathrm{v}}_{2}\|^{2}$,
for
${\mathrm{v}}_{1},{\mathrm{v}}_{2}\in{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}$,
and, function $\rho:{\mathbb{R}}_{+}\to{\mathbb{R}}_{+}$ is defined as
$\rho(r)=r^{2}$, for $r\in{\mathbb{R}}_{+}$. Subsequently, the existence and
the uniqueness of the solution, and also the parametric form (39) are provided
by Theorem 8. Moreover, for the given $\ell$ and $\rho$, the optimization
problem (40) is reformulated as the following quadratic program

$\begin{array}[]{cl}\operatornamewithlimits{min}\limits_{{\mathrm{x}}\in{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}+1}}&\
\|{\mathrm{A}}{\mathrm{x}}-{\mathrm{y}}\|^{2}+\lambda{\mathrm{x}}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!\Phi{\mathrm{x}}\\\ \mathrm{s.t.}&\
{\mathrm{a}}_{0}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!{\mathrm{x}}=\delta.\end{array}$ (53)

One can see that (51) is the first-order necessary optimality condition for
(53), and $\gamma$ is the Lagrange multiplier corresponding to the steady-
state gain constraint ${\mathrm{a}}_{0}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!{\mathrm{x}}=\delta$. When
$\varphi_{0},\ldots,\varphi_{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}$ are
linearly independent, the Gram matrix $\Phi$ is positive definite, and
consequently, the objective function of (53) is strongly convex. Therefore,
(53) has a unique solution ${\mathrm{x}}^{\star}$. By replacing $\gamma$ with
$\delta-{\mathrm{a}}_{0}^{{\scalebox{0.63}{$\mathsf{T}$ }}}\\!{\mathrm{x}}$ in
(51) and applying matrix inversion lemma, one can solve linear system of
equations (51) and obtain ${\mathrm{x}}^{\star}$ as in (52). ∎

##  V The Optimization Problem: Settings and Algorithm

Based on Theorem 8, addressing the estimation problem 1, or equivalently,
optimization problem (19), reduces to solving the convex program (40). In this
section, we discuss how to configure this optimization problem.

The main elements of optimization problems (40) are ${\mathrm{a}}_{0}$,
${\mathrm{A}}$, and $\Phi$. We know that ${\mathrm{A}}$ is a sub-matrix of
$\Phi$ and ${\mathrm{a}}_{0}$ is the first column of $\Phi$. Hence, it
suffices to calculate the matrix $\Phi$. According to (36), the entries of
$\Phi$ are inner products
${\langle{\varphi_{i},\varphi_{j}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}$, for
$i,j\in\\{0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\\}$. To obtain the
value of these inner products, we need to calculate improper double integrals
when ${\mathbb{T}}={\mathbb{R}}_{+}$, or infinite double summations when
${\mathbb{T}}={\mathbb{Z}}_{+}$. In general, these calculations can be
performed using techniques such as numerical integration. On the other hand,
these values can be obtained analytically in certain but fairly general
situations. For example, when ${\mathbb{T}}={\mathbb{Z}}_{+}$ and the system
is initially at rest, or, when ${\mathbb{T}}={\mathbb{R}}_{+}$, the standard
kernels are employed and the input of the system is a step function. In the
remainder of this section, the details of these calculations are discussed.

###  V-A Optimization Problem Configuration: Discrete-Time Case

Let ${\mathbb{T}}={\mathbb{Z}}_{+}$ and the system be initially at rest, i.e.,
we have $u_{t}=0$, for $t<0$. Also, let the measurement time instants be
${\mathscr{T}}=\\{0,1,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1\\}$.
Given an integrable kernel
$\mathbbm{k}:{\mathbb{Z}}_{+}\times{\mathbb{Z}}_{+}\to{\mathbb{R}}$ and an
input ${\mathrm{u}}$, define matrices
${\mathrm{K}},{\mathrm{T}}_{{\mathrm{u}}}\in{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\times
n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}$ such that

$[{\mathrm{K}}]_{(i,j)}=\mathbbm{k}(i-1,j-1),\quad\forall
i,j\in\\{1,2,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\\},$ (54)

and

$[{\mathrm{T}}_{{\mathrm{u}}}]_{(i,j)}=u_{i-j},\quad\forall
i,j\in\\{1,2,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\\}.$ (55)

Following these definitions, we have the next theorem.

######  Theorem 10.

Let $\varphi\in{\mathbb{R}}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}$ be the
column vector defined as
$\varphi:=[\varphi_{0,i}]_{i=0}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}-1}$.
Then, we have

$\Phi=\begin{bmatrix}\|\varphi_{0}\|^{2}&\varphi{\mathrm{T}}_{{\mathrm{u}}}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!\\\
{\mathrm{T}}_{{\mathrm{u}}}\varphi&{\mathrm{T}}_{{\mathrm{u}}}{\mathrm{K}}{\mathrm{T}}_{{\mathrm{u}}}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!\end{bmatrix}.$ (56)

###### Proof.

From Lemma 6, the definition of matrices ${\mathrm{K}}$ and
${\mathrm{T}}_{{\mathrm{u}}}$, and since $u_{t}=0$ for $t<0$, one can see that

$\begin{split}{\langle{\varphi_{i},\varphi_{j}}\rangle}&=\sum_{s=0}^{i-1}\varphi_{j,s}u_{i-1-s}\\\
&=\sum_{s=0}^{i-1}\sum_{t=0}^{j-1}\mathbbm{k}(s,t)u_{j-1-t}u_{i-1-s}\\\
&=[{\mathrm{T}}_{{\mathrm{u}}}{\mathrm{K}}{\mathrm{T}}_{{\mathrm{u}}}^{{\scalebox{0.63}{$\mathsf{T}$
}}}\\!]_{(i,j)},\end{split}$ (57)

for any $i,j\in\\{1,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\\}$.
Moreover, due to (22), we have

${\langle{\varphi_{i},\varphi_{0}}\rangle}=\sum_{s=0}^{i-1}\varphi_{0,s}u_{i-1-s}=[{\mathrm{T}}_{{\mathrm{u}}}\varphi]_{(i)},$
(58)

for any $i\in\\{1,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}\\}$.
Following this, the claim concludes from the definition of matrix $\Phi$ and
the fact that $\Phi=\Phi^{{\scalebox{0.63}{$\mathsf{T}$ }}}\\!$. ∎

######  Remark 1.

Appendix A-D provides $\varphi_{0}$ and
$\|\varphi_{0}\|^{2}_{{\mathscr{H}}_{\mathbbm{k}}}$ for the standard kernels
introduced in (9), (10), and (11), when ${\mathbb{T}}={\mathbb{Z}}_{+}$.

###  V-B Optimization Problem Configuration: Continuous-Time Case

The set of step functions is dense in $L^{p}({\mathbb{R}})$, for
$p\in[1,\infty)$, and also, any function in $L^{\infty}({\mathbb{R}})$ is an
almost everywhere the point-wise limit of a sequence of step functions [61].
In other words, any signal of interest can be approximated arbitrarily closely
by step functions. Accordingly, for the case of
${\mathbb{T}}={\mathbb{R}}_{+}$, one can assume that the input signal
${\mathrm{u}}=(u_{t})_{t\in{\mathbb{R}}_{+}}$ is a step function. More
precisely, there exist $n_{\text{s}}\in{\mathbb{N}}$ real scalars
$\xi_{1},\ldots,\xi_{n_{\text{s}}}$ and a finite increasing sequence
$(s_{0},s_{1},\ldots,s_{n_{\text{s}}})$ in ${\mathbb{R}}_{+}$ such that we
have

$u_{t}=\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\mathbf{1}_{[s_{i},s_{i+1})}(t),\quad\forall
t\in{\mathbb{R}}_{+}.$ (59)

For ${\mathrm{u}}=(u_{t})_{t\in{\mathbb{R}}_{+}}$ given in (59), a closed-form
for $\varphi_{\tau}^{\text{\rm{(u)}}}$ can be introduced. To this end, we
require the function
$\psi:{\mathbb{R}}_{+}\times{\mathbb{R}}_{+}\times{\mathbb{R}}_{+}\to{\mathbb{R}}$
defined as

$\psi(t,a,b):=\int_{a}^{b}\mathbbm{k}(t,s)\mathrm{d}s,$ (60)

for any $a,b,t\in{\mathbb{R}}_{+}$. This function is denoted by
$\psi_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$,
$\psi_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$ or $\psi_{\SS}$ respectively
when $\mathbbm{k}$ is $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$,
$\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$ or
$\mathbbm{k}_{\SS}$.

######  Theorem 11.

For any $t\in{\mathbb{R}}_{+}$, we have

$\varphi_{\tau,t}^{\text{\rm{(u)}}}=\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\psi(t,\bar{s}_{i+1}(\tau),\bar{s}_{i}(\tau)),$
(61)

where, for $i=0,\ldots,n_{\text{s}}$, function
$\bar{s}_{i}:{\mathbb{R}}_{+}\to{\mathbb{R}}_{+}$ is defined as
$\bar{s}_{i}(\tau):=\max(\tau-s_{i},0)$, for any $\tau\in{\mathbb{R}}_{+}$.

###### Proof.

From (35) and (59), one can see that

$\begin{split}\varphi_{\tau,t}^{\text{\rm{(u)}}}&=\int_{{\mathbb{R}}_{+}}\mathbbm{k}(t,s)u_{\tau-s}\mathrm{d}s\\\
&=\int_{{\mathbb{R}}_{+}}\mathbbm{k}(t,s)\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\mathbf{1}_{[s_{i},s_{i+1})}(\tau-s)\mathrm{d}s\\\
&=\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\int_{\bar{s}_{i+1}(\tau)}^{\bar{s}_{i}(\tau)}\mathbbm{k}(t,s)\mathrm{d}s.\end{split}$
(62)

Following this, the claim is implied by the definition of function $\psi$ in
(60). ∎

For the standard kernels defined in (9), (10) and (11), one can obtain the
closed-form of $\varphi_{\tau}^{\text{\rm{(u)}}}$ using (61). To this end, we
need $\psi_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$,
$\psi_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$, and $\psi_{\SS}$ which are
provided by the next theorem.

######  Theorem 12.

Define the function
$\eta:{\mathbb{R}}_{+}\times{\mathbb{R}}_{+}\times{\mathbb{R}}_{+}\to{\mathbb{R}}$
as

$\eta(s,\tau_{1},\tau_{2})=\min(\max(t,\tau_{1}),\tau_{2}),$ (63)

for any $s,\tau_{1},\tau_{2}\in{\mathbb{R}}_{+}$, and let
$t,a,b\in{\mathbb{R}}_{+}$ such that $a\leq b$.
i) For kernel $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$, we have

$\psi_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}(t,a,b)=\big{(}\eta(t,a,b)-a\big{)}\alpha^{t}+\frac{\alpha^{b}-\alpha^{\eta(t,a,b)}}{\ln(\alpha)}.$
(64)

ii) For kernel $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$, we
have

$\begin{split}\psi_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}(t,a,b)&=\frac{\gamma^{-a}-\gamma^{-\eta(t,a,b)}}{\ln(\gamma)}(\alpha\gamma)^{t}\\\
&\qquad+\frac{(\alpha\gamma)^{b}-(\alpha\gamma)^{\eta(t,a,b)}}{\ln(\alpha\gamma)}\gamma^{-t}.\end{split}$
(65)

iii) For kernel $\mathbbm{k}_{\SS}$, we have

$\begin{split}\\!\\!\\!\\!\psi_{\SS}(t,a,b)&=\frac{\alpha^{\eta(t,a,b)}-\alpha^{a}}{\ln(\alpha)}\alpha^{2t}+\frac{\alpha^{2b}-\alpha^{2\eta(t,a,b)}}{2\ln(\alpha)}\alpha^{t}\\\
&\ \ \ \
-\frac{1}{3}\big{(}\eta(t,a,b)-a\big{)}\alpha^{3t}\\!-\\!\frac{\alpha^{3b}-\alpha^{3\eta(t,a,b)}}{9\ln(\alpha)}.\end{split}$
(66)

###### Proof.

See Appendix A-E. ∎

Similar to the previous theorem, one can obtain the closed-form of
$\varphi_{0}$ for the standard kernels (9), (10) and (11).

######  Theorem 13.

_i)_ For kernel $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$, we
have

$\varphi_{0,t}=\left(t-\frac{1}{\ln(\alpha)}\right)\alpha^{t},\quad\forall
t\in{\mathbb{R}}_{+}.$ (67)

_ii)_ For kernel $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$, we
have

$\varphi_{0,t}=-\Big{[}\frac{(1-\gamma^{t})}{\ln(\gamma)}+\frac{1}{\ln(\alpha\gamma)}\Big{]}\alpha^{t},\quad\forall
t\in{\mathbb{R}}_{+}.$ (68)

_iii)_ For kernel $\mathbbm{k}_{\SS}$, we have

$\varphi_{0,t}=\Big{[}\frac{11}{18\ln(\alpha)}\alpha^{t}-\frac{1}{\ln(\alpha)}-\frac{t\alpha^{t}}{3}\Big{]}\alpha^{2t},\quad\forall
t\in{\mathbb{R}}_{+}.$ (69)

###### Proof.

From the definition of $\varphi_{0}$ and function $\psi$, we have

$\varphi_{0,t}=\int_{0}^{\infty}\\!\mathbbm{k}(s,t)\mathrm{d}s=\lim_{b\to\infty}\int_{0}^{b}\\!\mathbbm{k}(s,t)\mathrm{d}s=\lim_{b\to\infty}\psi(t,0,b).$

Subsequently, one can see that the theorem holds according to Theorem 12. ∎

Given $\\{\varphi_{i}\\}_{i=0}^{n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}}$, we
can obtain
${\langle{\varphi_{i},\varphi_{j}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}$, for
$i,j=0,\ldots,n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}$. To this end, define
functions $\nu:{\mathbb{R}}_{+}\times{\mathbb{R}}_{+}\to{\mathbb{R}}$ and
$\bar{\nu}:{\mathbb{R}}_{+}\to{\mathbb{R}}$ respectively as

$\nu(x,y):=\int_{0}^{x}\\!\int_{0}^{y}\\!\mathbbm{k}(s,t)\mathrm{d}t\mathrm{d}s,\qquad\forall
x,y\in{\mathbb{R}}_{+},$ (70)

and

$\bar{\nu}(x):=\int_{0}^{x}\\!\int_{0}^{\infty}\\!\mathbbm{k}(s,t)\mathrm{d}t\mathrm{d}s,\qquad\forall
x\in{\mathbb{R}}_{+}.$ (71)

When $\mathbbm{k}$ is one of the standard kernels (9), (10) and (11), we
include a suitable subscript in $\nu$ and $\bar{\nu}$ to indicate the
corresponding kernel. For each $i,j\in\\{0,1,\ldots,n_{\text{s}}-1\\}$, let
functions
$\kappa_{ij}:{\mathbb{R}}_{+}\times{\mathbb{R}}_{+}\to{\mathbb{R}}_{+}$ and
$\bar{\kappa}_{i}:{\mathbb{R}}_{+}\times{\mathbb{R}}_{+}\to{\mathbb{R}}_{+}$
be defined such that, for any $\tau,\tau_{1},\tau_{2}\in{\mathbb{R}}_{+}$, we
have

$\begin{split}&\kappa_{ij}(\tau_{1},\tau_{2})=\nu\big{(}\bar{s}_{i}(\tau_{1}),\bar{s}_{j}(\tau_{2})\big{)}-\nu\big{(}\bar{s}_{i+1}(\tau_{1}),\bar{s}_{j}(\tau_{2})\big{)}\\\
&\quad-\nu\big{(}\bar{s}_{i}(\tau_{1}),\bar{s}_{j+1}(\tau_{2})\big{)}+\nu\big{(}\bar{s}_{i+1}(\tau_{1}),\bar{s}_{j+1}(\tau_{2})\big{)},\end{split}$
(72)

and

$\bar{\kappa}_{i}(\tau)=\bar{\nu}\big{(}\bar{s}_{i}(\tau)\big{)}-\bar{\nu}\big{(}\bar{s}_{i+1}(\tau)\big{)}.$
(73)

Based on these definitions, the next theorem presents the closed-form for
${\langle{\varphi_{0},\varphi_{\tau}^{\text{\rm{(u)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}$
and
${\langle{\varphi_{\tau_{1}}^{\text{\rm{(u)}}},\varphi_{\tau_{2}}^{\text{\rm{(u)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}$.

######  Theorem 14.

i) For any $\tau\in{\mathbb{R}}_{+}$, we have

$\begin{split}&{\langle{\varphi_{0},\varphi_{\tau}^{\text{\rm{(u)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}=\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\bar{\kappa}_{i}(\tau).\end{split}$
(74)

ii) For any $\tau_{1},\tau_{2}\in{\mathbb{R}}_{+}$, we have

$\begin{split}&{\langle{\varphi_{\tau_{1}}^{\text{\rm{(u)}}},\varphi_{\tau_{2}}^{\text{\rm{(u)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}=\sum_{i=0}^{n_{\text{s}}-1}\sum_{j=0}^{n_{\text{s}}-1}\xi_{i+1}\xi_{j+1}\kappa_{ij}(\tau_{1},\tau_{2}).\end{split}$
(75)

###### Proof.

i) Due to Theorem 4, Lemma 6 and (59), one has

$\begin{split}&{\langle{\varphi_{0},\varphi_{\tau}^{\text{\rm{(u)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}=\int_{0}^{\infty}\varphi_{0,s}\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\mathbf{1}_{[s_{i},s_{i+1})}(\tau-s)\mathrm{d}s\\\
&=\int_{0}^{\infty}\int_{0}^{\infty}\mathbbm{k}(s,t)\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\mathbf{1}_{[s_{i},s_{i+1})}(\tau-s)\mathrm{d}t\mathrm{d}s.\end{split}$
(76)

Accordingly, from the definition of function $\bar{\nu}$ and functions
$\bar{\kappa}_{i}$, $i=0,\ldots,n_{\text{s}}-1$, we have

$\begin{split}&\\!\\!\\!{\langle{\varphi_{0},\varphi_{\tau}^{\text{\rm{(u)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}=\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\int_{\bar{s}_{i+1}(\tau)}^{\bar{s}_{i}(\tau)}\int_{0}^{\infty}\mathbbm{k}(s,t)\mathrm{d}t\
\\!\mathrm{d}s\\\
&=\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\big{(}\bar{\nu}(\bar{s}_{i}(\tau))-\bar{\nu}(\bar{s}_{i+1}(\tau)\big{)}\\\
&=\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\bar{\kappa}_{i}(\tau).\end{split}$ (77)

ii) From Lemma 6 and due to (35) and (59), we know that

$\begin{split}&{\langle{\varphi_{\tau_{1}}^{\text{\rm{(u)}}},\varphi_{\tau_{2}}^{\text{\rm{(u)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}=\int_{0}^{\infty}\varphi_{\tau_{2},s}^{\text{\rm{(u)}}}\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\mathbf{1}_{[s_{i},s_{i+1})}(\tau_{1}-s)\mathrm{d}s\\\
&=\int_{0}^{\infty}\int_{0}^{\infty}\bigg{(}\mathbbm{k}(s,t)\sum_{j=0}^{n_{\text{s}}-1}\xi_{j+1}\mathbf{1}_{[s_{j},s_{j+1})}(\tau_{2}-t)\\\
&\qquad\qquad\qquad\sum_{i=0}^{n_{\text{s}}-1}\xi_{i+1}\mathbf{1}_{[s_{i},s_{i+1})}(\tau_{1}-s)\bigg{)}\mathrm{d}t\
\\!\mathrm{d}s\\\
&=\sum_{i=0}^{n_{\text{s}}-1}\sum_{j=0}^{n_{\text{s}}-1}\xi_{i+1}\xi_{j+1}\int_{0}^{\infty}\int_{0}^{\infty}\bigg{(}\mathbbm{k}(s,t)\\\
&\qquad\qquad\qquad\mathbf{1}_{[s_{j},s_{j+1})}(\tau_{2}-t)\mathbf{1}_{[s_{i},s_{i+1})}(\tau_{1}-s)\bigg{)}\mathrm{d}t\
\\!\mathrm{d}s.\end{split}\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!$ (78)

From (70), it follows that

$\begin{split}&\\!\\!\\!\int_{0}^{\infty}\\!\\!\\!\int_{0}^{\infty}\\!\\!\mathbbm{k}(s,t)\mathbf{1}_{[s_{j},s_{j+1})}(\tau_{2}-t)\mathbf{1}_{[s_{i},s_{i+1})}(\tau_{1}-s)\mathrm{d}t\
\\!\mathrm{d}s\\\
&=\int_{\bar{s}_{i+1}(\tau_{1})}^{\bar{s}_{i}(\tau_{1})}\\!\int_{\bar{s}_{j+1}(\tau_{2})}^{\bar{s}_{j}(\tau_{2})}\\!\mathbbm{k}(s,t)\mathrm{d}t\
\\!\mathrm{d}s\\\
&=\nu\big{(}\bar{s}_{i}(\tau_{1}),\bar{s}_{j}(\tau_{2})\big{)}-\nu\big{(}\bar{s}_{i+1}(\tau_{1}),\bar{s}_{j}(\tau_{2})\big{)}\\\
&\qquad-\nu\big{(}\bar{s}_{i}(\tau_{1}),\bar{s}_{j+1}(\tau_{2})\big{)}+\nu\big{(}\bar{s}_{i+1}(\tau_{1}),\bar{s}_{j+1}(\tau_{2})\big{)}.\end{split}$
(79)

Accordingly, one can see that (78) and (79) implies (75). ∎

Due to (72) and (73), in order to employ (74) and (75) to calculate the
entries of $\Phi$, we need to obtain the functions $\nu$ and $\bar{\nu}$,
which can be done in general using numerical techniques. However, the closed-
form of $\nu$ and $\bar{\nu}$ can be explicitly derived for the standard
kernels.

######  Theorem 15.

Let $x,y\in{\mathbb{R}}_{+}$. i) For kernel
$\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$, we have

$\nu_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}(x,y)=\frac{\min(x,y)(\alpha^{x}+\alpha^{y})}{\ln(\alpha)}+\frac{2(1-\alpha^{\min(x,y)})}{\ln(\alpha)^{2}},$
(80)

and

$\bar{\nu}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}(x)=\frac{x\alpha^{x}\ln(\alpha)+2(1-\alpha^{x})}{\ln(\alpha)^{2}}.$
(81)

ii) For kernel $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$, we
have

$\begin{split}\nu_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}(x,y)&=\frac{(1-\gamma^{-\min(x,y)})\big{(}(\alpha\gamma)^{x}+(\alpha\gamma)^{y}\big{)}}{\ln(\gamma)\ln(\alpha\gamma)}\\\
&\qquad+\frac{2-2\alpha^{\min(x,y)}}{\ln(\alpha)\ln(\alpha\gamma)},\end{split}$
(82)

and

$\bar{\nu}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}(x)=\frac{\alpha^{x}\gamma^{x}-\alpha^{x}}{\ln(\gamma)\ln(\alpha\gamma)}+\frac{2-2\alpha^{x}}{\ln(\alpha)\ln(\alpha\gamma)}.$
(83)

iii) For kernel $\mathbbm{k}_{\SS}$, we have

$\begin{split}\\!\\!\nu_{\SS}(x,y)&:=\frac{(\alpha^{\min(x,y)}-1)(\alpha^{2x}+\alpha^{2y})}{2\ln(\alpha)^{2}}\\\
&-\frac{\min(x,y)(\alpha^{3x}+\alpha^{3y})}{9\ln(\alpha)}+\frac{7-7\alpha^{3\min(x,y)}}{27\ln(\alpha)^{2}},\end{split}$
(84)

and

$\begin{split}\\!\\!\bar{\nu}_{\SS}(x)&:=\frac{14-27\alpha^{2x}+13\alpha^{3x}}{54\ln(\alpha)^{2}}-\frac{x\alpha^{3x}}{9\ln(\alpha)}.\end{split}$
(85)

###### Proof.

See Appendix A-F. ∎

Similar to the previous theorem, we can calculate the closed-form of
$\|\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}={\langle{\varphi_{0},\varphi_{0}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}$
for the standard stable kernels. The next theorem presents these closed-forms.

######  Theorem 16.

_i)_ For $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$, we have
$\|\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}=2(\ln(\alpha))^{-2}$.
_ii)_ For $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$, we have
$\|\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}=2(\ln(\alpha)\ln(\alpha\gamma))^{-2}$.
_iii)_ For $\mathbbm{k}_{\SS}$, we have
$\|\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}=7(27\ln(\alpha))^{-2}$.

###### Proof.

From (31) and the definition of function $\nu$, we have

$\begin{split}\|\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}&=\int_{0}^{\infty}\\!\int_{0}^{\infty}\\!\mathbbm{k}(s,t)\mathrm{d}s\mathrm{d}t\\\
&=\lim_{x,y\to\infty}\int_{0}^{x}\\!\int_{0}^{y}\\!\mathbbm{k}(s,t)\mathrm{d}s\mathrm{d}t=\lim_{x,y\to\infty}\nu(x,y).\end{split}$

Accordingly, the proof of the theorem follows directly from Theorem 15. ∎

Based on the above discussion, we can derive the key elements of optimization
(40) and solve Problem 1 to estimate the impulse response of system. The
outline of this procedure is summarized in Algorithm 1.

Algorithm 1 System Identification with Steady-State Gain Side-Information

1:Input: Set of data ${\mathscr{D}}$, integrable kernel $\mathbbm{k}$, index
set ${\mathcal{I}}$, convex function $\ell$, regularization weight $\lambda$,
and, real scalar $\delta$ or interval $[\underline{\delta},\overline{\delta}]$
for the steady-state gain.

2:Calculate matrix $\Phi$ in (36).

3:$\triangleright$ For discrete-time case, use Theorem 10 and Remark 1.

4:$\triangleright$ For continuous-time case and step input, use (72), (73),
Theorems 14, 15 and 16.

5:Obtain matrix ${\mathrm{A}}$ introduced in (37) as a sub-matrix of $\Phi$.

6:Obtain vector ${\mathrm{a}}_{0}$ as the first column of $\Phi$.

7:Solve convex program (40) to obtain ${\mathrm{x}}^{\star}$.

8:$\triangleright$ If the steady-state gain is known to be $\delta$ and the
empirical loss is the sum of squared errors, obtain ${\mathrm{x}}^{\star}$ by
(52) or (51).

9:Calculate $\varphi_{0}\\!$ by (22) and (29), or by Theorems 13 and 17.

10:Calculate
$\varphi_{1},\ldots,\varphi_{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}$ based on
(35).

11:$\triangleright$ For continuous-time case and step input, use Theorems 11
and 12.

12:Given ${\mathrm{x}}^{\star}$ and
$\\{\varphi_{i}\\}_{i=0}^{n_{{\scalebox{0.5}{${\mathcal{I}}$}}}}$, obtain
${\mathrm{g}}^{\star}$ based on (39).

13:Output: ${\mathrm{g}}^{\star}$ and ${\mathrm{x}}^{\star}$.

###  V-C Hyperparameter Tuning

To employ Algorithm 1, in addition to the set of data ${\mathscr{D}}$, an
appropriate stable kernel $\mathbbm{k}$ and the regularization weight
$\lambda$ are required. The general form of the kernel depends on the shape
and smoothness of the impulse response to be identified. Following specifying
the type of kernel, in addition to $\lambda$, it is required to determine the
hyperparameters $\theta_{\mathbbm{k}}$ characterizing kernel $\mathbbm{k}$.
Therefore, we need to estimate the vector of hyperparameters
$\theta:=[\lambda,\theta_{\mathbbm{k}}]$ in the admissible set
$\Theta\subseteq{\mathbb{R}}^{n_{\theta}}$. For this purpose, we use a cross-
validation mechanism equipped with a Bayesian optimization heuristic [62].
More precisely, the index set of data is partitioned into disjoint sets
${\mathcal{I}}_{\text{T}}$ and ${\mathcal{I}}_{\text{V}}$ to be utilized
respectively for training and validation. The prediction error on the
validation data, the model evaluation metric $v:\Theta\to{\mathbb{R}}$, is
defined as

$v(\theta)=\frac{1}{|{\mathcal{I}}_{\text{V}}|}\sum_{i\in{\mathcal{I}}_{\text{V}}}\big{(}y_{t_{i}}-{\mathrm{L}}^{\\!{\mathrm{u}}}_{t_{i}}({\mathrm{g}})\big{)}^{2},$
(86)

where ${\mathrm{g}}$ is the impulse response identified using the proposed
identification technique given the training data and the hyperparameters
$\theta$. Then, $\theta$ is estimated as
$\hat{\theta}:={\mathrm{argmin}}_{\theta\in\Theta}\\!\ v(\theta)$. One can see
that the dependency of model evaluation metric $v$ to the vector of
hyperparameters is in a black-box oracle form. To solve this optimization
problem, we use a Bayesian optimization algorithm such as GP-LCB, which is
available through Matlab’s bayesopt function [62].

##  VI Numerical Examples

In this section, we demonstrate and compare the performance of the proposed
method through several numerical examples.

######  Example 1.

For the example provided in Section III, we employ the proposed identification
scheme presuming that the steady-state gain of the system is given, i.e., we
know that $\ell_{0}({\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}})=1$.
Accordingly, we apply Algorithm 1 given the set of measurement data
${\mathscr{D}}$ and $\delta=1$. For the choice of kernel, we utilize
$\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$ introduced in (10).
The hyperparameters of kernel are tuned based on the cross-validation
mechanism introduced in Section V-C. To this end, the first $80$% of data
points are chosen for training and the remaining $20$% for validation. Given
this partitioning of the measurement data, the hyperparameters
$\theta=[\lambda,\alpha]$ are estimated as
${\mathrm{argmin}}_{\theta\in\Theta}\\!\ v(\theta)$, where $v$ is the model
evaluation metric defined in (86). The solution of this optimization problem
is obtained by utilizing GP-LCB Bayesian optimization approach [62].

Figure 2 shows the step response corresponding to ${\mathrm{g}}^{\star}$
together with the step responses of the models $\hat{{\mathrm{g}}}_{1}$ and
$\hat{{\mathrm{g}}}_{2}$, estimated in Section III. For the estimated impulse
response ${\mathrm{g}}^{\star}$, we have
$\ell_{0}({\mathrm{g}}^{\star})=1.00$. To evaluate and compare quantitatively
the estimated impulse responses, we employ the _coefficient of determination_
, also known as _R-squared_ , defined as

$\mathrm{fit}({\mathrm{g}})=100\times\left(1-\frac{\|{\mathrm{g}}-{\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}}\|_{2}}{\|{\mathrm{g}}^{{\scalebox{0.5}{$({\mathcal{S}})$}}}\|_{2}}\right),\quad\forall{\mathrm{g}}\in{\mathscr{H}}_{\mathbbm{k}}.$
(87)

For the estimated impulse responses, we have
$\mathrm{fit}(\hat{{\mathrm{g}}}_{1})=70.88$%,
$\mathrm{fit}(\hat{{\mathrm{g}}}_{2})=58.60$%, and
$\mathrm{fit}({\mathrm{g}}^{\star})=95.84$%. Accordingly, one can see that the
proposed scheme outperforms in terms of fitting performance and the precision
of resulting steady-state gain.  $\triangle\\!\\!$

Figure 2: The step responses for system ${\mathcal{S}}$ and the model
estimated in Example 1. The impulse response ${\mathrm{s}}^{\star}$
corresponds to the proposed method.

######  Example 2.

In this example, we compare the performance of the proposed identification
method with the existing schemes through a Monte Carlo analysis. To this end,
with respect to each $(n,r)$ in
$\\{16,17,\ldots,25\\}\times\\{0.8,0.82,\ldots,0.96\\}$, we employ Matlab’s
drss function to randomly generate a discrete-time LTI system with order $n$
and spectral radius $r$. We normalize these systems with their
${\mathcal{H}}_{2}$-norm and set them initially at rest. For each of these
systems, a random zero-mean white Gaussian input signal with length
$n_{{\scalebox{0.5}{$\\!{\mathscr{D}}$}}}=200$ is generated using Matlab’s
idinput function. By applying these input signals to the systems, we obtain
their noiseless output signals. We consider three levels of signal-to-noise
ratio (SNR): high, medium, and low, which are $5\ \\!$dB, $15\ \\!$dB, and
$25\ \\!$dB, respectively. For the additive measurement uncertainty, we
generate a zero-mean white Gaussian signal for each of these SNR levels and
each output signal. The noiseless output signal is then corrupted with the the
corresponding additive noise signals, and the resulting noisy output is
measured at time instants $t_{i}=i$, for $i=0,1,\ldots,199$. As a result, we
have $100$ sets of input-output data for each of the aforementioned SNR values
as following

$\begin{split}{\mathscr{D}}_{i}^{{{\scalebox{0.63}{$\mathrm{(}5\text{dB}\mathrm{)}$}}}}&\\!=\\!\big{\\{}(u_{s}^{{{\scalebox{0.63}{$\mathrm{(}i\mathrm{)}$}}}},y_{s}^{{{\scalebox{0.63}{$\mathrm{(}i,5\text{dB}\mathrm{)}$}}}})\big{|}s\\!=\\!0,\ldots,199\big{\\}},\
\ i\\!=\\!1,\ldots,100,\\\
{\mathscr{D}}_{i}^{{{\scalebox{0.63}{$\mathrm{(}15\text{dB}\mathrm{)}$}}}}&\\!=\\!\big{\\{}(u_{s}^{{{\scalebox{0.63}{$\mathrm{(}i\mathrm{)}$}}}},y_{s}^{{{\scalebox{0.63}{$\mathrm{(}i,15\text{dB}\mathrm{)}$}}}})\big{|}s\\!=\\!0,\ldots,199\big{\\}},\
\ i\\!=\\!1,\ldots,100,\\\
{\mathscr{D}}_{i}^{{{\scalebox{0.63}{$\mathrm{(}25\text{dB}\mathrm{)}$}}}}&\\!=\\!\big{\\{}(u_{s}^{{{\scalebox{0.63}{$\mathrm{(}i\mathrm{)}$}}}},y_{s}^{{{\scalebox{0.63}{$\mathrm{(}i,25\text{dB}\mathrm{)}$}}}})\big{|}s\\!=\\!0,\ldots,199\big{\\}},\
\ i\\!=\\!1,\ldots,100,\end{split}$

where the superscript indicates the SNR value in the respective data set. We
employ the above input-output data sets and the following identification
methods to estimate the impulse response of corresponding systems:

  * A.

This method is a modified subspace approach incorporating steady-state
features of output [47].

  * B.

This method estimates impulse response by solving a constrained optimization
problem formulated based on a behavioral approach [49].

  * C.

This method considers the interpretation of subspace identification as the
optimal multi-step ahead prediction and modifies it to a constrained least-
squares problem where the imposed equality constraint models approximately the
steady-state gain information [46].

  * D.

This method is a general Bayesian variant of the optimal multi-step ahead
predictor interpretation of subspace identification approach, where steady-
state gain information is integrated into the covariance of the prior
distribution [4].

  * E.

In this method, the step response of system is first estimated by a kernel-
based Bayesian approach, and then, the FIR is calculated using discrete
derivative [50].

  * F.

This method estimates a FIR model for the system based on a kernel-based
Bayesian approach where the steady-state gain information is enforced on the
total summation of the estimated FIR [51].

  * G.

The last method is the scheme proposed in this paper and summarized in
Algorithm 1.

The kernel-based methods E, F, and G employ the same kernel type (9) to give a
fair comparison. To evaluate and compare the estimation performances of these
methods, we employ the R-squared metric introduced in (87).

Figure 3: Box plots of the R-squared metric for the estimation results of the
methods and SNR levels discussed in Example 2. The y-axis range has been
adjusted to improve visibility in each of these box plots.

Figure 3 demonstrates and compares the values of R-squared metric for the
estimation results of the above mentioned methods and SNR levels.

Discussion: From Figure 3, we observe that the proposed identification scheme
demonstrates better estimation performance than other methods. Indeed, methods
A, B, C, and D are based on the subspace approach and prediction error
minimization. Meanwhile, the proposed scheme is a kernel-based method in which
the stability of the system is included and the model complexity tuning is
implemented based on the effective approach of estimating continuous
regularization hyperparameters rather than selecting an integer order [26,
25]. The methods E and F estimate a FIR model for the system, which can be
inexact when the spectral radius of the system is close to one, and the
impulse response of system can not be approximated well by a short FIR.
Moreover, the discrete derivative employed in method F makes the estimation
prone to numerical sensitivity, particularly when the data is noisy. On the
other hand, the proposed scheme estimates directly the impulse response
without any truncation and inexact numerical procedures such as discrete
derivatives.  $\triangle$

##  VII Conclusion

In this work, we have addressed the impulse response identification problem
when side-information on the steady-state gain of the system is provided. The
problem is formulated as a generic nonparametric identification in the form of
an infinite-dimensional constrained convex program over the RKHS of stable
impulse responses. This optimization problem is designed such that the
objective function corresponds to the regularized empirical loss, and the
imposed linear constraints enforce the integration of the given side-
information into the solution. We have shown that the steady-state gain is a
bounded operator over the employed RKHS, which results in guaranteeing the
existence and uniqueness of the solution. By using the representer theorem,
the optimization problem is reduced to a finite-dimensional convex program,
which can be solved efficiently. In the case of exact side-information,
quadratic empirical loss, and quadratic regularization, the identification
problem has a closed-form solution. Compared to the existing methods, the
proposed identification approach can utilize non-uniform measurement samples
and integrate steady-state gain side-information into the direct approach of
continuous-time system identification. Moreover, through an extensive Monte
Carlo numerical experiment, we have verified that the introduced methodology
outperforms the benchmark approaches. The proposed scheme has several features
which have led to the observed superior performance, including direct
estimation of the impulse response without any truncation and without inexact
numerical procedures such as discrete derivatives. The method uses kernel-
based regularization to enforce the BIBO stability of the estimated impulse
response, and, effective model selection and complexity tuning through
estimating continuous variables such as regularization weight and
hyperparameters.

##  Appendix A Appendix

###  A-A Integrability of the Standard Kernels

For a Mercer kernel $\mathbbm{k}$, let assume that there exist
$C\in{\mathbb{R}}_{+}$ and $\alpha\in(0,1)$ such that $|\mathbbm{k}(s,t)|\leq
C\alpha^{\frac{1}{2}(s+t)}$, for any $s,t\in{\mathbb{T}}$. Subsequently, when
${\mathbb{T}}={\mathbb{Z}}_{+}$, we have

$\begin{split}\sum_{s,t\geq 0}|\mathbbm{k}(s,t)|&\leq C\sum_{s\geq
0}\sum_{t\geq 0}\alpha^{\frac{1}{2}(s+t)}\\\ &=C\sum_{s\geq
0}\alpha^{\frac{1}{2}s}\sum_{t\geq 0}\alpha^{\frac{1}{2}t}\\\
&=\frac{C}{(1-\alpha^{\frac{1}{2}})^{2}}<\infty.\end{split}$ (88)

Similarly, when ${\mathbb{T}}={\mathbb{R}}_{+}$, one has

$\begin{split}\\!\\!\\!\\!\int_{{\mathbb{R}}_{+}\times{\mathbb{R}}_{+}}|\mathbbm{k}(s,t)|\mathrm{d}s\mathrm{d}t&\leq
C\int_{{\mathbb{R}}_{+}}\\!\int_{{\mathbb{R}}_{+}}\\!\alpha^{\frac{1}{2}(s+t)}\mathrm{d}s\mathrm{d}t\\\
&=C\int_{{\mathbb{R}}_{+}}\\!\alpha^{\frac{1}{2}s}\mathrm{d}s\int_{{\mathbb{R}}_{+}}\\!\alpha^{\frac{1}{2}t}\mathrm{d}t\\\
&=\frac{4C}{(\ln(\alpha))^{2}}<\infty.\end{split}$ (89)

Therefore, $\mathbbm{k}$ is an integrable kernel. For any
$s,t\in{\mathbb{T}}$, one can easily see that
$|\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}(s,t)|\leq\alpha^{\frac{1}{2}(s+t)}$,
$|\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}(s,t)|\leq\alpha^{\frac{1}{2}(s+t)}$,
and

$\begin{split}\\!\\!\\!\\!\\!\\!|\mathbbm{k}_{\SS}(s,t)|&=\Big{|}\alpha^{\max(s,t)}\Big{[}\alpha^{s+t}-\frac{1}{3}\alpha^{2\max(s,t)}\Big{]}\Big{|}\\\
&\leq\Big{[}|\alpha^{s+t}|+\frac{1}{3}|\alpha^{2\max(s,t)}|\Big{]}\alpha^{\max(s,t)}\\\
&\leq\frac{4}{3}\alpha^{\frac{1}{2}(s+t)}.\end{split}$ (90)

According to our previous discussion, this implies that
$\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$,
$\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$ and
$\mathbbm{k}_{\SS}$ are integrable kernels. ∎

###  A-B Proof of Theorem 4

Let $r\in{\mathbb{R}}_{+}$ and
$I_{r}:=\int_{0}^{r}\\!\int_{0}^{r}\\!\mathbbm{k}(s,t)\mathrm{d}s\mathrm{d}t$,
which is well-defined since $\mathbbm{k}$ is an integrable kernel. Define
${\mathrm{f}}^{{\scalebox{0.63}{($r$)}}}_{n}:=(f_{n,s}^{{\scalebox{0.63}{($r$)}}})_{s\in{\mathbb{R}}_{+}}\in{\mathscr{H}}_{\mathbbm{k}}$
as

$f^{{\scalebox{0.63}{($r$)}}}_{n,s}=\frac{r}{n}\sum_{i=0}^{n-1}\mathbbm{k}(\frac{i}{n}r,s),\qquad\forall
s\in{\mathbb{R}}_{+},$ (91)

for each $n\in{\mathbb{N}}$. From the reproducing property, for any
$n,m\in{\mathbb{Z}}_{+}$, we know that

$\begin{split}\Big{\langle}{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}}&,{\mathrm{f}}_{m}^{{\scalebox{0.63}{($r$)}}}\Big{\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}-I_{r}\\\
&=\Big{\langle}\frac{r}{n}\\!\sum_{i=0}^{n-1}\mathbbm{k}(\frac{i}{n}r,\cdot),\frac{r}{m}\\!\sum_{j=0}^{m-1}\mathbbm{k}(\frac{j}{m}r,\cdot)\Big{\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}-I_{r}\\\
&=\sum_{i=0}^{n-1}\sum_{j=0}^{m-1}\bigg{[}\mathbbm{k}(\frac{i}{n}r,\frac{j}{m}r)\frac{r^{2}}{nm}-\int_{\frac{i}{n}r}^{\frac{i+1}{n}r}\\!\\!\int_{\frac{j}{m}r}^{\frac{j+1}{m}r}\\!\\!\mathbbm{k}(s,t)\mathrm{d}s\mathrm{d}t\bigg{]}\\\
&=\sum_{i=0}^{n-1}\sum_{j=0}^{m-1}\int_{\frac{i}{n}r}^{\frac{i+1}{n}r}\\!\\!\int_{\frac{j}{m}r}^{\frac{j+1}{m}r}\bigg{[}\mathbbm{k}(\frac{i}{n}r,\frac{j}{m}r)-\mathbbm{k}(s,t)\bigg{]}\mathrm{d}s\mathrm{d}t.\end{split}$

Since $[0,r]^{2}$ is a compact region, continuity of $\mathbbm{k}$ implies
that $\mathbbm{k}$ is uniformly continuous on $[0,r]^{2}$. Therefore, for any
$\varepsilon>0$, there exists ${\delta}_{\varepsilon}\in{\mathbb{R}}_{+}$ such
that we have

$|\mathbbm{k}(s_{1},t_{1})-\mathbbm{k}(s_{2},t_{2})|\leq\frac{\varepsilon^{2}}{4r^{2}},$
(92)

for any $s_{1},s_{2},t_{1},t_{2}\in[0,r]$ where
$|s_{1}-s_{2}|+|t_{1}-t_{2}|\leq{\delta}_{\varepsilon}$. Accordingly, if
$n,m\geq n_{\varepsilon}$, where $n_{\varepsilon}$ is the smallest positive
integer larger than $\frac{1}{\delta_{\varepsilon}}2r$, one has

$|{\langle{{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}},{\mathrm{f}}_{m}^{{\scalebox{0.63}{($r$)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}-I_{r}|\leq\sum_{i=0}^{n-1}\sum_{j=0}^{m-1}\int_{\frac{i}{n}r}^{\frac{i+1}{n}r}\\!\\!\int_{\frac{j}{m}r}^{\frac{j+1}{m}r}\\!\frac{\varepsilon^{2}}{4r^{2}}\\!\
\mathrm{d}s\mathrm{d}t=\frac{1}{4}\varepsilon^{2}.$

Therefore, we know that

$I_{r}-\frac{1}{4}\varepsilon^{2}\leq{\langle{{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}},{\mathrm{f}}_{m}^{{\scalebox{0.63}{($r$)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\leq
I_{r}+\frac{1}{4}\varepsilon^{2},\quad\forall n,m\geq n_{\varepsilon}.$ (93)

Subsequently, one can see that

$\begin{split}\|{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}}-{\mathrm{f}}_{m}^{{\scalebox{0.63}{($r$)}}}\|^{2}&={\langle{{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}},{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}-2{\langle{{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}},{\mathrm{f}}_{m}^{{\scalebox{0.63}{($r$)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}+{\langle{{\mathrm{f}}_{m}^{{\scalebox{0.63}{($r$)}}},{\mathrm{f}}_{m}^{{\scalebox{0.63}{($r$)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&\leq(I+\frac{1}{4}\varepsilon^{2})-2(I-\frac{1}{4}\varepsilon^{2})+(I+\frac{1}{4}\varepsilon^{2})=\varepsilon^{2},\end{split}$

for any $n,m\geq n_{\varepsilon}$. Accordingly,
$\\{{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}}\\}_{n=1}^{\infty}$ is a Cauchy
sequence in the Hilbert space ${\mathscr{H}}_{\mathbbm{k}}$, which implies
that there exists
${\mathrm{f}}^{{\scalebox{0.63}{($r$)}}}=(f_{s}^{{\scalebox{0.63}{($r$)}}})_{s\in{\mathbb{R}}_{+}}$
in ${\mathscr{H}}_{\mathbbm{k}}$ such that
$\\{{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}}\\}_{n=1}^{\infty}$ converges
to ${\mathrm{f}}^{{\scalebox{0.63}{($r$)}}}$. Due to the reproducing property,
we know that
$f_{s}^{{\scalebox{0.63}{($r$)}}}-f_{n,s}^{{\scalebox{0.63}{($r$)}}}={\langle{{\mathrm{f}}^{{\scalebox{0.63}{($r$)}}}-{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}},\mathbbm{k}_{s}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}$,
for any $s\in{\mathbb{R}}_{+}$. Consequently, from the Cauchy-Schwartz
inequality, it follows that

$\lim_{n\to\infty}|f_{s}^{{\scalebox{0.63}{($r$)}}}-f_{n,s}^{{\scalebox{0.63}{($r$)}}}|\leq\lim_{n\to\infty}\|{\mathrm{f}}^{{\scalebox{0.63}{($r$)}}}-{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}}\|_{{\mathscr{H}}_{\mathbbm{k}}}\\!\
\|\mathbbm{k}_{s}\|_{{\mathscr{H}}_{\mathbbm{k}}}=0,$ (94)

i.e., we have
$\lim_{n\to\infty}f_{n,s}^{{\scalebox{0.63}{($r$)}}}=f_{s}^{{\scalebox{0.63}{($r$)}}}$.
On the other hand, according to (92), one can see that

$\begin{split}|f_{n,s}^{{\scalebox{0.63}{($r$)}}}\\!\\!\\!\\!&\ \
-\int_{0}^{r}\\!\mathbbm{k}(s,t)\mathrm{d}t|=\bigg{|}\sum_{i=0}^{n-1}\int_{\frac{i}{n}r}^{\frac{i+1}{n}r}\\!\Big{[}\mathbbm{k}(\frac{i}{n}r,s)-\mathbbm{k}(t,s)\Big{]}\mathrm{d}t\bigg{|}\\\
&\\!\\!\\!\\!\leq\sum_{i=0}^{n-1}\int_{\frac{i}{n}r}^{\frac{i+1}{n}r}\\!\Big{|}\mathbbm{k}(\frac{i}{n}r,s)-\mathbbm{k}(t,s)\Big{|}\mathrm{d}t\\\
&\\!\\!\\!\\!\leq\sum_{i=0}^{n-1}\frac{1}{n}\frac{\varepsilon^{2}}{4r^{2}}\\\
&\\!\\!\\!\\!=\frac{\varepsilon^{2}}{4r^{2}}.\end{split}$

Subsequently, we have
$f_{s}^{{\scalebox{0.63}{($r$)}}}=\lim_{n\to\infty}f_{n,s}^{{\scalebox{0.63}{($r$)}}}=\int_{0}^{r}\mathbbm{k}(s,t)\mathrm{d}t$,
i.e.,
${\mathrm{f}}^{{\scalebox{0.63}{($r$)}}}=\int_{0}^{r}\mathbbm{k}(\cdot,t)\mathrm{d}t$.
Accordingly, from
$\lim_{n\to\infty}{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}}={\mathrm{f}}^{{\scalebox{0.63}{($r$)}}}$,
(93) and the definition of $I_{r}$, it follows that

$\begin{split}\Big{\|}\int_{0}^{r}&\\!\mathbbm{k}(\cdot,t)\mathrm{d}t\Big{\|}_{{\mathscr{H}}_{\mathbbm{k}}}^{2}=\|{\mathrm{f}}^{{\scalebox{0.63}{($r$)}}}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}\\\
&=\operatornamewithlimits{lim}\limits_{n\to\infty}\|{\mathrm{f}}_{n}\|^{2}\\\
&=\operatornamewithlimits{lim}\limits_{n\to\infty}{\langle{{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}},{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}=\int_{0}^{r}\\!\int_{0}^{r}\\!\mathbbm{k}(s,t)\mathrm{d}s\mathrm{d}t.\end{split}$
(95)

Take an arbitrary ${\mathrm{g}}=(g_{t})_{t\in{\mathbb{R}}_{+}}$ in
${\mathscr{H}}_{\mathbbm{k}}$ and $t,\varepsilon\in{\mathbb{R}}_{+}$ such that
$t+\varepsilon\in{\mathbb{R}}_{+}$. Since $\mathbbm{k}$ is symmetric and
continuous, and due to the Cauchy-Schwartz inequality and the reproducing
property, we have

$\begin{split}\lim_{\varepsilon\to 0}&|g_{t+\varepsilon}-g_{t}|\\\
&\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!=\lim_{\varepsilon\to
0}|{\langle{\mathbbm{k}_{t+\varepsilon}-\mathbbm{k}_{t},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}|\\\
&\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\leq\lim_{\varepsilon\to
0}\|\mathbbm{k}_{t+\varepsilon}-\mathbbm{k}_{t}\|_{{\mathscr{H}}_{\mathbbm{k}}}\|{\mathrm{g}}\|_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!=\lim_{\varepsilon\to
0}\big{[}\mathbbm{k}(t+\varepsilon,t+\varepsilon)-2\mathbbm{k}(t,t+\varepsilon)+\mathbbm{k}(t,t)\big{]}^{\frac{1}{2}}\|{\mathrm{g}}\|_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!=0,\end{split}$

which implies the continuity of ${\mathrm{g}}=(g_{t})_{t\in{\mathbb{R}}_{+}}$
as a function of $t$. Therefore, the Riemann integral of ${\mathrm{g}}$ on
$[0,r]$ is well-defined. Accordingly, from the definition of
${\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}}$, the reproducing property and
$\lim_{n\to\infty}{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}}={\mathrm{f}}^{{\scalebox{0.63}{($r$)}}}$,
it follows that

$\begin{split}\int_{0}^{r}\\!g_{t}\mathrm{d}t&=\lim_{n\to\infty}\frac{r}{n}\sum_{i=0}^{n-1}g(\frac{i}{n}r)\\\
&=\lim_{n\to\infty}{\langle{{\mathrm{f}}_{n}^{{\scalebox{0.63}{($r$)}}},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&={\langle{{\mathrm{f}}^{{\scalebox{0.63}{($r$)}}},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&=\Big{\langle}\int_{0}^{r}\\!\mathbbm{k}(\cdot,t)\mathrm{d}t,{\mathrm{g}}\Big{\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}.\end{split}$

With respect to each $n\in{\mathbb{N}}$, let $r_{n}$ be

$r_{n}:=\min\Big{\\{}r\geq
n\Big{|}\int_{r}^{\infty}\\!\int_{r}^{\infty}\\!|\mathbbm{k}(s,t)|\mathrm{d}s\mathrm{d}t\leq\frac{1}{n}\Big{\\}},$

which is well-defined since $\mathbbm{k}$ is continuous and integrable.
Moreover, one can see that $\\{r_{n}\\}_{n=1}^{\infty}$ is an unbounded
strictly increasing sequence. For any $n,m\in{\mathbb{N}}$, such that $n\leq
m$, we have

$\begin{split}f_{t}^{{\scalebox{0.63}{($r_{n}$)}}}-f_{t}^{{\scalebox{0.63}{($r_{m}$)}}}&=\int_{0}^{r_{n}}\\!\mathbbm{k}(s,t)\mathrm{d}s-\int_{0}^{r_{m}}\\!\mathbbm{k}(s,t)\mathrm{d}s\\\
&=\int_{r_{m}}^{r_{n}}\\!\mathbbm{k}(s,t)\mathrm{d}s.\end{split}$ (96)

Accordingly, one can see that

$\begin{split}\|{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}}-&{\mathrm{f}}^{{\scalebox{0.63}{($r_{m}$)}}}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}={\langle{{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}}-{\mathrm{f}}^{{\scalebox{0.63}{($r_{m}$)}}},{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}}-{\mathrm{f}}^{{\scalebox{0.63}{($r_{m}$)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&={\langle{{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}},{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}}-{\mathrm{f}}^{{\scalebox{0.63}{($r_{m}$)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\!-\\!{\langle{{\mathrm{f}}^{{\scalebox{0.63}{($r_{m}$)}}},{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}}-{\mathrm{f}}^{{\scalebox{0.63}{($r_{m}$)}}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&=\int_{0}^{r_{n}}\\!\\!f_{t}^{{\scalebox{0.63}{($r_{n}$)}}}-f_{t}^{{\scalebox{0.63}{($r_{m}$)}}}\mathrm{d}t-\int_{0}^{r_{m}}\\!\\!f_{t}^{{\scalebox{0.63}{($r_{n}$)}}}-f_{t}^{{\scalebox{0.63}{($r_{m}$)}}}\mathrm{d}t\\\
&=\int_{r_{m}}^{r_{n}}\\!\int_{r_{m}}^{r_{n}}\\!\mathbbm{k}(s,t)\mathrm{d}s\mathrm{d}t\\\
&\leq\int_{r_{m}}^{r_{n}}\\!\int_{r_{m}}^{r_{n}}\\!|\mathbbm{k}(s,t)|\mathrm{d}s\mathrm{d}t\\\
&\leq\frac{1}{m}.\end{split}$

This implies that
$\\{{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}}\\}_{n=1}^{\infty}$ is a Cauchy
sequence in ${\mathscr{H}}_{\mathbbm{k}}$. Therefore, we know that there
exists ${\mathrm{f}}=(f_{s})_{s\in{\mathbb{R}}_{+}}$ in
${\mathscr{H}}_{\mathbbm{k}}$ such that
$\lim_{n\to\infty}\|{\mathrm{f}}-{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}}\|_{{\mathscr{H}}_{\mathbbm{k}}}=0$.
Subsequently, due to the reproducing property, for any $s\in{\mathbb{R}}_{+}$,
we have

$\begin{split}f_{s}={\langle{{\mathrm{f}},\mathbbm{k}_{s}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}&=\lim_{n\to\infty}{\langle{{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}},\mathbbm{k}_{s}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&=\lim_{n\to\infty}\int_{0}^{r_{n}}\\!\mathbbm{k}(s,t)\mathrm{d}t\\\
&=\int_{0}^{\infty}\\!\mathbbm{k}(s,t)\mathrm{d}t,\end{split}$

where the last equality holds due to
$\mathbbm{k}_{s}\in{\mathscr{H}}_{\mathbbm{k}}$,
${\mathscr{H}}_{\mathbbm{k}}\subset{\mathscr{L}}^{1}$, and the dominated
convergence theorem. Note that ${\mathrm{f}}$ coincides with $\varphi_{0}$
defined by (29). Accordingly, due to
$\lim_{n\to\infty}{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}}={\mathrm{f}}$,
(95), the definition of ${\mathrm{f}}$, and the dominated convergence theorem,
we have

$\begin{split}\Big{\|}\int_{0}^{\infty}\mathbbm{k}(\cdot,t)\mathrm{d}t\Big{\|}_{{\mathscr{H}}_{\mathbbm{k}}}^{2}&=\|{\mathrm{f}}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}\\\
&=\lim_{n\to\infty}\|{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}\\\
&=\lim_{n\to\infty}\int_{0}^{r_{n}}\\!\int_{0}^{r_{n}}\\!\mathbbm{k}(s,t)\mathrm{d}s\mathrm{d}t\\\
&=\int_{0}^{\infty}\\!\int_{0}^{\infty}\\!\mathbbm{k}(s,t)\mathrm{d}s\mathrm{d}t.\end{split}$

Based on similar arguments, for any
${\mathrm{g}}=(g_{t})_{t\in{\mathbb{R}}_{+}}\in{\mathscr{H}}_{\mathbbm{k}}$,
one can see that

$\begin{split}\Big{\langle}\int_{0}^{\infty}\\!\mathbbm{k}(\cdot,t)\mathrm{d}t,{\mathrm{g}}\Big{\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}&={\langle{{\mathrm{f}},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&=\lim_{n\to\infty}{\langle{{\mathrm{f}}^{{\scalebox{0.63}{($r_{n}$)}}},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&=\lim_{n\to\infty}\Big{\langle}\int_{0}^{r_{n}}\mathbbm{k}(\cdot,t)\mathrm{d}t,{\mathrm{g}}\Big{\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}\\\
&=\lim_{n\to\infty}\int_{0}^{r_{n}}\\!\\!g_{t}\mathrm{d}t=\int_{0}^{\infty}\\!\\!g_{t}\mathrm{d}t,\end{split}$

where the last equality is due to the dominated convergence theorem and the
fact that ${\mathrm{g}}$ is integrable. Therefore, we have
$\ell_{0}({\mathrm{g}})={\langle{{\mathrm{f}},{\mathrm{g}}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}$,
which implies (30). This concludes the proof. ∎

###  A-C Proof of Lemma 6

The first part of theorem can be obtained directly from the Riesz
representation theorem [61]. Due to the reproducing property of kernel, for
each $\tau\in{\mathscr{T}}$, we have

$\varphi_{\tau,t}^{\text{\rm{(u)}}}={\langle{\varphi_{\tau}^{\text{\rm{(u)}}},\mathbbm{k}_{t}}\rangle}_{{\mathscr{H}}_{\mathbbm{k}}}={\mathrm{L}}^{\\!{\mathrm{u}}}_{\tau}(\mathbbm{k}_{t}),\quad\forall
t\in{\mathbb{T}}.$ (97)

Subsequently, one can see (35) holds due to the definition of operator
${\mathrm{L}}^{\\!{\mathrm{u}}}_{\tau}$. This concludes the proof. ∎

###  A-D Steady-State Gain Representer for Discrete-Time Standard Stable
Kernels

######  Theorem 17.

_i)_ For kernel $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$, we
have

$\varphi_{0,t}=\left(t+\frac{1}{1-\alpha}\right)\alpha^{t},\quad\forall
t\in{\mathbb{Z}}_{+}.$ (98)

_ii)_ For kernel $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$, we
have

$\varphi_{0,t}=\Big{[}\frac{\gamma-\gamma^{t+1}}{1-\gamma}+\frac{1}{1-\alpha\gamma}\Big{]}\alpha^{t},\quad\forall
t\in{\mathbb{Z}}_{+}.$ (99)

_iii)_ For kernel $\mathbbm{k}_{\SS}$, we have

$\varphi_{0,t}=\Big{[}\frac{1+\alpha-\alpha^{t+1}}{1-\alpha^{2}}-\frac{\alpha^{t}}{3(1-\alpha^{3})}-\frac{t\alpha^{t}}{3}\Big{]}\alpha^{2t}\\!,\quad\forall
t\in{\mathbb{Z}}_{+}.$ (100)

###### Proof.

i) From (22), we have

$\begin{split}\\!\\!\\!\\!\\!\varphi_{0,t}&=\sum_{s=0}^{t-1}\alpha^{\max(s,t)}+\sum_{s\geq
t}\alpha^{\max(s,t)}\\\ &=t\alpha^{t}+\frac{\alpha^{t}}{1-\alpha}\\\
&=\left(t+\frac{1}{1-\alpha}\right)\alpha^{t}.\end{split}$

ii) Due to (22), one has

$\begin{split}\\!\\!\\!\\!\\!\varphi_{0,t}&=\sum_{s=0}^{t-1}\alpha^{\max(s,t)}\gamma^{|s-t|}+\sum_{s\geq
t}\alpha^{\max(s,t)}\gamma^{|s-t|}\\\
&=\alpha^{t}\sum_{s=0}^{t-1}\gamma^{-s+t}+\alpha^{t}\sum_{s\geq
t}\alpha^{s-t}\gamma^{s-t}\\\
&=\alpha^{t}\gamma\frac{1-\gamma^{t}}{1-\gamma}+\frac{\alpha^{t}}{1-\alpha\gamma}\\\
&=\Big{[}\frac{\gamma-\gamma^{t+1}}{1-\gamma}+\frac{1}{1-\alpha\gamma}\Big{]}\alpha^{t}.\end{split}$

iii) We know that

$\begin{split}\\!\\!\\!\\!\\!\sum_{s=0}^{t-1}&\alpha^{\max(s,t)}\alpha^{s+t}+\sum_{s\geq
t}\alpha^{\max(s,t)}\alpha^{s+t}\\\
&=\alpha^{2t}\sum_{s=0}^{t-1}\alpha^{s}+\alpha^{t}\sum_{s\geq t}\alpha^{2s}\\\
&=\alpha^{2t}\frac{1-\alpha^{t}}{1-\alpha}+\frac{\alpha^{3t}}{1-\alpha^{2}}\\\
&=\Big{[}\frac{1+\alpha-\alpha^{t+1}}{1-\alpha^{2}}\Big{]}\alpha^{2t}.\end{split}$

Following this, from (22) and the value of $\varphi_{0,t}$ for
$\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$, we have

$\begin{split}\varphi_{0,t}&=\Big{[}\frac{1+\alpha-\alpha^{t+1}}{1-\alpha^{2}}\Big{]}\alpha^{2t}-\frac{1}{3}(t+\frac{1}{1-\alpha^{3}})\alpha^{3t}\\\
&=\Big{[}\frac{1+\alpha-\alpha^{t+1}}{1-\alpha^{2}}-\frac{\alpha^{t}}{3(1-\alpha^{3})}-\frac{1}{3}t\alpha^{t}\Big{]}\alpha^{2t}.\end{split}$
(101)

This concludes the proof. ∎

######  Theorem 18.

_i)_ For kernel $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}$, we
have

$\|\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}=\frac{\alpha+1}{(1-\alpha)^{2}}.$
(102)

_ii)_ For kernel $\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$, we
have

$\|\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}=\frac{1+\alpha\gamma}{(1-\alpha)(1-\alpha\gamma)}.$
(103)

_iii)_ For kernel $\mathbbm{k}_{\SS}$, we have

$\|\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}=\frac{2}{3}\frac{\alpha^{4}+\alpha^{3}+3\alpha^{2}+\alpha+1}{(\alpha^{3}-1)^{2}(\alpha+1)}.$
(104)

###### Proof.

i) We know that
$\|\varphi_{0}\|^{2}=\sum_{t\in{\mathbb{Z}}_{+}}\varphi_{0,t}$. Hence,

$\begin{split}\\!\\!\\!\\!\\!\|\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}&=\sum_{t\in{\mathbb{Z}}_{+}}t\alpha^{t}+\frac{1}{1-\alpha}\sum_{t\in{\mathbb{Z}}_{+}}\alpha^{t}\\\
&=\frac{\alpha}{(1-\alpha)^{2}}+\frac{1}{(1-\alpha)^{2}}=\frac{1+\alpha}{(1-\alpha)^{2}}.\end{split}$

ii) Similarly to the proof of part i), for
$\mathbbm{k}=\mathbbm{k}_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}$, we have

$\begin{split}\\!\\!\\!\\!\\!\|\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}&=\Big{[}\frac{\gamma}{1-\gamma}+\frac{1}{1-\alpha\gamma}\Big{]}\sum_{t\in{\mathbb{Z}}_{+}}\alpha^{t}-\frac{\gamma}{(1-\gamma)}\sum_{t\in{\mathbb{Z}}_{+}}(\alpha\gamma)^{t}\\\
&=\Big{[}\frac{\gamma}{1-\gamma}+\frac{1}{1-\alpha\gamma}\Big{]}\frac{1}{1-\alpha}-\frac{\gamma}{(1-\gamma)}\frac{1}{(1-\alpha\gamma)}\\\
&=\frac{1-\alpha\gamma^{2}-\gamma+\alpha\gamma}{(1-\alpha)(1-\gamma)(1-\alpha\gamma)}=\frac{1+\alpha\gamma}{(1-\alpha)(1-\alpha\gamma)}.\end{split}$

iii) Similar to the previous parts, one has

$\begin{split}\|&\varphi_{0}\|_{{\mathscr{H}}_{\mathbbm{k}}}^{2}\\\
&=\\!\sum_{t\in{\mathbb{Z}}_{+}}\\!\frac{\alpha^{2t}}{1-\alpha}-\Big{[}\frac{\alpha}{1-\alpha^{2}}+\frac{1}{3(1-\alpha^{3})}\Big{]}\\!\sum_{t\in{\mathbb{Z}}_{+}}\\!\alpha^{3t}-\\!\sum_{t\in{\mathbb{Z}}_{+}}\\!\frac{t\alpha^{3t}}{3}\\\
&=\frac{1}{(1-\alpha)(1-\alpha^{2})}-\frac{3\alpha-3\alpha^{4}+1-\alpha^{2}}{3(1-\alpha^{2})(1-\alpha^{3})^{2}}-\frac{\alpha^{3}}{3(1-\alpha^{3})^{2}}\\\
&=\frac{2}{3}\frac{\alpha^{4}+\alpha^{3}+3\alpha^{2}+\alpha+1}{(\alpha^{3}-1)^{2}(\alpha+1)}.\end{split}$

This concludes the proof. ∎

###  A-E Proof of Theorem 12

i) For $t\leq a$, we have

$\psi_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}(t,a,b)=\int_{a}^{b}\alpha^{s}\mathrm{d}s=\frac{\alpha^{b}-\alpha^{a}}{\ln(\alpha)}.$
(105)

Also, for $t\geq b$, one has

$\psi_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}(t,a,b)=\int_{a}^{b}\alpha^{t}\mathrm{d}s=(b-a)\alpha^{t}.$
(106)

Similar to the previous cases, when $t\in(a,b)$, we have

$\begin{split}\psi_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}(t,a,b)&=\int_{a}^{t}\alpha^{t}\mathrm{d}s+\int_{t}^{b}\alpha^{s}\mathrm{d}s\\\
&=(t-a)\alpha^{t}+\frac{\alpha^{b}-\alpha^{t}}{\ln(\alpha)}.\end{split}$ (107)

Due to (105), (106), (107), and the definition of $\eta$, one can see (64)
holds.
ii) For $t\leq a$, one can see that

$\begin{split}\psi_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}(t,a,b)&=\int_{a}^{b}\alpha^{s}\gamma^{s-t}\mathrm{d}s\\\
&=\gamma^{-t}\int_{a}^{b}(\alpha\gamma)^{s}\mathrm{d}s\\\
&=\frac{(\alpha\gamma)^{b}-(\alpha\gamma)^{a}}{\ln(\alpha\gamma)}\gamma^{-t}.\end{split}$
(108)

Also, for $t\geq b$, we have

$\begin{split}\psi_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}(t,a,b)&=\int_{a}^{b}\alpha^{t}\gamma^{t-s}\mathrm{d}s\\\
&=(\alpha\gamma)^{t}\int_{a}^{b}\gamma^{-s}\mathrm{d}s\\\
&=\frac{\gamma^{-a}-\gamma^{-b}}{\ln(\gamma)}(\alpha\gamma)^{t}.\end{split}$
(109)

Similarly, if $t\in(a,b)$, one has

$\begin{split}\psi_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}(t,a,b)&=\int_{a}^{t}\alpha^{t}\gamma^{t-s}\mathrm{d}s+\int_{t}^{b}\alpha^{s}\gamma^{s-t}\mathrm{d}s\\\
&=\frac{\gamma^{-a}-\gamma^{-t}}{\ln(\gamma)}(\alpha\gamma)^{t}+\frac{(\alpha\gamma)^{b}-(\alpha\gamma)^{t}}{\ln(\alpha\gamma)}\gamma^{-t}.\end{split}\\!\\!\\!\\!$
(110)

From the definition of $\eta$, one can see that (108), (109) and (110) implies
(64).
iii) For $t\leq a$, we have

$\begin{split}\int_{a}^{b}\alpha^{\max(s,t)+s+t}\mathrm{d}s&=\int_{a}^{b}\alpha^{2s+t}\mathrm{d}s\\\
&=\alpha^{t}\int_{a}^{b}\alpha^{2s}\mathrm{d}s\\\
&=\frac{\alpha^{2b}-\alpha^{2a}}{2\ln(\alpha)}\alpha^{t}.\end{split}$ (111)

Also, if $t\geq b$, one can see that

$\begin{split}\int_{a}^{b}&\alpha^{\max(s,t)+s+t}\mathrm{d}s=\int_{a}^{b}\alpha^{2t+s}\mathrm{d}s=\frac{\alpha^{b}-\alpha^{a}}{\ln(\alpha)}\alpha^{2t}.\end{split}$
(112)

Similarly, if $t\in(a,b)$, one has

$\begin{split}\int_{a}^{b}\alpha^{\max(s,t)+s+t}\mathrm{d}s&=\int_{a}^{t}\alpha^{2t+s}\mathrm{d}s+\int_{t}^{b}\alpha^{2s+t}\mathrm{d}s\\\
&=\frac{\alpha^{t}-\alpha^{a}}{\ln(\alpha)}\alpha^{2t}+\frac{\alpha^{2b}-\alpha^{2t}}{2\ln(\alpha)}\alpha^{t}.\end{split}$
(113)

Based on the definition of $\eta$, (111), (112), and (113), one can see that

$\begin{split}\int_{a}^{b}&\alpha^{\max(s,t)+s+t}\mathrm{d}s\\\
&=\frac{\alpha^{\eta(t,a,b)}-\alpha^{a}}{\ln(\alpha)}\alpha^{2t}+\frac{\alpha^{2b}-\alpha^{2\eta(t,a,b)}}{2\ln(\alpha)}\alpha^{t}.\end{split}$
(114)

Accordingly, by replacing $\alpha$ with $\alpha^{3}$ is (64) and due to the
definition of $\mathbbm{k}_{\SS}$, one can obtain (66). ∎

###  A-F Proof of Theorem 15

Since $\bar{\nu}(x)=\lim_{y\to\infty}\nu(x,y)$, it is enough to obtain
$\nu(x,y)$. Throughout the proof, without loss of generality we assume $x\leq
y$.
i) Due to the definition of $\nu$ in (70), we know that

$\begin{split}\nu_{\text{{\scalebox{0.75}{$\mathrm{TC}$}}}}(x,y)&=\int_{0}^{x}\int_{0}^{s}\alpha^{s}\mathrm{d}t\mathrm{d}s+\int_{0}^{x}\int_{s}^{y}\alpha^{t}\mathrm{d}t\mathrm{d}s\\\
&=\int_{0}^{x}s\alpha^{s}\mathrm{d}s+\int_{0}^{x}\frac{\alpha^{y}-\alpha^{s}}{\ln(\alpha)}\mathrm{d}s\\\
&=\frac{x\alpha^{x}\ln(\alpha)+1-\alpha^{x}}{\ln(\alpha)^{2}}+\frac{x\alpha^{y}}{\ln(\alpha)}-\frac{\alpha^{x}-1}{\ln(\alpha)^{2}}.\end{split}$

Reordering the terms and replacing $x$ and $y$ respectively with $\min(x,y)$
and $\max(x,y)$, we obtain (80). Letting $y$ go to $\infty$ in (80), we obtain
(81).
ii) From the definition of $\nu$, we have

$\begin{split}\nu_{\text{{\scalebox{0.75}{$\mathrm{DC}$}}}}&(x,y)=\int_{0}^{x}\int_{0}^{s}\alpha^{s}\gamma^{s-t}\mathrm{d}t\mathrm{d}s+\int_{0}^{x}\int_{s}^{y}\alpha^{t}\gamma^{t-s}\mathrm{d}t\mathrm{d}s\\\
&=\int_{0}^{x}\alpha^{s}\gamma^{s}\int_{0}^{s}\gamma^{-t}\mathrm{d}t\mathrm{d}s+\int_{0}^{x}\gamma^{-s}\int_{s}^{y}\alpha^{t}\gamma^{t}\mathrm{d}t\mathrm{d}s\\\
&=\int_{0}^{x}(\alpha\gamma)^{s}\frac{1-\gamma^{-s}}{\ln(\gamma)}\mathrm{d}s+\int_{0}^{x}\gamma^{-s}\frac{(\alpha\gamma)^{y}-(\alpha\gamma)^{s}}{\ln(\alpha\gamma)}\mathrm{d}s\\\
&=\frac{(\alpha\gamma)^{x}-1}{\ln(\gamma)\ln(\alpha\gamma)}-\frac{\alpha^{x}-1}{\ln(\gamma)\ln(\alpha)}-\frac{(\alpha\gamma)^{y}(\gamma^{-x}-1)}{\ln(\gamma)\ln(\alpha\gamma)}\\\
&\qquad-\frac{\alpha^{x}-1}{\ln(\alpha)\ln(\alpha\gamma)}.\end{split}$

Similar to the previous part, by replacing $x$ and $y$ respectively with
$\min(x,y)$ and $\max(x,y)$, and reordering the terms, we obtain (82).
Moreover, one can see that (83) results when $y$ goes to $\infty$.
iii) Replacing $\alpha$ with $\alpha^{3}$ in (80), we obtain

$\begin{split}\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\int_{0}^{x}&\\!\\!\\!\\!\int_{0}^{y}\\!\\!\alpha^{3\max(s,t)}\mathrm{d}t\mathrm{d}s\\\
&=\frac{\min(x,y)(\alpha^{3x}+\alpha^{3y})}{3\ln(\alpha)}+\frac{2(1-\alpha^{3\min(x,y)})}{9\ln(\alpha)^{2}}.\end{split}$
(115)

On the other hand, we have

$\begin{split}\int_{0}^{x}\int_{0}^{y}&\alpha^{\max(s,t)+s+t}\mathrm{d}t\mathrm{d}s\\\
&=\int_{0}^{x}\int_{0}^{s}\alpha^{2s+t}\mathrm{d}t\mathrm{d}s+\int_{0}^{x}\int_{s}^{y}\alpha^{s+2t}\mathrm{d}t\mathrm{d}s\\\
&=\int_{0}^{x}\alpha^{2s}\frac{\alpha^{s}-1}{\ln(\alpha)}+\alpha^{s}\frac{\alpha^{2y}-\alpha^{2s}}{2\ln(\alpha)}\mathrm{d}s\\\
&=\frac{\alpha^{3x}-1}{6\ln(\alpha)^{2}}-\frac{\alpha^{2x}-1}{2\ln(\alpha)^{2}}+\frac{\alpha^{2y}(\alpha^{x}-1)}{2\ln(\alpha)^{2}}.\end{split}$
(116)

Using the definition of $\nu$, (115), (116), replacing $x$ and $y$
respectively with $\min(x,y)$ and $\max(x,y)$, and reordering the terms, we
obtain (84). Also, letting $y\to\infty$, we get (85). ∎

## References

  * [1] L. Zadeh, “On the identification problem,” _IRE Transactions on Circuit Theory_ , vol. 3, no. 4, pp. 277–281, 1956.
  * [2] L. Ljung, _System identification: Theory for the user_. Prentice Hall, 1999.
  * [3] T. A. Johansen, “Constrained and regularized system identification,” _Modeling, Identification and Control_ , vol. 19, no. 2, pp. 109–116, 1998\.
  * [4] P. Trnka and V. Havlena, “Subspace like identification incorporating prior information,” _Automatica_ , vol. 45, no. 4, pp. 1086–1091, 2009.
  * [5] J. Umenberger and I. R. Manchester, “Specialized interior-point algorithm for stable nonlinear system identification,” _IEEE Transactions on Automatic Control_ , vol. 64, no. 6, pp. 2442–2456, 2018.
  * [6] M. Khosravi and R. S. Smith, “Nonlinear system identification with prior knowledge on the region of attraction,” _IEEE Control Systems Letters_ , vol. 5, no. 3, pp. 1091–1096, 2021.
  * [7] A. A. Ahmadi and B. El Khadir, “Learning dynamical systems with side information (short version),” _Proceedings of Machine Learning Research_ , vol. 120, pp. 718–727, 2020.
  * [8] M. Khosravi and R. S. Smith, “Convex nonparametric formulation for identification of gradient flows,” _IEEE Control Systems Letters_ , vol. 5, no. 3, pp. 1097–1102, 2021.
  * [9] C. Lyzell, M. Enqvist, and L. Ljung, “Handling certain structure information in subspace identification,” _IFAC Proceedings Volumes_ , vol. 42, no. 10, pp. 90–95, 2009.
  * [10] P. Massioni and M. Verhaegen, “Subspace identification of circulant systems,” _Automatica_ , vol. 44, no. 11, pp. 2825–2833, 2008.
  * [11] D. N. Miller and R. A. De Callafon, “Subspace identification with eigenvalue constraints,” _Automatica_ , vol. 49, no. 8, pp. 2468–2473, 2013.
  * [12] T. McKelvey and S. R. Moheimani, “Estimation of phase constrained mimo transfer functions with application to flexible structures with mixed collocated and non-collocated actuators and sensors,” _IFAC Proceedings Volumes_ , vol. 38, no. 1, pp. 219–224, 2005.
  * [13] J. B. Hoagg, S. L. Lacy, R. S. Erwin, and D. S. Bernstein, “Subspace identification with lower bounded modal frequencies,” in _Proceedings of the 2004 American control conference_ , vol. 1. IEEE, 2004, pp. 867–872.
  * [14] M. Inoue, “Subspace identification with moment matching,” _Automatica_ , vol. 99, pp. 22–32, 2019.
  * [15] I. Goethals, T. Van Gestel, J. Suykens, P. Van Dooren, and B. De Moor, “Identification of positive real models in subspace identification by using regularization,” _IEEE Transactions on Automatic Control_ , vol. 48, no. 10, pp. 1843–1847, 2003.
  * [16] L. F. Rodrigues, L. P. Ihlenfeld, and G. H. da Costa Oliveira, “A novel subspace identification approach with passivity enforcement,” _Automatica_ , vol. 132, p. 109798, 2021.
  * [17] A. De Santis and L. Farina, “Identification of positive linear systems with Poisson output transformation,” _Automatica_ , vol. 38, no. 5, pp. 861–868, 2002.
  * [18] L. Benvenuti, A. De Santis, and L. Farina, “On model consistency in compartmental systems identification,” _Automatica_ , vol. 38, no. 11, pp. 1969–1976, 2002.
  * [19] R. S. Smith, “Frequency domain subspace identification using nuclear norm minimization and Hankel matrix realizations,” _IEEE Transactions on Automatic Control_ , vol. 59, no. 11, pp. 2886–2896, 2014.
  * [20] M. Fazel, T. K. Pong, D. Sun, and P. Tseng, “Hankel matrix rank minimization with applications to system identification and realization,” _SIAM Journal on Matrix Analysis and Applications_ , vol. 34, no. 3, pp. 946–977, 2013\.
  * [21] P. Shah, B. N. Bhaskar, G. Tang, and B. Recht, “Linear system identification via atomic norm regularization,” in _Conference on Decision and Control_ , 2012, pp. 6265–6270.
  * [22] S. L. Lacy and D. S. Bernstein, “Subspace identification with guaranteed stability using constrained optimization,” _IEEE Transactions on automatic control_ , vol. 48, no. 7, pp. 1259–1263, 2003.
  * [23] T. Van Gestel, J. A. Suykens, P. Van Dooren, and B. De Moor, “Identification of stable models in subspace identification by using regularization,” _IEEE Transactions on Automatic control_ , vol. 46, no. 9, pp. 1416–1420, 2001.
  * [24] G. Pillonetto and G. De Nicolao, “A new kernel-based approach for linear system identification,” _Automatica_ , vol. 46, no. 1, pp. 81–93, 2010.
  * [25] L. Ljung, T. Chen, and B. Mu, “A shift in paradigm for system identification,” _International Journal of Control_ , vol. 93, no. 2, pp. 173–180, 2020.
  * [26] G. Pillonetto, F. Dinuzzo, T. Chen, G. De Nicolao, and L. Ljung, “Kernel methods in system identification, machine learning and function estimation: A survey,” _Automatica_ , vol. 50, no. 3, pp. 657–682, 2014.
  * [27] M. Khosravi and R. S. Smith, “On robustness of kernel-based regularized system identification,” _IFAC-PapersOnLine_ , vol. 54, no. 7, pp. 749–754, 2021, IFAC Symposium on System Identification.
  * [28] Y. Fujimoto, I. Maruta, and T. Sugie, “Extension of first-order stable spline kernel to encode relative degree,” _IFAC-PapersOnLine_ , vol. 50, no. 1, pp. 14 016–14 021, 2017.
  * [29] T. Chen, H. Ohlsson, and L. Ljung, “On the estimation of transfer functions, regularizations and Gaussian processe – Revisited,” _Automatica_ , vol. 48, no. 8, pp. 1525–1535, 2012.
  * [30] M. A. H. Darwish, G. Pillonetto, and R. Tóth, “The quest for the right kernel in Bayesian impulse response identification: The use of OBFs,” _Automatica_ , vol. 87, pp. 318–329, 2018.
  * [31] T. Chen, “On kernel design for regularized LTI system identification,” _Automatica_ , vol. 90, pp. 109–122, 2018.
  * [32] A. Marconato, M. Schoukens, and J. Schoukens, “Filter-based regularisation for impulse response modelling,” _IET Control Theory and Applications_ , vol. 11, no. 2, pp. 194–204, 2016.
  * [33] R. S. Risuleo, G. Bottegal, and H. Hjalmarsson, “A nonparametric kernel-based approach to Hammerstein system identification,” _Automatica_ , vol. 85, pp. 234–247, 2017.
  * [34] R. S. Risuleo, F. Lindsten, and H. Hjalmarsson, “Bayesian nonparametric identification of Wiener systems,” _Automatica_ , vol. 108, p. 108480, 2019\.
  * [35] N. Everitt, G. Bottegal, and H. Hjalmarsson, “An empirical Bayes approach to identification of modules in dynamic networks,” _Automatica_ , vol. 91, pp. 144–151, 2018.
  * [36] M. Khosravi and R. S. Smith, “Kernel-based identification with frequency domain side-information,” _arXiv preprint_ , 2021.
  * [37] M. Khosravi, M. Yin, A. Iannelli, A. Parsi, and R. S. Smith, “Low-complexity identification by sparse hyperparameter estimation,” _IFAC-PapersOnLine_ , vol. 53, no. 2, pp. 412–417, 2020, IFAC World Congress 2020.
  * [38] M. Khosravi, A. Iannelli, M. Yin, A. Parsi, and R. S. Smith, “Regularized system identification: A hierarchical Bayesian approach,” _IFAC-PapersOnLine_ , vol. 53, no. 2, pp. 406–411, 2020, IFAC World Congress 2020.
  * [39] G. Pillonetto, T. Chen, A. Chiuso, G. D. Nicolao, and L. Ljung, “Regularized linear system identification using atomic, nuclear and kernel-based norms: The role of the stability constraint,” _Automatica_ , vol. 69, pp. 137–149, 2016.
  * [40] M. Khosravi and R. S. Smith, “Kernel-based identification of positive systems,” in _Conference on Decision and Control_ , 2019, pp. 1740–1745.
  * [41] M. Zheng and Y. Ohta, “Bayesian positive system identification: Truncated Gaussian prior and hyperparameter estimation,” _Systems and Control Letters_ , vol. 148, p. 104857, 2021.
  * [42] M. Khosravi and R. S. Smith, “Regularized identification with internal positivity side-information,” _arXiv preprint_ , 2021.
  * [43] P. Grosdidier, M. Morari, and B. R. Holt, “Closed-loop properties from steady-state gain information,” _Industrial and engineering chemistry fundamentals_ , vol. 24, no. 2, pp. 221–235, 1985.
  * [44] P. J. Campo and M. Morari, “Achievable closed-loop properties of systems under decentralized control: Conditions involving the steady-state gain,” _IEEE Transactions on Automatic Control_ , vol. 39, no. 5, pp. 932–943, 1994\.
  * [45] Z. Q. Zheng and M. Morari, “Robust stability of constrained model predictive control,” in _1993 American Control Conference_. IEEE, 1993, pp. 379–383.
  * [46] A. Alenany, H. Shang, M. Soliman, and I. Ziedan, “Improved subspace identification with prior information using constrained least squares,” _IET Control Theory and Applications_ , vol. 5, no. 13, pp. 1568–1576, 2011\.
  * [47] S. Yoshimura, A. Matsubayashi, and M. Inoue, “System identification method inheriting steady-state characteristics of existing model,” _International Journal of Control_ , vol. 92, no. 11, pp. 2701–2711, 2019\.
  * [48] Y. Abe, M. Inoue, and S. Adachi, “Subspace identification method incorporated with a priori information characterized in frequency domain,” in _European Control Conference_. IEEE, 2016, pp. 1377–1382.
  * [49] I. Markovsky and G. Mercère, “Subspace identification with constraints on the impulse response,” _International Journal of Control_ , vol. 90, no. 8, pp. 1728–1735, 2017.
  * [50] Y. Fujimoto and T. Sugie, “Kernel-based impulse response estimation with a priori knowledge on the DC gain,” _IEEE control systems letters_ , vol. 2, no. 4, pp. 713–718, 2018.
  * [51] A. H. Tan and D. S. Ong, “Kernel-based impulse response estimation with prior DC gain using built-in self-scaling technique,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 69, no. 9, pp. 7295–7305, 2020.
  * [52] T. Chen and G. Pillonetto, “On the stability of reproducing kernel Hilbert spaces of discrete-time impulse responses,” _Automatica_ , vol. 95, pp. 529–533, 2018.
  * [53] H. Garnier, L. Wang, and P. C. Young, “Direct identification of continuous-time models from sampled data: Issues, basic solutions and relevance,” in _Identification of continuous-time models from sampled data_. Springer, 2008, pp. 1–29.
  * [54] H. Garnier and M. Gilson, “CONTSID: a MATLAB toolbox for standard and advanced identification of black-box continuous-time models,” _IFAC-PapersOnLine_ , vol. 51, no. 15, pp. 688–693, 2018.
  * [55] L. Ljung and R. Singh, “Version 8 of the MATLAB system identification toolbox,” _IFAC-PapersOnLine_ , vol. 45, no. 16, pp. 1826–1831, 2012.
  * [56] A. Berlinet and C. Thomas-Agnan, _Reproducing Kernel Hilbert Spaces in Probability and Statistics_. Springer Science and Business Media, 2011.
  * [57] C. Carmeli, E. De Vito, and A. Toigo, “Vector valued reproducing kernel Hilbert spaces of integrable functions and Mercer theorem,” _Analysis and Applications_ , vol. 4, no. 4, pp. 377–408, 2006.
  * [58] R. A. Maronna, R. D. Martin, and V. J. Yohai, _Robust Statistics: Theory and Methods_. John Wiley and Sons, 2006.
  * [59] F. Dinuzzo and B. Schölkopf, “The representer theorem for Hilbert spaces: A necessary and sufficient condition,” in _Advances in neural information processing systems_ , 2012, pp. 189–196.
  * [60] J. Peypouquet, _Convex Optimization in Normed Spaces: Theory, Methods and Examples_. Springer, 2015\.
  * [61] H. Brézis, “Functional Analysis, Sobolev Spaces and Partial Differential Equations,” 2011.
  * [62] N. Srinivas, A. Krause, S. M. Kakade, and M. W. Seeger, “Information-theoretic regret bounds for Gaussian process optimization in the bandit setting,” _IEEE Transactions on Information Theory_ , vol. 58, no. 5, pp. 3250–3265, 2012.