# Regret-Optimal Control under Partial Observability

Joudi Hajar, Oron Sabag, Babak Hassibi J. Hajar and B. Hassibi are with the
Department of Electrical Engineering at Caltech (emails:
{jhajar,hassibi}@caltech.edu).O. Sabag is with the School of Computer Science
and Engineering at the Hebrew University of Jerusalem (email:
oron.sabag@mail.huji.ac.il).The work of O. Sabag was conducted at Caltech
during 2020-2022. The work of J. Hajar was done at Caltech in 2023.

(November 2020; July 2021)

###### Abstract

This paper studies online solutions for regret-optimal control in partially
observable systems over an infinite-horizon. Regret-optimal control aims to
minimize the difference in LQR cost between causal and non-causal controllers
while considering the worst-case regret across all $\ell_{2}$-norm-bounded
disturbance and measurement sequences. Building on ideas from [1] on the the
full-information setting, our work extends the framework to the scenario of
partial observability (measurement-feedback). We derive an explicit state-
space solution when the non-causal solution is the one that minimizes the
$\mathcal{H}_{2}$ criterion, and demonstrate its practical utility on several
practical examples. These results underscore the framework’s significant
relevance and applicability in real-world systems.

##  I Introduction

Regret-optimal control stands as a robust framework for designing controllers
capable of dealing with uncertainty within dynamic systems through the lens of
competitive analysis. The main idea is to design a controller that minimizes
the disparity between its own performance and that of a reference controller
that has a superior performance. Typical competitive metrics are the regret
and the competitive ratio, defined as the difference or ratio between the
controller’s performance and the reference controller’s performance. One of
the fundamental questions studied in the current manuscript is how the
designed policy will be affected by the choice of the reference controller, an
impractical entity that is aimed to be mimicked.

The regret optimal control framework was introduced for the full-information
control setup where the designed controller has a strictly causal access to
the system’s state and the reference controller is the one that has non-causal
access to the states [1] (and its extended version [2]). Interestingly, the
optimal regret and, as a result, the regret optimal controller can be derived
explicitly in the infinite horizon scenario via a reduction to the Nehari
problem [3]. This is in contrast to typical solutions in robust control that
involve an additional optimization over the corresponding sub-optimal problem.
The study of this framework was then extended to the finite horizon setup with
and without safety constraints [4, 5, 6], and to the competitive ratio metric
[7, 8]. In all of those works (see also [9, 10, 11, 12, 13, 14]), regret has
been consistently shown to be a compelling metric for controllers’ design as
it nicely interpolates the robust and average performance of
$\mathcal{H}_{\infty}$ and $\mathcal{H}_{2}$ control. Motivated by this, we
consider the regret optimal control approach to partially observable control
systems.

For the full information control setup, a natural candidate for the reference
controller is a clairvoyant controller that has a non-causal access to the
entire disturbances sequence (equivalently, the state generated by the
uncontrolled system) before hand. This clairvoyant controller is unique and
outperforms any controller for any disturbance sequence [15]. For partially
observable systems, the clairvoyant controller has a non-causal access to the
system’s measurements generated by the uncontrolled dynamical system. However,
there does not exist a unique controller that outperforms any other controller
_for any noise sequence_. Therefore, the clairvoyant controller has to be
optimal under particular metric e.g. $\mathcal{H}_{2}$ or
$\mathcal{H}_{\infty}$. Our first result shows that the regret-optimal control
problem can be reduced to a Nehari problem no matter what is the reference
controller. This is a generalization of [16] for the $\mathcal{H}_{2}$
clairvoyant controller and it elucidates the structure of the regret-optimal
controller. Previously, it was shown that the regret optimal control problem
can be reduced to an $\mathcal{H}_{\infty}$ problem if the reference
controller has a direct access to the system disturbance [17]. In contrast,
our reference controller only assumes non-causal access to the measurements.
This implies that the operational comparison between the LQR costs is made
over both the disturbance and the measurement sequences. The reduction to the
Nehari problem also serves as the first step towards an explicit solution for
the state-space setting as the Nehari solution has an explicit solution [2].

For the state space setting, we focus on the non-causal $\mathcal{H}_{2}$
controller as our reference for the regret metric. Through spectral
factorizations and decompositions, we are able to derive an explicit regret-
optimal controller. The controller is represented by a finite-dimensional and
time invariant state-space whose inputs are the system measurements. Thus, it
can be implemented in real-time. This opens up new avenues for practical
application of regret-optimal control, with potential impacts in various
fields, including robotics, autonomous vehicles, and control systems design.

We summarize our main contribution:

  1. 1.

We study regret-optimal control with partial observability and present a
reduction to the Nehari problem for _any reference controller_. This is
presented in Theorem 1, and holds for any time horizon.

  2. 2.

For the state-space setting, we derive an explicit solution for regret-optimal
control under partial observability. This solution applies to the doubly
infinite horizon regime and the reference controller is the $\mathcal{H}_{2}$
non-causal controller that has access to the system measurements. To the best
of our knowledge, this is the first solution of partial observability in the
context of regret optimal control that involves a comparison of control costs
(of the designed and the reference controllers) that depend on _two sequences_
, the disturbances and the measurements noise.

To validate the effectiveness of our approach, we conduct simulations on a
range of systems, demonstrating the efficacy of our proposed method in various
settings. We organize the paper as follows: Section II provides the
preliminaries, Section III presents the reduction of the regret control
problem to a Nehari problem, Section IV offers the state space solution of the
controller, and Section V presents the simulations. All proofs are to be found
in the appendix of the extended version of this paper available online.

##  II Problem Definition and Preliminaries

###  II-A Notation

$\mathbb{R}$ denotes the set of real numbers, $\|\cdot\|_{2}$ is the 2-norm,
$\|\cdot\|$ is the operator norm, ∗ stands for adjoint, $z$ is the complex
variable of the $z$-domain, and $I$ is the identity matrix. A calligraphic
letter represents a doubly infinite operator, and its corresponding standard
letter denotes the transfer matrix of the operator.

###  II-B The control setting

We examine a discrete-time, linear time-invariant (LTI) dynamical system
described by the following state-space model:

$\displaystyle x_{i+1}$
$\displaystyle=Fx_{i}+G_{1}w_{i}+G_{2}u_{i}\operatorname*{}$ $\displaystyle
s_{i}$ $\displaystyle=Lx_{i}\operatorname*{}$ $\displaystyle y_{i}$
$\displaystyle=Hx_{i}+v_{i}.$ (1)

In this representation, $x_{i}\in\mathbb{R}^{n}$ denotes the system’s state,
$u_{i}\in\mathbb{R}^{m}$ represents the control input,
$w_{i}\in\mathbb{R}^{n}$ is the process noise, $s_{i}\in\mathbb{R}^{q}$ is the
regulated signal while $y_{i}\in\mathbb{R}^{p}$ represents the noisy state
measurements, and $v_{i}\in\mathbb{R}^{p}$ is the measurement noise.

Throughout this paper, we adopt an operator form representation of the system
dynamics (II-B). Considering a _doubly infinite horizon_ , let us define
$~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}s\coloneqq\left(\begin{array}[]{c}\vdots\\\
s_{-1}\\\ s_{0}\\\ s_{1}\\\
\vdots\end{array}\right)~{}~{}~{},~{}~{}~{}u\coloneqq\left(\begin{array}[]{c}\vdots\\\
u_{-1}\\\ u_{0}\\\ u_{1}\\\
\vdots\end{array}\right),~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$ and similarly
for $w$, $x$, $y$, and $v$. With these definitions, we can express the system
dynamics (II-B) equivalently using an operator-based formulation:

$\displaystyle s$
$\displaystyle=\mathcal{P}_{11}w+\mathcal{P}_{12}u\operatorname*{}$
$\displaystyle y$
$\displaystyle=\mathcal{P}_{21}w+\mathcal{P}_{22}u+v\operatorname*{}$
$\displaystyle u$ $\displaystyle=\mathcal{K}y,$ (2)

where the operators $\mathcal{P}_{ij}$ for $i,j\in\\{1,2\\}$ are strictly
causal time-invariant operators (i.e, strictly lower triangular block Toeplitz
operators) corresponding to the dynamics (II-B), and $\mathcal{K}$ is a linear
causal controller to be designed.

We consider the Linear-Quadratic Gaussian (LQG) cost given as

$J(u,w,v)\coloneqq s^{\ast}\bar{Q}s+u^{\ast}\bar{R}u$ (3)

where $\bar{Q},\bar{R}$ are block diagonal operators with $Q,R\succ 0$ on
their diagonals, respectively. To simplify the notation, we redefine $s$ and
$u$ as $s\leftarrow\bar{Q}^{\frac{1}{2}}s$, and
$u\leftarrow\bar{R}^{\frac{1}{2}}u$, which results in (3) taking the form:

$\displaystyle J(u,w,v)$
$\displaystyle=s^{\ast}s+u^{\ast}u=\begin{pmatrix}w^{\ast}&v^{\ast}\end{pmatrix}T_{\mathcal{K}}^{\ast}T_{\mathcal{K}}\begin{pmatrix}w\\\
v\end{pmatrix}.$ (4)

In this expression, $T_{\mathcal{K}}$ represents the transfer operator
associated with $\mathcal{K}$, mapping the input sequences
$\begin{pmatrix}w\\\ v\end{pmatrix}$ to the regulated state and control
sequences $\begin{pmatrix}s\\\ u\end{pmatrix}$, i.e.,

$T_{\mathcal{K}}:\begin{pmatrix}w\\\ v\end{pmatrix}\to\begin{pmatrix}s\\\
u\end{pmatrix}.$

The operator $T_{\mathcal{K}}$ can be computed explicitly as

$\displaystyle T_{\mathcal{K}}$
$\displaystyle=\begin{pmatrix}\mathcal{P}_{11}+\mathcal{P}_{12}\mathcal{K}(I-\mathcal{P}_{22}\mathcal{K})^{-1}\mathcal{P}_{21}&\mathcal{P}_{12}\mathcal{K}(I-\mathcal{P}_{22}\mathcal{K})^{-1}\\\
\mathcal{K}(I-\mathcal{P}_{22}\mathcal{K})^{-1}\mathcal{P}_{21}&\mathcal{K}(I-\mathcal{P}_{22}\mathcal{K})^{-1}\end{pmatrix}.$

As $\mathcal{P}_{22}$ and $\mathcal{P}_{22}\mathcal{K}$ are strictly causal,
we can define the Youla parameterization:

$\mathcal{Q}=\mathcal{K}(I-\mathcal{P}_{22}\mathcal{K})^{-1}$ (5)

or equivalently,

$\mathcal{K}=\mathcal{Q}(\mathcal{I}+\mathcal{Q}\mathcal{P}_{22})^{-1}$ (6)

which implies that $\mathcal{K}$ is causal (/strictly-causal) if and only if
$\mathcal{Q}$ is causal (/strictly-causal) [15]. $T_{\mathcal{K}}$ can be
rewritten in terms of $\mathcal{Q}$ as

$\displaystyle T_{\mathcal{Q}}$
$\displaystyle=\begin{pmatrix}\mathcal{P}_{11}+\mathcal{P}_{12}\mathcal{Q}\mathcal{P}_{21}&\mathcal{P}_{12}\mathcal{Q}\\\
\mathcal{Q}\mathcal{P}_{21}&\mathcal{Q}\end{pmatrix}.$ (7)

###  II-C Regret-optimal control under partial observability

Given a benchmark non-causal controller $\mathcal{Q}_{0}$, we define the
regret as:

$\displaystyle R(\mathcal{Q},w,v)$ $\displaystyle\coloneqq
J(\mathcal{Q},w,v)-J(\mathcal{Q}_{0},w,v)\operatorname*{}$
$\displaystyle=\begin{pmatrix}w^{\ast}&v^{\ast}\end{pmatrix}(T_{\mathcal{Q}}^{\ast}T_{\mathcal{Q}}-T_{\mathcal{Q}_{0}}^{\ast}T_{\mathcal{Q}_{0}})\begin{pmatrix}w\\\
v\\\ \end{pmatrix},$ (8)

which quantifies the additional cost incurred by a causal controller due to
its lack of knowledge about future disturbances. In simpler terms, regret
represents the disparity between the cost accrued by a causal controller and
the cost accumulated by a reference non-causal controller equipped with
complete foresight of the disturbance trajectory. For a reference controller
$\mathcal{Q}_{0}$, the regret-optimal control problem is defined below:

######  Problem 1 (The Regret Problem).

The problem of minimizing regret in the measurement-feedback, infinite-horizon
setting is:

$\inf_{\mathcal{Q}}\sup_{\begin{subarray}{c}w,v\end{subarray}}\frac{R(\mathcal{Q},w,v)}{\|w\|_{2}^{2}+\|v\|_{2}^{2}},$
(9)

with $R(\mathcal{Q},w,v)$ as in (8).

The goal in the next sections is to solve explicitely the regret problem,
first by reducing it to a Nehari problem (Section III) and then finding a
state-space solution for the controller $\mathcal{Q}$ (Section IV).

##  III Main result I: Reduction to a Nehari Problem

In this section, we focus on reducing the regret problem to a Nehari problem,
facilitating the derivation of an explicit solution for the causal regret-
optimal controller $\mathcal{Q}_{c}$. We will later transform this solution
into a state-space representation in Section IV.

The Nehari problem aims to find the optimal causal approximation for an
operator that is strictly anti-causal, as measured by the operator norm:

######  Problem 2 (The Nehari Problem).

Find a causal block Toeplitz operator $\mathcal{Y}$ that minimizes
$\|\mathcal{Y}-\mathcal{Z}\|$, where $\mathcal{Z}$ is a given strictly anti-
causal block Toeplitz operator.

Theorem 5 of [2] demonstrates that in cases where the operator has a state-
space structure, we can explicitly determine both the minimal norm and the
approximation $\mathcal{Y}$, as we will endeavor in Section IV.

Now, to tackle the regret problem, we first need to choose the reference (non-
causal) controller, $Q_{0}$. $Q_{0}$ should be chosen as an operationally-
superior controller with respect to the one to be designed. Otherwise, the
designed controller can be chosen as $Q_{0}$ and the optimal regret is zero.
We first present a general result which shows that the regret problem with
respect to _any reference controller_ can be reduced to a Nehari problem.

######  Theorem 1 (Reduction to a Nehari problem).

A $\gamma$-solution to the causal regret problem with respect to
$\mathcal{Q}_{0}$, i.e, a suboptimal solution to Problem 1 that satisfies

$\sup_{\begin{subarray}{c}w,v\end{subarray}}\frac{R(\mathcal{Q},w,v)}{\|w\|_{2}^{2}+\|v\|_{2}^{2}}\leq\gamma^{2},$
(10)

exists if and only if there exists a solution to the Nehari problem

$\displaystyle\inf_{\text{causal}\
\mathcal{C}}\|\mathcal{C}-\\{\mathcal{X}\\}_{-}\|\leq 1$ (11)

where $\\{\mathcal{X}\\}_{-}$ is defined to be the strictly anti-causal part
of the operator

$\mathcal{X}=\mathcal{M}^{1/2}\mathcal{Q}_{0}\mathcal{W}^{-/2}+\gamma^{-2}\mathcal{M}^{-\ast/2}\mathcal{T}(\mathcal{Q}_{2}-\mathcal{Q}_{0})\mathcal{W}^{-/2},$
(12)

and can be computed from the canonical factorizations

$\displaystyle\mathcal{M}$
$\displaystyle=\mathcal{M}^{\ast/2}\mathcal{M}^{1/2}\operatorname*{}$
$\displaystyle=\gamma^{-2}\mathcal{I}+\gamma^{-2}\mathcal{P}_{12}^{\ast}(\mathcal{I}+\gamma^{-2}\mathcal{P}_{11}(\mathcal{I}+\mathcal{P}_{21}^{\ast}\mathcal{P}_{21})^{-1}\mathcal{P}_{11}^{\ast})\times\operatorname*{}$
$\displaystyle\ \ \ \ \mathcal{P}_{12},$ (13) $\displaystyle\mathcal{W}$
$\displaystyle=\mathcal{W}^{\ast/2}\mathcal{W}^{1/2}\operatorname*{}$
$\displaystyle=(I+\mathcal{P}_{21}\mathcal{P}_{21}^{\ast})^{-1}+\gamma^{-4}(\mathcal{Q}_{2}-\mathcal{Q}_{0})^{\ast}\mathcal{T}\mathcal{M}^{-1}\mathcal{T}\times\operatorname*{}$
$\displaystyle\ \ \ \ (\mathcal{Q}_{2}-\mathcal{Q}_{0}).$ (14)

Here,

$\mathcal{T}=I+\mathcal{P}_{12}^{\ast}\mathcal{P}_{12}$ (15)

and

$\mathcal{Q}_{2}=-{\mathcal{T}}^{-1}\mathcal{P}_{12}^{\ast}\mathcal{P}_{11}\mathcal{P}_{21}^{\ast}(I+\mathcal{P}_{21}\mathcal{P}_{21}^{\ast})^{-1}$
(16)

is the optimal non-causal $\mathcal{H}_{2}$ controller that minimizes the
Frobenius norm of $T_{\mathcal{Q}}$ [15]. Moreover, the optimal regret is
equal to $\gamma^{2}$ where $\gamma$ achieves equality in (10). If a solution
$\mathcal{C}$ to (11) exists, then the regret-optimal controller is given by

$\displaystyle\mathcal{Q}_{c}=\mathcal{M}^{-/2}(\mathcal{C}+\\{\mathcal{X}\\}_{+})\mathcal{W}^{1/2}.$
(17)

The proofs for Theorem 1 and all subsequent lemmas and theorems can be found
in Appendix B of the extended paper online. We also remark that the reduction
is not limited to the doubly-infinite horizon regime and holds for the finite
and one-sided infinite regimes as well. The special case of
$\mathcal{Q}_{0}=\mathcal{Q}_{2}$ in Theorem 1 appeared in [16]. Theorem 1
shows that the control problem is equivalent to a Nehari problem, and thus the
minimum norm can be determined if one is able to perform the required
factorizations.

However, formulating an implementable controller is not straightforward since
canonical factorizations cannot be directly computed for high-dimensional
operators. In such cases, the reference controller can be selected based on
its superiority in the time-domain, controllability space, or observability.
Remarkably, the regret problem can be reduced to a Nehari problem in all these
scenarios, even though the Nehari problem primarily deals with operator
comparisons in the time-domain. A specific instance of Theorem 1 occurs when
the reference controller aligns with the optimal non-causal $\mathcal{H}_{2}$
controller.

We proceed to provide an explicit solution for the regret-optimal controller
in the infinite-horizon state-space control problem while focusing on the the
special case $\mathcal{Q}_{0}=\mathcal{Q}_{2}$. In this case, the operators
$\mathcal{X}$ (12) and $\mathcal{W}$ (14) simplify to

$\displaystyle\mathcal{X}$
$\displaystyle=\mathcal{M}^{1/2}\mathcal{Q}_{2}\mathcal{W}^{-/2}$ (18)
$\displaystyle\mathcal{W}$
$\displaystyle=\mathcal{W}^{\ast/2}\mathcal{W}^{1/2}=\mathcal{(}I+\mathcal{P}_{21}\mathcal{P}_{21}^{\ast})^{-1}.$
(19)

##  IV Main result II: The state-space model

In this section, our primary objective is to establish the state-space
formulation of the controller. To achieve this, we begin in Section IV-A by
representing the operators in the $z$-domain. We then proceed to Section IV-B,
where the main focus is on factorizing $\mathcal{W}$ (14) and $\mathcal{M}$
(13). Subsequently, in Section IV-C, we delve into the decomposition of
$\mathcal{X}$ (12) into causal and strictly-anti-causal operators. Finally, in
Section IV-D, we leverage these factorizations and operator decompositions to
provide both the explicit solution to the Nehari problem and a state-space
representation of the regret-optimal controller.

###  IV-A Operators in $z$-domain

The operators in the general problem (II-B) take the form of Toeplitz
operators that can be described concisely using the transfer matrices in the
$z$-domain:

$\displaystyle P_{11}(z)$ $\displaystyle\triangleq L(zI-F)^{-1}G_{1},$
$\displaystyle\quad P_{12}(z)$ $\displaystyle\triangleq
L(zI-F)^{-1}G_{2}\operatorname*{}$ $\displaystyle P_{21}(z)$
$\displaystyle\triangleq H(zI-F)^{-1}G_{1},$ $\displaystyle\quad P_{22}(z)$
$\displaystyle\triangleq H(zI-F)^{-1}G_{2}.$ (20)

Also, the solution can be expressed using the $z$-domain counterpart of the
operators (12)-(14), and (16) defined in Theorem 1, as follows:

$\displaystyle X(z)$ $\displaystyle=M^{1/2}(z)Q_{2}(z)W^{-/2}(z)$ (21)
$\displaystyle M(z)$
$\displaystyle=M^{\ast/2}(z^{\ast})M^{1/2}(z)=\gamma^{-2}I+\gamma^{-2}P_{12}^{\ast}(z^{-\ast})(I+\operatorname*{}$
$\displaystyle\ \ \ \
\gamma^{-2}P_{11}(z)(I+P_{21}^{\ast}(z^{-\ast})P_{21}(z))^{-1}P_{11}^{\ast}(z^{-\ast}))\times\operatorname*{}$
$\displaystyle\ \ \ \ P_{12}(z)$ (22) $\displaystyle W(z)$
$\displaystyle=W^{\ast/2}(z^{\ast})W^{1/2}(z)=\mathcal{(}I+P_{21}(z)P_{21}^{\ast}(z^{-\ast}))^{-1}$
(23) $\displaystyle Q_{2}(z)$
$\displaystyle=-(I+P_{12}^{\ast}(z^{-\ast})P_{12})^{-1}(z)P_{12}^{\ast}(z^{-\ast})P_{11}(z)\times\operatorname*{}$
$\displaystyle\ \ \ \
P_{21}^{\ast}(z^{-\ast})(I+P_{21}(z)P_{21}^{\ast}(z^{-\ast}))^{-1}$ (24)

We also define the operator

$\mathcal{S}=\mathcal{I}+\mathcal{P}_{12}\mathcal{P}^{\ast}_{12}$ (25)

with its transfer matrix

$\displaystyle S(z)$ $\displaystyle\triangleq
I+L(zI-F)^{-1}G_{2}G_{2}^{\ast}(zI-F^{\ast})^{-1}L^{\ast},$ (26)

whose factorization will be used in the derivation of the solution to the
Nehari problem.

###  IV-B Factorizations

To be able to perform the required factorizations, we introduce the following
assumption:

######  Assumption 1.

Assume that ($F,H$) and ($F,L$) are detectable, ($F,G_{1}$) is stabilizable,
and ($F,G_{1}$) and ($F,G_{2}$) are controllable on the unit circle.

For the rest of this paper, we rely on assumption 1, and its significance will
become apparent when reviewing the proofs in the appendix. We now factorize
the operator $\mathcal{W}$ (14) in Lemma 1:

######  Lemma 1 (Factorization of $\mathcal{W}$).

The transfer matrix

$W^{-1}(z)=I+P_{21}(z)P_{21}^{\ast}(z^{-\ast})$ (27)

can be factorized as

$\displaystyle W^{-/2}(z)W^{-\ast/2}(z^{-\ast})$
$\displaystyle=I+H(zI-F)^{-1}G_{1}G_{1}^{\ast}\times\operatorname*{}$
$\displaystyle\ \ \ \ (z^{-1}I-F^{\ast})^{-1}H^{\ast}$ (28)

with

$\displaystyle W^{-/2}(z)$ $\displaystyle=(I+H(zI-F)^{-1}K_{W})R_{W}^{1/2},$
(29)

where

$\displaystyle K_{W}$
$\displaystyle=FP_{U}H^{\ast}R_{W}^{-1},\operatorname*{}\quad F_{W}=F-K_{W}H$
$\displaystyle R_{W}$
$\displaystyle=I+HP_{U}H^{\ast}=R_{W}^{1/2}R_{W}^{\ast/2},$ (30)

and $P_{U}$ is the stabilizing solution to the Riccati equation

$\displaystyle P_{U}$
$\displaystyle=FP_{U}F^{\ast}+G_{1}G_{1}^{\ast}-K_{W}R_{W}K_{W}^{\ast}.$ (31)

Finally,

$\displaystyle W^{1/2}(z)$
$\displaystyle=R_{W}^{-/2}(I+H(zI-F)^{-1}K_{W})^{-1}\operatorname*{}$
$\displaystyle=R_{W}^{-/2}(I-H(zI-F_{W})^{-1}K_{W})$ (32) $\displaystyle
W^{\ast/2}(z^{-\ast})$
$\displaystyle=(I+K_{W}^{\ast}(z^{-1}I-F^{\ast})^{-1}H^{\ast})^{-1}R_{W}^{-\ast/2},$
(33)

and $W^{1/2}(z)$ is causal and stable (bounded on the unit circle).

Now, we present a series of helpful lemmas (Lemma 2-4) that we use to get to
the factorization of the operator $\mathcal{M}$ (13) in Lemma 5.

######  Lemma 2 (Factorization of $\mathcal{M}$).

The operator $\mathcal{M}$ (13) can be directly factorized as

$\displaystyle\mathcal{M}^{\ast/2}\mathcal{M}^{1/2}$
$\displaystyle=\gamma^{-2}(\mathcal{I}+\mathcal{P}_{12}^{\ast}\nabla^{-\ast}\nabla^{-1}\mathcal{P}_{12}),$
(34)

where $\mathcal{M}^{1/2}$ is a causal operator and $\Gamma,\nabla$ are causal
operators obtained from

$\displaystyle\Gamma^{\ast}\Gamma$
$\displaystyle=\gamma^{2}(I+\mathcal{P}_{21}^{\ast}\mathcal{P}_{21})+\mathcal{P}_{11}^{\ast}\mathcal{P}_{11}$
(35) $\displaystyle\nabla\nabla^{\ast}$
$\displaystyle=\mathcal{I}-\mathcal{P}_{11}\Gamma^{-1}\Gamma^{-\ast}\mathcal{P}_{11}^{\ast}.$
(36)

We proceed to factorize $\Gamma^{\ast}\Gamma$ (35) and
$\nabla\nabla^{\ast}$(36) in Lemmas 3,4 respectively.

######  Lemma 3.

The transfer matrix corresponding to (35),

$\Gamma^{\ast}(z^{-\ast})\Gamma(z)=\gamma^{2}(I+P_{21}^{\ast}(z^{-\ast})P_{21}(z))+P^{\ast}_{11}(z^{-\ast})P_{11}(z),$
(37)

can be factorized as $\Gamma^{\ast}(z^{-\ast})\Gamma(z)$ with

$\displaystyle\Gamma(z)$
$\displaystyle=R_{\Gamma}^{1/2}(I+K_{\Gamma}(zI-F)^{-1}G_{1}),$ (38)

where

$\displaystyle K_{\Gamma}$
$\displaystyle=R_{\Gamma}^{-1}G_{1}^{\ast}P_{\Gamma}F,\quad
F_{\Gamma}=F-G_{1}K_{\Gamma}\operatorname*{}$ $\displaystyle R_{\Gamma}$
$\displaystyle=\gamma^{2}I+G_{1}^{\ast}P_{\Gamma}G_{1}=R_{\Gamma}^{\ast/2}R_{\Gamma}^{1/2},$
(39)

and $P_{\Gamma}$ is the stabilizing solution to the Riccati equation

$\displaystyle P_{\Gamma}$
$\displaystyle=F^{\ast}P_{\Gamma}F+(\gamma^{2}H^{\ast}H+L^{\ast}L)-K_{\Gamma}^{\ast}R_{\Gamma}K_{\Gamma}.$
(40)

######  Lemma 4.

The transfer matrix corresponding to (36),

$\nabla(z)\nabla^{\ast}(z^{-\ast})=I-P_{11}(z)\Gamma^{-1}\Gamma^{-\ast}P_{11}^{\ast}(z^{-\ast}),$
(41)

can be factorized as

$\displaystyle\nabla(z)\nabla^{\ast}(z^{-\ast})$
$\displaystyle=I-G(z)\Gamma^{-1}(z)\Gamma^{-\ast}(z^{-\ast})G^{\ast}(z^{-\ast}),$
(42)

with

$\displaystyle\nabla(z)$ $\displaystyle=(I+L(zI-
F_{\Gamma})^{-1}K_{\nabla})R_{\nabla}^{1/2},$ (43)

where

$\displaystyle K_{\nabla}$
$\displaystyle=F_{\Gamma}P_{\nabla}L^{\ast}R_{\nabla}^{-1},\operatorname*{}\quad
F_{\nabla}=F_{\Gamma}-K_{\nabla}L$ $\displaystyle R_{\nabla}$
$\displaystyle=I+LP_{\nabla}L^{\ast}=R_{\nabla}^{1/2}R_{\nabla}^{\ast/2},$
(44)

and $P_{\nabla}$ is the stabilizing solution to the Riccati equation

$\displaystyle P_{\nabla}$
$\displaystyle=F_{\Gamma}P_{\nabla}F_{\Gamma}^{\ast}-G_{1}R_{\Gamma}^{-1}G_{1}^{\ast}-K_{\nabla}R_{\nabla}K_{\nabla}^{\ast}.$
(45)

Now, we get to the desired factorization of $\mathcal{M}$ (13):

######  Lemma 5.

The transfer matrix corresponding to (34),

$M(z)=\gamma^{-2}(I+P_{12}^{\ast}(z^{-\ast})\nabla^{-\ast}(z^{-\ast})\nabla^{-1}(z)P_{12}(z)),$
(46)

can be factored as $M^{\ast/2}(z^{-\ast})M^{1/2}(z)$ with

$\displaystyle M^{1/2}(z)$ $\displaystyle=R_{M}^{1/2}(I+K_{M}(zI-
F_{E})^{-1}G_{E}),$ (47)

where

$\displaystyle K_{M}$
$\displaystyle=R_{M}^{-1}G_{E}^{\ast}P_{M}F_{E}\operatorname*{}\quad
F_{M}=F_{E}-G_{E}K_{M}$ $\displaystyle R_{M}$
$\displaystyle=\gamma^{-2}I+G_{E}^{\ast}P_{M}G_{E}=R_{M}^{\ast/2}R_{M}^{1/2},$
(48)

and $P_{M}$ is a solution to the Riccati equation

$\displaystyle P_{M}$
$\displaystyle=F_{E}^{\ast}P_{M}F_{E}+H_{E}^{\ast}H_{E}-K_{M}^{\ast}R_{M}K_{M}.$
(49)

The constants are defined as

$\displaystyle H_{E}$
$\displaystyle\triangleq\begin{pmatrix}R_{\nabla}^{-/2}L&R_{\nabla}^{-/2}L\end{pmatrix}\operatorname*{}$
$\displaystyle F_{E}$ $\displaystyle\triangleq\begin{pmatrix}F&0\\\
-K_{\nabla}L&F_{\nabla}\end{pmatrix},\quad
G_{E}\triangleq\begin{pmatrix}\gamma^{-1}G_{2}\\\ 0\end{pmatrix}.$ (50)

Also,

$\displaystyle M^{-1/2}(z)$ $\displaystyle=(I+K_{M}(zI-
F_{E})^{-1}G_{E})^{-1}R_{M}^{-/2}$ (51)

Finally, we factorize the operator $\mathcal{S}$ (25) in the following lemma
that we leverage later in Section IV-C to find the Nehari solution.

######  Lemma 6 (Factorization of $\mathcal{S}$).

The transfer matrix $S(z)$ (26) can be factorized as
$S^{1/2}(z)S^{\ast/2}(z^{-\ast})$ with

$\displaystyle S^{1/2}(z)$ $\displaystyle=(I+L(zI-F)^{-1}K_{S})R_{S}^{1/2},$
(52)

where

$\displaystyle K_{S}$ $\displaystyle=FP_{S}L^{\ast}R_{S}^{-1},\quad
F_{S}=F-K_{S}L,\operatorname*{}$ $\displaystyle R_{S}$
$\displaystyle=I+LP_{S}L^{\ast}=R_{S}^{1/2}R_{S}^{\ast/2},$ (53)

and $P_{S}$ is the stabilizing solution to the Riccati equation

$\displaystyle P_{S}$
$\displaystyle=FP_{S}F^{\ast}+G_{2}G_{2}^{\ast}-K_{S}R_{S}K_{S}^{\ast}.$ (54)

Finally,

$\displaystyle S^{-/2}(z)$
$\displaystyle=R_{S}^{-/2}(I+L(zI-F)^{-1}K_{S})^{-1}$ (55)

and $S^{-/2}(z)=(S^{1/2}(z))^{-1}$ is causal and stable (bounded on the unit
circle).

###  IV-C Decomposition

Here in Lemma 7, we decompose $X(z)$ (21) into its causal and strictly anti-
causal parts, to be used in solving the Nehari problem (11) as we will see in
Lemma 8.

######  Lemma 7.

The transfer matrix $X(z)$ (21) can be expressed as a sum of two causal and a
strictly anti-causal transfer matrices given by

$\displaystyle C_{1}(z)$ $\displaystyle=-M^{1/2}(z)G_{2}^{\ast}U_{1}F_{S}[(zI-
F_{S})^{-1}F_{S}U_{2}+U_{2}]\times\operatorname*{}$ $\displaystyle\ \ \ \
H^{\ast}R_{W}^{-\ast/2}$ (56) $\displaystyle C_{2}(z)$
$\displaystyle=-R_{M}^{1/2}K_{M}[(zI-F_{E})^{-1}F_{E}U_{3}+U_{3}]G_{A}$ (57)
$\displaystyle A(z)$ $\displaystyle=H_{A}(z^{-1}I-F_{A}^{\ast})^{-1}G_{A},$
(58)

where $U_{1},U_{2},U_{3}$ are given as the solution to the Lyapunov/Sylvester
equations

$\displaystyle U_{1}$
$\displaystyle=L^{\ast}R_{S}^{-1}L+F_{S}^{\ast}U_{1}F_{S}\operatorname*{}$
$\displaystyle U_{2}$
$\displaystyle=G_{1}G_{1}^{\ast}+F_{S}U_{2}F_{W}^{\ast}\operatorname*{}$
$\displaystyle U_{3}$
$\displaystyle=F_{E}U_{3}F_{A}^{\ast}+G_{E}\tilde{H}_{A}$ (59)

with

$\tilde{H}_{A}=G_{2}^{\ast}\begin{pmatrix}U_{1}U_{2}&F_{S}^{\ast}\end{pmatrix},$
(60)

and the constants for the anti-causal function are

$\displaystyle H_{A}$
$\displaystyle=-R_{M}^{1/2}(\tilde{H}_{A}+K_{M}U_{3}F_{A}^{\ast}),\operatorname*{}$
$\displaystyle F_{A}^{\ast}$ $\displaystyle=\begin{pmatrix}F_{W}^{\ast}&0\\\
U_{1}G_{1}G_{1}^{\ast}&F_{S}^{\ast}\end{pmatrix},\quad
G_{A}=\begin{pmatrix}H^{\ast}R_{W}^{-\ast/2}\\\ 0\end{pmatrix}.$ (61)

###  IV-D Final Solution

In Lemma 8, we employ Theorem 5 from [1] to determine the optimal causal
operator $\mathcal{C}_{N}$. This operator serves as the solution to the Nehari
problem (11), which can also be expressed in terms of transfer matrices as
follows:

$\displaystyle\inf_{\text{causal}\ C(z)}\|C(z)-\\{X(z)\\}_{-}\|\leq 1,$ (62)

with

$\\{X(z)\\}_{-}=A(z)=H_{A}(z^{-1}I-F_{A}^{\ast})^{-1}G_{A}.$ (63)

######  Lemma 8.

The solution to the Nehari problem (62) is

$\displaystyle C_{N}(z)$ $\displaystyle=H_{A}\Pi(F_{N}(zI-
F_{N})^{-1}+I)K_{N},$ (64)

where

$\displaystyle K_{N}$
$\displaystyle=(I-F_{A}ZF_{A}^{\ast}\Pi)^{-1}F_{A}ZG_{A},\quad
F_{N}=F_{A}-K_{N}G_{A}^{\ast},$ (65)

and $Z$ and $\Pi$ are the solutions to the Lyapunov equations

$\displaystyle Z$
$\displaystyle=F_{A}ZF_{A}^{\ast}+H_{A}^{\ast}H_{A}\operatorname*{}$
$\displaystyle\Pi$ $\displaystyle=F_{A}^{\ast}\Pi F_{A}+G_{A}G_{A}^{\ast}.$
(66)

Recall from Theorem 1 that the Youla parameterization of the causal regret-
optimal controller is $\mathcal{Q}_{c}$ (17) and thus its transfer matrix can
be written as

$\displaystyle Q_{c}(z)$
$\displaystyle=M^{-/2}(z)(\\{X(z)\\}_{+}+C_{N}(z))W^{1/2}(z)\operatorname*{}$
$\displaystyle=M^{-/2}(z)(C_{1}(z)+C_{2}(z)+C_{N}(z))W^{1/2}(z)$ (67)

with $C_{1}(z)$ (56), $C_{2}(z)$ (57) are as found by Lemma 7, and $C_{N}(z)$
(64) is the solution to the Nehari problem as found by Lemma 8. $M^{-/2}(z)$
(51) is as found by Lemma 5, and $W^{1/2}(z)$ (32) is as found by Lemma 1.

With these components, we can now present our main result: in Theorem 2, we
establish a state-space formulation for the regret-optimal controller $Q_{c}$
(IV-D) which, according to Theorem 1, solves Problem 1:

######  Theorem 2.

Under the condition of Assumption 1, the causal regret-optimal controller
corresponding to $Q_{c}(z)$ (IV-D) which solves Problem 1 suboptimally is:

$\displaystyle\xi_{i+1}$ $\displaystyle=A\xi_{i}+By_{i}\operatorname*{}$
$\displaystyle q_{i}$ $\displaystyle=C\xi_{i}+Dy_{i}$ (68)

where:

$\displaystyle A$ $\displaystyle=\begin{pmatrix}F_{W}&0&0&0\\\
-K_{N}R_{W}^{-/2}H&F_{N}&0&0\\\
-U_{3}G_{A}R_{W}^{-/2}H&G_{E}R_{M}^{-/2}H_{A}\Pi&F_{M}&0\\\
-U_{2}H^{\ast}R_{W}^{-1}H&0&0&F_{S}\end{pmatrix}\operatorname*{}$
$\displaystyle B$ $\displaystyle=\begin{pmatrix}F_{W}K_{W}\\\
-K_{N}R_{W}^{-/2}HK_{W}+F_{N}K_{N}R_{W}^{-/2}\\\ B_{3}\\\
-U_{2}H^{\ast}R_{W}^{-1}HK_{W}+F_{S}U_{2}H^{\ast}R_{W}^{-1}\end{pmatrix}\operatorname*{}$
$\displaystyle C$
$\displaystyle=\begin{pmatrix}0&R_{M}^{-/2}H_{A}\Pi&-K_{M}&-G_{2}^{\ast}U_{1}F_{S}\end{pmatrix}\operatorname*{}$
$\displaystyle D$ $\displaystyle=[R_{M}^{-/2}H_{A}\Pi
K_{N}-K_{M}U_{3}G_{A}\operatorname*{}$ $\displaystyle\ \ \
-G_{2}^{\ast}U_{1}F_{S}U_{2}H^{\ast}R_{W}^{-\ast/2}]R_{W}^{-/2}.$ (69)

with

$\displaystyle B_{3}$ $\displaystyle\triangleq-
U_{3}G_{A}R_{W}^{-/2}HK_{W}+G_{E}R_{M}^{-/2}H_{A}\Pi
K_{N}R_{W}^{-/2}\operatorname*{}$ $\displaystyle\ \ \
+F_{M}U_{3}G_{A}R_{W}^{-/2}$ (70)

This theorem shows that we are able to find an online controller that
minimizes regret over an infinite-horizon in a partially observable system. It
allows for practical implementation of the controller as we will see in
Section V.

##  V Simulations

In this section, we showcase the the practicality of implementing the regret-
optimal controller and its effectiveness as compared to the $\mathcal{H}_{2}$
and $\mathcal{H}_{\infty}$ controllers, for various systems. We assess them in
the frequency domain to enable a comprehensive comparison across a wide
spectrum of disturbances.

We examine four distinct systems: 1) A system with $n=m=p=q=4$ featuring a
randomly generated unstable matrix $F$, with all other matrices set to $I$, 2)
a system with $n=m=p=q=2$ characterized by a randomly generated marginally
stable matrix $F$, while other matrices set to $I$, 3) an aircraft dynamic
model (AC15) [18] with $F$ unstable and 4) an aircraft dynamic model (AC5)
with $F$ marginally stable as described in [18]. In our analysis, we depict
three key performance metrics: i) The squared Frobenius norm minimized by the
$\mathcal{H}_{2}$ controller (71), ii) the squared operator norm minimized by
$\mathcal{H}_{\infty}$ controller (72), and iii) the regret minimized by the
regret-optimal controller (73), as is shown in the equations below:

$\displaystyle\|T_{Q}\|_{F}^{2}=\frac{1}{2\pi}\int_{0}^{2\pi}Tr(T_{Q}^{\ast}(e^{jw})T_{Q}(e^{jw}))dw,$
(71) $\displaystyle\|T_{Q}\|^{2}=\max_{0\leq w\leq
2\pi}\sigma_{max}(T_{Q}^{\ast}(e^{jw})T_{Q}(e^{jw})),$ (72)
$\displaystyle\|T_{Q}^{\ast}T_{Q}-T_{Q_{0}}^{\ast}T_{Q_{0}}\|=\max_{0\leq
w\leq 2\pi}\sigma_{max}(T_{Q}^{\ast}(e^{jw})T_{Q}(e^{jw})-\operatorname*{}$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad
T_{Q_{0}}^{\ast}(e^{jw})T_{Q_{0}}(e^{jw})).$ (73)

The performance metrics across all frequencies are shown in Figure 1 for the
(AC15) system. Table I(b) shows the different norms (71)-(73) resulting for
each controller for each of the four systems.

The non-causal controller achieves minimal Frobenius and operator norms.
Specifically, $H_{2}$ minimizes the Frobenius norm (area under the curve) but
exhibits notable peaks at certain frequencies (Figure 1(a), Table I(b)), while
$H_{\infty}$ minimizes the operator norm (Figure 1(b), Table I(b)) but
demonstrates subpar average performance (large area under the curve in Figure
1(a)). In contrast, the regret-optimal controller combines attributes of both
$H_{2}$ and $H_{\infty}$. Its Frobenius and operator norms interpolate between
those of $H_{2}$ and $H_{\infty}$, and it boasts the lowest regret (Figure
1(c), Table I(b)). Furthermore, it consistently maintains a stable separation
from the non-causal controller, distinguishing itself from $H_{2}$ and
$H_{\infty}$ controllers. By minimizing the maximum cost deviation from the
non-causal controller, it achieves balanced performance across all input
disturbances.

TABLE I: Controllers’ performance across different systems with highlighted
best metrics

(a) Controllers’ performance on random systems |  System 1 (random, unstable $F$) |  System 2 (random, marginally stable $F$)
---|---|---
| $\|T_{Q}\|_{F}^{2}$ | $\|T_{Q}\|^{2}$ | Regret | $\|T_{Q}\|_{F}^{2}$ | $\|T_{Q}\|^{2}$ | Regret
NC | 2.974 | 1.1371 | - | 1.4338 | 1.1366 | -
RO | 6.7121 | 4.0726 | 3.1322 | 6.1077 | 3.7573 | 2.7821
H2 | 5.6905 | 7.8981 | 6.9475 | 4.5032 | 6.8703 | 5.8715
H$\infty$ | 6.9961 | 3.6896 | 3.6365 | 6.6593 | 3.3806 | 3.2927

(b) Controllers’ performance on aircraft models [18] | System 3 (AC5) | System 4 (AC15)
---|---|---
| $\|T_{Q}\|_{F}^{2}$ | $\|T_{Q}\|^{2}$ | Regret | $\|T_{Q}\|_{F}^{2}$ | $\|T_{Q}\|^{2}$ | Regret
NC | 41.060 | 354.28 | - | 66.614 | 222.93 | -
RO | 1048.1 | 1277.1 | 1006.8 | 434.21 | 577.79 | 363.24
H2 | 269.03 | 2858.3 | 2663.8 | 343.34 | 1054.2 | 841.49
H$\infty$ | 1187.1 | 1166.1 | 1166.1 | 497.36 | 477.22 | 470.57

(a) Squared Frobenius norm

(b) Squared operator norm

(c) Regret

Figure 1: Performance metrics for AC15 [18]

## References

  * [1] O. Sabag, G. Goel, S. Lale, and B. Hassibi, “Regret-optimal controller for the full-information problem,” in _2021 American Control Conference (ACC)_ , 2021, pp. 4777–4782.
  * [2] ——, “Regret-optimal LQR control,” _arXiv preprint arXiv:2105.01244_ , 2021.
  * [3] Z. Nehari, “On bounded bilinear forms,” _Annals of Mathematics_ , vol. 65, no. 1, pp. 153–162, 1957.
  * [4] G. Goel and B. Hassibi, “Regret-optimal control in dynamic environments,” _arXiv preprint arXiv:2010.10473_ , 2020.
  * [5] A. Martin, L. Furieri, F. Dörfler, J. Lygeros, and G. Ferrari-Trecate, “Safe control with minimal regret,” in _Proceedings of The 4th Annual Learning for Dynamics and Control Conference_ , vol. 168. PMLR, Jun. 2022, pp. 726–738.
  * [6] A. Didier, J. Sieber, and M. N. Zeilinger, “A system level approach to regret optimal control,” _IEEE Control Systems Letters_ , vol. 6, pp. 2792–2797, 2022.
  * [7] O. Sabag, S. Lale, and B. Hassibi, “Optimal competitive-ratio control,” _arXiv preprint arXiv:2206.01782_ , 2022.
  * [8] G. Goel and B. Hassibi, “Competitive control,” _IEEE Transactions on Automatic Control_ , 2022.
  * [9] A. Karapetyan, A. Iannelli, and J. Lygeros, “On the regret of ${H}_{\infty}$ control,” in _2022 IEEE 61st Conference on Decision and Control (CDC)_. IEEE, 2022, pp. 6181–6186.
  * [10] A. Martin, L. Furieri, F. Dörfler, J. Lygeros, and G. Ferrari-Trecate, “Follow the clairvoyant: an imitation learning approach to optimal control,” 2022.
  * [11] J. Liu and P. Seiler, “Robust regret optimal control,” _arXiv preprint arXiv:2307.14297_ , 2023.
  * [12] J. Hajar, T. Kargin, and B. Hassibi, “Wasserstein distributionally robust regret-optimal control under partial observability,” _arXiv preprint arXiv:2307.04966_ , 2023.
  * [13] S. Yan, F. A. Taha, and E. Bitar, “A distributionally robust approach to regret optimal control using the wasserstein distance,” _arXiv preprint arXiv:2304.06783_ , 2023.
  * [14] O. Sabag and B. Hassibi, “Regret-optimal filtering for prediction and estimation,” _IEEE Transactions on Signal Processing_ , vol. 70, pp. 5012–5024, 2022.
  * [15] B. Hassibi, A. H. Sayed, and T. Kailath, _Indefinite-Quadratic Estimation and Control_. Society for Industrial and Applied Mathematics, 1999. [Online]. Available: https://epubs.siam.org/doi/abs/10.1137/1.9781611970760
  * [16] G. Goel and B. Hassibi, “Regret-optimal measurement-feedback control,” in _Learning for Dynamics and Control_. PMLR, 2021, pp. 1270–1280.
  * [17] ——, “Measurement-feedback control with optimal data-dependent regret,” _arXiv preprint arXiv:2209.06425_ , 2022.
  * [18] F. Leibfritz and W. Lipinski, “Description of the benchmark examples in compleib 1.0,” _Dept. Math, Univ. Trier, Germany_ , vol. 32, 2003.
  * [19] T. Kailath, A. Sayed, and B. Hassibi, _Linear Estimation_ , ser. Prentice-Hall information and system sciences series. Prentice Hall, 2000. [Online]. Available: https://books.google.com/books?id=zNJFAQAAIAAJ

##  Appendix A Canonical Factorizations

In this appendix, we introduce two key lemmas illustrating the canonical
factorization of a causal operator. In Lemma 9, we demonstrate the
anticausal/causal or upper/lower (UL) factorization, while in Lemma 10, we
showcase the causal/anticausal or lower/upper (LU) factorization.

######  Lemma 9.

Consider a general strictly causal transfer matrix
$\underline{M}(z)=\underline{H}(zI-\underline{F})^{-1}\underline{G}$ with
$\underline{R}\succeq 0$ and ($\underline{F},\underline{G}$) stabilizable.
Then, the following canonical factorization holds

$\displaystyle\Omega^{\ast}(z^{-\ast})\Omega(z)$
$\displaystyle=R+\underline{M}^{\ast}(z^{-\ast})\underline{M}(z)$ (74)

with the causal transfer matrix

$\displaystyle\Omega(z)$
$\displaystyle=R_{P}^{1/2}(I+K_{P}(zI-\underline{F})^{-1}\underline{G}),$ (75)

where

$\displaystyle K_{P}$
$\displaystyle=R_{P}^{-1}\underline{G}^{\ast}P\underline{F}$ (76)
$\displaystyle R_{P}$
$\displaystyle=R+\underline{G}^{\ast}P\underline{G}=R_{P}^{\ast/2}R_{P}^{1/2},$
(77)

and $P=P^{\ast}$ is the unique stabilizing solution to the Riccati equation

$\displaystyle P$
$\displaystyle=\underline{F}^{\ast}P\underline{F}+\underline{H}^{\ast}\underline{H}-K_{P}^{\ast}R_{P}K_{P}.$
(78)

The inverse of the factorization is causal and bounded on the unit circle, and
is equal to

$\displaystyle\Omega^{-1}(z)$
$\displaystyle=(I-K_{P}(zI-\underline{F}_{P})^{-1}\underline{G})R_{P}^{-/2}$
(79)

with

$F_{P}=\underline{F}-\underline{G}K_{P}.$ (80)

###### Proof of Lemma 9.

We can express $R+\underline{M}^{\ast}(z^{-\ast})\underline{M}(z)$ in matrix
form as

$\displaystyle\begin{pmatrix}\underline{G}^{*}(z^{-1}I-\underline{F}^{\ast})^{-1}&I\end{pmatrix}\begin{pmatrix}\underline{H}^{\ast}\underline{H}&0\\\
0&R\end{pmatrix}\begin{pmatrix}(zI-\underline{F})^{-1}\underline{G}\\\
I\end{pmatrix}\operatorname*{}$ $\displaystyle\
=\begin{pmatrix}\underline{G}^{*}(z^{-1}I-\underline{F}^{\ast})^{-1}&I\end{pmatrix}\times\operatorname*{}$
$\displaystyle\ \ \
\begin{pmatrix}\underline{H}^{\ast}\underline{H}-P+\underline{F}^{*}P\underline{F}&\underline{F}^{*}P\underline{G}\\\
\underline{G}^{\ast}P\underline{F}&R+\underline{G}^{\ast}P\underline{G}\end{pmatrix}\begin{pmatrix}(zI-\underline{F})^{-1}\underline{G}\\\
I\end{pmatrix},$ (81)

where the equality holds for any $P=P^{\ast}$ from the identity

$\displaystyle 0$
$\displaystyle=\begin{pmatrix}\underline{G}^{*}(z^{-1}I-\underline{F}^{\ast})^{-1}&I\end{pmatrix}\begin{pmatrix}-P+\underline{F}^{*}P\underline{F}&\underline{F}^{*}P\underline{G}\\\
\underline{G}^{\ast}P\underline{F}&\underline{G}^{\ast}P\underline{G}\end{pmatrix}\times\operatorname*{}$
$\displaystyle\ \ \ \ \begin{pmatrix}(zI-\underline{F})^{-1}\underline{G}\\\
I\end{pmatrix}.$ (82)

The middle matrix in (A) can be factored as

$\begin{pmatrix}I&K_{P}^{\ast}\\\
0&I\end{pmatrix}\begin{pmatrix}\Gamma(P)&0\\\
0&R_{P}\end{pmatrix}\begin{pmatrix}I&0\\\ K_{P}&I\end{pmatrix},$ (83)

with

$\displaystyle\Gamma(P)$
$\displaystyle=-P+\underline{F}^{*}P\underline{F}+\underline{H}^{\ast}\underline{H}-K_{P}^{\ast}R_{P}K_{P}\operatorname*{}$
$\displaystyle K_{P}$
$\displaystyle=R_{P}^{-1}\underline{G}^{*}P\underline{F}\operatorname*{}$
$\displaystyle R_{P}$ $\displaystyle=R+\underline{G}^{\ast}P\underline{G}.$
(84)

Since $(\underline{F},\underline{G})$ is stabilizable, then the Riccati
equation $\Gamma(P)=0$ has a unique stabilizing Hermitian solution. Suppose
$P$ is chosen to be this solution. Then, we can obtain the factorization by
defining

$\Omega(z)=R_{P}^{1/2}(I+K_{P}(zI-\underline{F})^{-1}\underline{G})$ (85)

with $R_{P}=R_{P}^{\ast/2}R_{P}^{1/2}$.

Moreover, $\Omega^{-1}(z)$ (79) is found using the matrix inversion lemma. Its
poles are at the eigenvalues of $F_{P}$ which is stable (since $P$ is the
stabilizing solution to the Riccati equation), which due to the causality of
$\Omega^{-1}(z)$ guarantees the boundedness of $\Omega^{-1}(z)$ on the unit
circle. ∎

######  Lemma 10.

Consider a general strictly causal transfer matrix
$\underline{M}(z)=\underline{H}(zI-\underline{F})^{-1}\underline{G}$ with
$R\succeq 0$ and ($\underline{F},\underline{H}$) detectable. Then, the
following canonical factorization holds

$\displaystyle\Omega(z)\Omega^{\ast}(z^{-\ast})$
$\displaystyle=R+\underline{M}(z)\underline{M}^{\ast}(z^{-\ast})$ (86)

with the causal transfer matrix

$\displaystyle\Omega(z)$
$\displaystyle=(I+\underline{H}(zI-F)^{-1}K_{P})R_{P}^{1/2},$ (87)

where

$\displaystyle K_{P}$
$\displaystyle=\underline{F}P\underline{H}^{\ast}R_{P}^{-1}$ (88)
$\displaystyle R_{P}$
$\displaystyle=R+\underline{H}P\underline{H}^{\ast}=R_{P}^{1/2}R_{P}^{\ast/2},$
(89)

and $P=P^{\ast}$ is the unique stabilizing solution to the Riccati equation

$\displaystyle P$
$\displaystyle=\underline{F}P\underline{F}^{\ast}+\underline{G}\underline{G}^{\ast}-K_{P}R_{P}K_{P}^{\ast}.$
(90)

The inverse of the factorization is causal and bounded on the unit circle, and
is equal to

$\displaystyle\Omega^{-1}(z)$ $\displaystyle=R_{P}^{-/2}(I-\underline{H}(zI-
F_{P})^{-1}K_{P})$ (91)

with

$F_{P}=\underline{F}-K_{P}H.$ (92)

###### Proof of Lemma 10.

The proof is similar to the proof of Lemma 9 and will be ommitted here. Refer
to Theorem 8.3.1 in [19] for a detailed proof. ∎

Corollary to Lemma 10.  Consider a general strictly causal transfer matrix
$\underline{M}(z)=\underline{H}(zI-\underline{F})^{-1}\underline{G}$ with
$R\succeq 0$ and ($\underline{F},\underline{H}$) detectable. Then, the
following canonical factorization holds

$\displaystyle\Omega(z)\Omega^{\ast}(z^{-\ast})$
$\displaystyle=R-\underline{M}(z)\underline{M}^{\ast}(z^{-\ast})$ (93)

with the causal transfer matrix $\Omega(z)$ as in (87), $K_{P}$ as in (88),
$R_{P}$ as in (89), and $P=P^{\ast}$ is the unique stabilizing solution to the
Riccati equation

$\displaystyle P$
$\displaystyle=\underline{F}P\underline{F}^{\ast}-\underline{G}\underline{G}^{\ast}-K_{P}R_{P}K_{P}^{\ast}.$
(94)

The inverse of the factorization is causal and bounded on the unit circle, and
is equal to $\Omega^{-1}(z)$ as in (91), and $F_{P}$ as in (92).

##  Appendix B Proofs

In this appendix, we present the proofs for the theorems and lemmas presented
in the paper.

###### Proof of Theorem 1.

To reduce the sub-optimal regret problem to a Nehari problem with respect to
the reference controller $\mathcal{Q}_{0}$, we should simplify the inequality

$\displaystyle
T_{\mathcal{Q}}^{\ast}T_{\mathcal{Q}}-T_{\mathcal{Q}_{0}}T_{\mathcal{Q}_{0}}\preceq\gamma^{2}\mathcal{I},$
(95)

which is equivalent to

$\displaystyle(\theta T_{\mathcal{Q}}\psi)^{\ast}\theta
T_{\mathcal{Q}}\psi-(\theta T_{\mathcal{Q}_{0}}\psi)^{\ast}\theta
T_{\mathcal{Q}_{0}}\psi\preceq\gamma^{2}\mathcal{I},$ (96)

for any unitary operators $\theta$ and $\psi$. We choose the unitary operators

$\displaystyle\theta$ $\displaystyle=\begin{pmatrix}\mathcal{S}^{-/2}&0\\\
0&\mathcal{T}^{-\ast/2}\end{pmatrix}\begin{pmatrix}I&-\mathcal{P}_{12}\\\
\mathcal{P}_{12}^{\ast}&I\end{pmatrix}\operatorname*{}$ $\displaystyle\psi$
$\displaystyle=\begin{pmatrix}I&\mathcal{P}_{21}^{\ast}\\\
-\mathcal{P}_{21}&I\end{pmatrix}\begin{pmatrix}\mathcal{V}^{-/2}&0\\\
0&\mathcal{U}^{-\ast/2}\end{pmatrix},$ (97)

where the canonical factorizations are defined as

$\displaystyle\mathcal{S}^{1/2}\mathcal{S}^{\ast/2}$
$\displaystyle=I+\mathcal{P}_{12}\mathcal{P}_{12}^{\ast},\ \ \
\mathcal{U}^{1/2}\mathcal{U}^{\ast/2}=I+\mathcal{P}_{21}\mathcal{P}_{21}^{\ast},\operatorname*{}$
$\displaystyle\mathcal{T}^{\ast/2}\mathcal{T}^{1/2}$
$\displaystyle=I+\mathcal{P}_{12}^{\ast}\mathcal{P}_{12},\ \ \
\mathcal{V}^{\ast/2}\mathcal{V}^{1/2}=I+\mathcal{P}_{21}^{\ast}\mathcal{P}_{21},$
(98)

with $\mathcal{E}^{1/2}$ being a causal operator for any positive operator
$\mathcal{E}$.

The operator $\theta T_{\mathcal{Q}}\psi$ can be computed from the product of
its three middle matrices as

$\displaystyle\begin{pmatrix}I&-\mathcal{P}_{12}\\\
\mathcal{P}_{12}^{\ast}&I\end{pmatrix}\begin{pmatrix}\mathcal{P}_{12}\mathcal{Q}\mathcal{P}_{21}+\mathcal{P}_{11}&\mathcal{P}_{12}\mathcal{Q}\\\
\mathcal{Q}\mathcal{P}_{21}&\mathcal{Q}\end{pmatrix}\times\operatorname*{}$
$\displaystyle\begin{pmatrix}I&\mathcal{P}_{21}^{\ast}\\\
-\mathcal{P}_{21}&I\end{pmatrix}\operatorname*{}$
$\displaystyle=\operatorname*{}$
$\displaystyle\begin{pmatrix}\mathcal{P}_{11}&\mathcal{P}_{11}\mathcal{P}_{21}^{\ast}\\\
\mathcal{P}_{12}^{\ast}\mathcal{P}_{11}&(I+\mathcal{P}_{12}^{\ast}\mathcal{P}_{12})\mathcal{Q}(I+\mathcal{P}_{21}\mathcal{P}_{21}^{\ast})+\mathcal{P}_{12}^{\ast}\mathcal{P}_{11}\mathcal{P}_{21}^{\ast}\end{pmatrix}.\operatorname*{}$

The remaining diagonal matrices can be easily added to obtain $\theta
T_{\mathcal{Q}}\psi$. It also follows directly that the non-causal $H_{2}$
controller is

$\mathcal{Q}_{2}\triangleq-(I+\mathcal{P}_{12}^{\ast}\mathcal{P}_{12})^{-1}\mathcal{P}_{12}^{\ast}\mathcal{P}_{11}\mathcal{P}_{21}^{\ast}(I+\mathcal{P}_{21}\mathcal{P}_{21}^{\ast})^{-1}$
(99)

since the Frobenius norm is minimized when the $(2,2)$ term is equal to zero.
Now, omitting additive terms that do not depend on $\mathcal{Q}$ (since they
have no effect on the regret), the squared cost operator is

$\displaystyle(\theta T_{Q}\psi)^{\ast}\theta T_{Q}\psi$
$\displaystyle\dot{=}\operatorname*{}$
$\displaystyle\begin{pmatrix}0&\mathcal{V}^{-\ast/2}\mathcal{P}_{11}^{\ast}\mathcal{P}_{12}(\mathcal{Q}-\mathcal{Q}_{2})\mathcal{U}^{1/2}\\\
(\cdot)^{\ast}&\mathcal{U}^{\ast/2}(\mathcal{Q}^{\ast}-\mathcal{Q}_{2}^{\ast})\mathcal{T}(\mathcal{Q}-\mathcal{Q}_{2})\mathcal{U}^{1/2}\end{pmatrix}.$
(100)

The sub-optimal regret problem in (96) can be written as

$\displaystyle\begin{pmatrix}\gamma^{2}\mathcal{I}&\mathcal{V}^{-\ast/2}\mathcal{P}_{11}^{\ast}\mathcal{P}_{12}(\mathcal{Q}_{0}-\mathcal{Q})\mathcal{U}^{1/2}\\\
(\cdot)^{\ast}&\mathcal{N}_{2,2}\end{pmatrix}\succeq 0,$ (101)

where

$\displaystyle\mathcal{N}_{2,2}$
$\displaystyle\triangleq\gamma^{2}\mathcal{I}+\mathcal{U}^{\ast/2}(\mathcal{Q}_{0}^{\ast}-\mathcal{Q}_{2}^{\ast})\mathcal{T}(\mathcal{Q}_{0}-\mathcal{Q}_{2})\mathcal{U}^{1/2}-\mathcal{U}^{\ast/2}\times\operatorname*{}$
$\displaystyle\ \ \ \
(\mathcal{Q}^{\ast}-\mathcal{Q}_{2}^{\ast})\mathcal{T}(\mathcal{Q}-\mathcal{Q}_{2})\mathcal{U}^{1/2}.$
(102)

Since $\gamma^{2}\mathcal{I}\succ 0$, the condition (101) can be rewritten
using its Schur complement as

$\displaystyle\mathcal{N}_{2,2}$
$\displaystyle\succeq\mathcal{U}^{\ast/2}(\mathcal{Q}-\mathcal{Q}_{0})^{\ast}\mathcal{P}_{12}^{\ast}\mathcal{P}_{11}\gamma^{-2}\mathcal{V}^{-1}\mathcal{P}_{11}^{\ast}\mathcal{P}_{12}\times\operatorname*{}$
$\displaystyle\ \ \ \ (\mathcal{Q}-\mathcal{Q}_{0})\mathcal{U}^{1/2}.$ (103)

Using the completion of squares, we can re-write (B) as

$\displaystyle(\mathcal{Q}^{\ast}-\mathcal{B}^{\ast}\mathcal{D}^{-1})\mathcal{D}(\mathcal{Q}-\mathcal{D}^{-1}\mathcal{B})\operatorname*{}$
$\displaystyle\ \ \
\preceq\gamma^{2}\mathcal{U}^{-1}+(\mathcal{Q}_{2}-\mathcal{Q}_{0})^{\ast}\mathcal{T}\mathcal{D}^{-1}\mathcal{T}(\mathcal{Q}_{2}-\mathcal{Q}_{0})$
(104)

with

$\mathcal{D}\triangleq\mathcal{T}+\mathcal{P}_{12}^{\ast}\mathcal{P}_{11}\gamma^{-2}\mathcal{V}^{-1}\mathcal{P}_{11}^{\ast}\mathcal{P}_{12}$
(105)

and

$\displaystyle\mathcal{B}$
$\displaystyle\triangleq\mathcal{T}\mathcal{Q}_{2}+\mathcal{P}_{12}^{\ast}\mathcal{P}_{11}\gamma^{-2}\mathcal{V}^{-1}\mathcal{P}_{11}^{\ast}\mathcal{P}_{12}\mathcal{Q}_{0}\operatorname*{}$
$\displaystyle=\mathcal{T}(\mathcal{Q}_{2}-\mathcal{Q}_{0})+\mathcal{D}\mathcal{Q}_{0}.$
(106)

To complete the proof, define the canonical factorizations of the positive
operators

$\displaystyle\mathcal{W}$
$\displaystyle=\mathcal{U}^{-1}+\gamma^{-4}(\mathcal{Q}_{2}-\mathcal{Q}_{0})^{\ast}\mathcal{T}\mathcal{M}^{-1}\mathcal{T}(\mathcal{Q}_{2}-\mathcal{Q}_{0})$
(107) $\displaystyle\mathcal{M}$
$\displaystyle=\gamma^{-2}\mathcal{I}+\gamma^{-2}\mathcal{P}_{12}^{\ast}(\mathcal{I}+\gamma^{-2}\mathcal{P}_{11}(\mathcal{I}+\mathcal{P}_{21}^{\ast}\mathcal{P}_{21})^{-1}\mathcal{P}_{11}^{\ast})\times\operatorname*{}$
$\displaystyle\ \ \ \ \mathcal{P}_{12},$ (108)

to express concisely the condition (B) as

$\displaystyle(\mathcal{M}^{1/2}\mathcal{Q}\mathcal{W}^{-/2}-\mathcal{X})^{\ast}(\mathcal{M}^{1/2}\mathcal{Q}\mathcal{W}^{-/2}-\mathcal{X})\preceq\mathcal{I}$
(109)

with

$\mathcal{X}\triangleq\mathcal{M}^{1/2}\mathcal{Q}_{0}\mathcal{W}^{-/2}+\gamma^{-2}\mathcal{M}^{-\ast/2}\mathcal{T}(\mathcal{Q}_{2}-\mathcal{Q}_{0})\mathcal{W}^{-/2}.$
(110)

By decomposing the operator
$\mathcal{X}=\\{\mathcal{X}\\}_{+}+\\{\mathcal{X}\\}_{-}$, and the fact that
the operator $\mathcal{M}^{1/2}{\mathcal{Q}}\mathcal{W}^{-/2}$ is a product of
three causal operators, we conclude that (107) is a Neahri problem of finding
a causal operator to approximate the anticausal operator
$\\{\mathcal{X}\\}_{-}$. Furthermore, if there exists a solution to this
Nehari problem, say $\mathcal{C}$, then the solution to the sub-optimal regret
problem in (95) is

$\mathcal{Q}=\mathcal{M}^{-/2}(\mathcal{C}+\\{\mathcal{X}\\}_{+})\mathcal{W}^{1/2}.$
(111)

∎

###### Proof of Lemma 1 .

To obtain the required factorization, we apply Lemma 10 with
$\underline{R}=I$, $\underline{H}=H=$, $\underline{F}=F$, and
$\underline{G}=G_{1}$. ∎

###### Proof of Lemma 2.

The inverse of the positive inner operator
$I+\gamma^{-2}\mathcal{P}_{11}(I+\mathcal{P}_{21}^{\ast}\mathcal{P}_{21})^{-1}\mathcal{P}_{11}^{\ast}$
can be factorized as

$\displaystyle\nabla\nabla^{\ast}$
$\displaystyle=(I+\gamma^{-2}\mathcal{P}_{11}(I+\mathcal{P}_{21}^{\ast}\mathcal{P}_{21})^{-1}\mathcal{P}_{11}^{\ast})^{-1}\operatorname*{}$
$\displaystyle=I-\mathcal{P}_{11}(\gamma^{2}(I+\mathcal{P}_{21}^{\ast}\mathcal{P}_{21})+\mathcal{P}_{11}^{\ast}\mathcal{P}_{11})^{-1}\mathcal{P}_{11}^{\ast}\operatorname*{}$
$\displaystyle=I-\mathcal{P}_{11}(\gamma^{2}(I+\mathcal{P}_{21}^{\ast}\mathcal{P}_{21})+\mathcal{P}_{11}^{\ast}\mathcal{P}_{11})^{-1}\mathcal{P}_{11}^{\ast}\operatorname*{}$
$\displaystyle=I-\mathcal{P}_{11}\Gamma^{-1}\Gamma^{-\ast}\mathcal{P}_{11}^{\ast},$
(112)

where in the last equality we use the canonical factorization

$\displaystyle\Gamma^{\ast}\Gamma$
$\displaystyle=\gamma^{2}(I+\mathcal{P}_{21}^{\ast}\mathcal{P}_{21})+\mathcal{P}_{11}^{\ast}\mathcal{P}_{11}.$
(113)

The operator can now be directly factorized as

$\mathcal{M}^{\ast/2}\mathcal{M}^{1/2}=\gamma^{-2}(\mathcal{I}+\mathcal{P}_{12}^{\ast}\nabla^{-\ast}\nabla^{-1}\mathcal{P}_{12}).$
(114)

∎

###### Proof of Lemma 3.

We start by writing the transfer matrix explicitly

$\displaystyle\gamma^{2}I+\gamma^{2}P_{21}^{\ast}(z^{-\ast})P_{21}(z)+P_{11}^{\ast}(z^{-\ast})P_{11}(z)\operatorname*{}$
$\displaystyle\
=\gamma^{2}I+G_{1}^{\ast}(z^{-1}I-F^{\ast})^{-1}(\gamma^{2}H^{\ast}H+L^{\ast}L)\times\operatorname*{}$
$\displaystyle\ \ \ \ (zI-F)^{-1}G_{1}.\operatorname*{}$

To obtain the required factorization, we apply Lemma 9 with
$\underline{R}=\gamma^{2}I$,
$\underline{H}=(\gamma^{2}H^{\ast}H+L^{\ast}L)^{1/2}$, $\underline{F}=F$, and
$\underline{G}=G_{1}$. ∎

###### Proof of Lemma 4.

We start by simplifying the product

$\displaystyle G(z)\Gamma^{-1}(z)$ $\displaystyle=L(zI-
F_{\Gamma})^{-1}G_{1}R_{\Gamma}^{-/2}.$ (115)

Thus, the factorization should be for the expression

$\displaystyle I-L(zI-F_{\Gamma})^{-1}G_{1}R_{\Gamma}^{-1}G_{1}^{\ast}(zI-
F_{\Gamma}^{\ast})^{-1}L^{\ast}.$ (116)

We obtain the desired result by applying the Corollary to Lemma 10 with
$\underline{R}=I$, $\underline{H}=L$, $\underline{F}=F_{\Gamma}$ and
$\underline{G}=-G_{1}R_{\Gamma}^{-/2}$. ∎

###### Proof of Lemma 5.

We start by simplifying $\nabla^{-1}(z)P_{12}(z)$ as follows:

$\displaystyle R_{\nabla}^{-/2}(I+L(zI-
F_{\Gamma})^{-1}K_{\nabla})^{-1}L(zI-F)^{-1}G_{2}\operatorname*{}$
$\displaystyle=R_{\nabla}^{-/2}L(I-(zI-
F_{\Gamma}+K_{\nabla}L)^{-1}K_{\nabla}L)(zI-F)^{-1}G_{2}\operatorname*{}$
$\displaystyle=R_{\nabla}^{-/2}L(I-(zI-
F_{\nabla})^{-1}K_{\nabla}L)(zI-F)^{-1}G_{2}.$ (117)

We can write (B) concisely as

$\gamma^{-1}\nabla^{-1}(z)F(z)=H_{E}(zI-F_{E})^{-1}G_{E},$

with

$\displaystyle H_{E}$
$\displaystyle\triangleq\begin{pmatrix}R_{\nabla}^{-/2}L&R_{\nabla}^{-/2}L\end{pmatrix},\operatorname*{}$
$\displaystyle F_{E}$ $\displaystyle\triangleq\begin{pmatrix}F&0\\\
-K_{\nabla}L&F_{\nabla}\end{pmatrix},\quad
G_{E}\triangleq\begin{pmatrix}\gamma^{-1}G_{2}\\\ 0\end{pmatrix}.$ (118)

In order to complete the factorization of $M(z)$, we apply Lemma 9 with
$\underline{R}=\gamma^{-2}I$, $\underline{H}=H_{E}$, $\underline{F}=F_{E}$,
and $\underline{G}=G_{E}$. ∎

###### Proof of Lemma 6.

The desired factorization can be obtained through applying Lemma 10 with
$\underline{R}=I$, $\underline{H}=L$, $\underline{F}=F$, and
$\underline{G}=G_{2}$. ∎

###### Proof of Lemma 7.

The operator

$M^{1/2}(z)Q_{2}(z)W^{-/2}(z)$

that needs to be decomposed can be expressed as

$\displaystyle-M^{1/2}(z)(I+P_{12}^{\ast}(z^{-\ast})P_{12}(z))^{-1}P_{12}^{\ast}(z^{-\ast})P_{11}(z)\times\operatorname*{}$
$\displaystyle\ \ \ \
P_{21}^{\ast}(z^{-\ast})(I+P_{21}(z)P_{21}^{\ast}(z^{-\ast}))^{-1}W^{-/2}(z)\operatorname*{}$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}-M^{1/2}(z)P_{12}^{\ast}(z^{-\ast})S^{-\ast/2}(z^{-\ast})S^{-/2}(z)P_{11}(z)\times\operatorname*{}$
$\displaystyle\ \ \ \ P_{21}^{\ast}(z^{-\ast})W^{\ast/2}(z^{-\ast}),$ (119)

where $(a)$ follows from the factorization

$S^{1/2}(z)S^{\ast/2}(z^{-\ast})=I+P_{12}(z)P_{12}^{\ast}(z^{-\ast}).$

For convenience, the canonical factorizations required to compute (B) (derived
in Lemmas 1, 5, and 6) are summarized as

$\displaystyle M^{1/2}(z)$ $\displaystyle=R_{M}^{1/2}(I+K_{M}(zI-
F_{E})^{-1}G_{E})\operatorname*{}$ $\displaystyle P_{12}^{\ast}(z^{-\ast})$
$\displaystyle=G_{2}^{\ast}(z^{-1}I-F^{\ast})^{-1}L^{\ast}\operatorname*{}$
$\displaystyle S^{-\ast/2}(z^{-\ast})$
$\displaystyle=(I+K_{S}^{\ast}(z^{-1}I-F^{\ast})^{-1}L^{\ast})^{-1}R_{S}^{-\ast/2}\operatorname*{}$
$\displaystyle S^{-/2}(z)$
$\displaystyle=R_{S}^{-/2}(I+L(zI-F)^{-1}K_{S})^{-1}\operatorname*{}$
$\displaystyle P_{11}(z)$ $\displaystyle=L(zI-F)^{-1}G_{1}\operatorname*{}$
$\displaystyle P_{21}^{\ast}(z^{-\ast})$
$\displaystyle=G_{1}^{\ast}(z^{-1}I-F^{\ast})^{-1}H^{\ast}\operatorname*{}$
$\displaystyle W^{\ast/2}(z^{-\ast})$
$\displaystyle=(I+K_{W}^{\ast}(z^{-1}I-F^{\ast})^{-1}H^{\ast})^{-1}R_{W}^{-\ast/2}.$
(120)

We begin by combining some of the pairs products in (B)

$\displaystyle P_{12}^{\ast}(z^{-\ast})S^{-\ast/2}(z^{-\ast})\operatorname*{}$
$\displaystyle=G_{2}^{\ast}(z^{-1}I-F^{\ast})^{-1}(I+L^{\ast}K_{S}^{\ast}(z^{-1}I-F^{\ast})^{-1})^{-1}L^{\ast}\times\operatorname*{}$
$\displaystyle\ \ \ \ R_{S}^{-\ast/2}$ (121)
$\displaystyle=G_{2}^{\ast}(z^{-1}I-F_{S}^{\ast})^{-1}L^{\ast}R_{S}^{-\ast/2},$
(122)

and similarly

$\displaystyle S^{-/2}(z)P_{11}(z)$ $\displaystyle=R_{S}^{-/2}L(zI-
F_{S})^{-1}G_{1}\operatorname*{}$ $\displaystyle
P_{21}^{\ast}(z^{-\ast})W^{\ast/2}(z^{-\ast})$
$\displaystyle=G_{1}^{\ast}(z^{-1}I-F_{W}^{\ast})^{-1}H^{\ast}R_{W}^{-\ast/2}.$
(123)

The combination of these products can be decomposed as

$\displaystyle
P_{12}^{\ast}(z^{-\ast})S^{-\ast/2}(z^{-\ast})S^{-/2}(z)P_{11}(z)P_{21}^{\ast}(z^{-\ast})W^{\ast/2}(z^{-\ast})\operatorname*{}$
$\displaystyle=G_{2}^{\ast}(z^{-1}I-F_{S}^{\ast})^{-1}L^{\ast}R_{S}^{-1}L(zI-
F_{S})^{-1}\times\operatorname*{}$ $\displaystyle\ \ \ \
G_{1}G_{1}^{\ast}(z^{-1}I-F_{W}^{\ast})^{-1}H^{\ast}R_{W}^{-\ast/2}\operatorname*{}$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}G_{2}^{\ast}[F_{S}^{\ast}(z^{-1}I-F_{S}^{\ast})^{-1}U_{1}+U_{1}F_{S}(zI-
F_{S})^{-1}\operatorname*{}$ $\displaystyle\ \ \ \
+U_{1}]G_{1}G_{1}^{\ast}(z^{-1}I-F_{W}^{\ast})^{-1}H^{\ast}R_{W}^{-\ast/2}\operatorname*{}$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}G_{2}^{\ast}[F_{S}^{\ast}(z^{-1}I-F_{S}^{\ast})^{-1}U_{1}G_{1}G_{1}^{\ast}(z^{-1}I-F_{W}^{\ast})^{-1}\operatorname*{}$
$\displaystyle\ \ \ \
+U_{1}G_{1}G_{1}^{\ast}(z^{-1}I-F_{W}^{\ast})^{-1}\operatorname*{}$
$\displaystyle\ \ \ \
+U_{1}F_{S}U_{2}F_{W}^{\ast}(z^{-1}I-F_{W}^{\ast})^{-1}\operatorname*{}$
$\displaystyle\ \ \ \ +U_{1}F_{S}(zI-
F_{S})^{-1}F_{S}U_{2}+U_{1}F_{S}U_{2}]H^{\ast}R_{W}^{-\ast/2}$ (124)

where $(a)$ follows from

$\displaystyle(z^{-1}I-F_{S}^{\ast})^{-1}L^{\ast}R_{S}^{-1}L(zI-
F_{S})^{-1}\operatorname*{}$
$\displaystyle=(z^{-1}I-F_{S}^{\ast})^{-1}F_{S}^{\ast}U_{1}+U_{1}F_{S}(zI-
F_{S})^{-1}+U_{1}$ (125)

where $U_{1}$ is the solution to the Lyapunov equation

$L^{\ast}R_{S}^{-1}L+F_{S}^{\ast}U_{1}F_{S}=U_{1},$ (126)

and $(b)$ follows from

$\displaystyle(zI-
F_{S})^{-1}G_{1}G_{1}^{\ast}(z^{-1}I-F_{W}^{\ast})^{-1}\operatorname*{}$
$\displaystyle=(zI-
F_{S})^{-1}F_{s}U_{2}+U_{2}F_{W}^{\ast}(z^{-1}I-F_{W}^{\ast})^{-1}+U_{2}$
(127)

where $U_{2}$ is the solution to the Sylvester equation

$G_{1}G_{1}^{\ast}+F_{S}U_{2}F_{W}^{\ast}=U_{2}.$ (128)

Before combining (B) with $-M^{1/2}(z)$, we denote the strictly anticausal
part of (B) as

$\displaystyle
G_{2}^{\ast}[F_{S}^{\ast}(z^{-1}I-F_{S}^{\ast})^{-1}U_{1}G_{1}G_{1}^{\ast}(z^{-1}I-F_{W}^{\ast})^{-1}\operatorname*{}$
$\displaystyle\ \ \
+U_{1}U_{2}(z^{-1}I-F_{W}^{\ast})^{-1}]H^{\ast}R_{W}^{-\ast/2}\operatorname*{}$
$\displaystyle=G_{2}^{\ast}\begin{pmatrix}U_{1}U_{2}&F_{S}^{\ast}\end{pmatrix}(z^{-1}I-\begin{pmatrix}F_{W}^{\ast}&0\\\
U_{1}G_{1}G_{1}^{\ast}&F_{S}^{\ast}\end{pmatrix})^{-1}\times\operatorname*{}$
$\displaystyle\ \ \ \begin{pmatrix}H^{\ast}R_{W}^{-\ast/2}\\\
0\end{pmatrix}\operatorname*{}$
$\displaystyle\triangleq\tilde{H}_{A}(z^{-1}I-F_{A}^{\ast})^{-1}G_{A}.$ (129)

We now decompose the product of $-M^{1/2}(z)$ and (B) as

$\displaystyle-M^{1/2}(z)\tilde{H}_{A}(z^{-1}I-F_{A}^{\ast})^{-1}G_{A}\operatorname*{}$
$\displaystyle=-R_{M}^{1/2}(I+K_{M}(zI-
F_{E})^{-1}G_{E})\tilde{H}_{A}(z^{-1}I-F_{A}^{\ast})^{-1}\times\operatorname*{}$
$\displaystyle\ \ \ \ G_{A}\operatorname*{}$
$\displaystyle=-R_{M}^{1/2}\tilde{H}_{A}(z^{-1}I-F_{A}^{\ast})^{-1}G_{A}-R_{M}^{1/2}K_{M}\times\operatorname*{}$
$\displaystyle\ \ \ \ [(zI-
F_{E})^{-1}F_{E}U_{3}+U_{3}F_{A}^{\ast}(z^{-1}I-F_{A}^{\ast})^{-1}+U_{3}]G_{A},$
(130)

where $U_{3}$ is the solution to the Lyapunov equation

$U_{3}=F_{E}U_{3}F_{A}^{\ast}+G_{E}\tilde{H}_{A}.$ (131)

To summarize, the strictly anticausal part of

$M^{1/2}(z)Q_{2}(z)W^{-/2}(z)$

is

$\displaystyle A(z)$
$\displaystyle=-R_{M}^{1/2}[\tilde{H}_{A}+K_{M}U_{3}F_{A}^{\ast}](z^{-1}I-F_{A}^{\ast})^{-1}G_{A}\operatorname*{}$
$\displaystyle\triangleq H_{A}(z^{-1}I-F_{A}^{\ast})^{-1}G_{A}$ (132)

with

$\displaystyle
H_{A}=-R_{M}^{1/2}(\tilde{H}_{A}+K_{M}U_{3}F_{A}^{\ast})\operatorname*{}$
$\displaystyle F_{A}^{\ast}=\begin{pmatrix}F_{W}^{\ast}&0\\\
U_{1}G_{1}G_{1}^{\ast}&F_{S}^{\ast}\end{pmatrix},\quad
G_{A}=\begin{pmatrix}H^{\ast}R_{W}^{-\ast/2}\\\ 0\end{pmatrix},$ (133)

and

$\tilde{H}_{A}=G_{2}^{\ast}\begin{pmatrix}U_{1}U_{2}&F_{S}^{\ast}\end{pmatrix},$
(134)

while the causal part can presented as a sum of two causal functions

$\displaystyle C_{1}(z)$ $\displaystyle=-M^{1/2}(z)G_{2}^{\ast}U_{1}F_{S}[(zI-
F_{S})^{-1}F_{S}U_{2}+U_{2}]\times\operatorname*{}$ $\displaystyle\ \ \ \
H^{\ast}R_{W}^{-\ast/2}$ (135) $\displaystyle C_{2}(z)$
$\displaystyle=-R_{M}^{1/2}K_{M}[(zI-F_{E})^{-1}F_{E}U_{3}+U_{3}]G_{A}.$ (136)

∎

###### Proof of Lemma 8.

This lemma is adapted from Theorem 5 in [2] and consequently, its proof
mirrors that of the aforementioned theorem. For comprehensive details, please
refer to [2]. ∎

###### Proof of Theorem 2.

Recall from Theorem 1 that the Youla parameterization of the causal regret-
optimal controller is:

$\displaystyle Q_{c}(z)$
$\displaystyle=M^{-1/2}(z)(C_{1}(z)+C_{2}(z)+C_{N}(z))W^{1/2}(z)$ (137)

where

$\displaystyle M^{-1/2}(z)$ $\displaystyle=(I+K_{M}(zI-
F_{E})^{-1}G_{E})^{-1}R_{M}^{-/2}\operatorname*{}$ $\displaystyle=(I-K_{M}(zI-
F_{M})^{-1}G_{E})R_{M}^{-/2}\operatorname*{}$ $\displaystyle C_{N}(z)$
$\displaystyle=H_{A}\Pi(F_{N}(zI-F_{N})^{-1}+I)K_{N}\operatorname*{}$
$\displaystyle W^{1/2}(z)$ $\displaystyle=R_{W}^{-/2}(I-H(zI-
F_{W})^{-1}K_{W})\operatorname*{}$ $\displaystyle C_{1}(z)$
$\displaystyle=-zM^{1/2}(z)G_{2}^{\ast}U_{1}F_{S}(zI-
F_{S})^{-1}U_{2}H^{\ast}\times\operatorname*{}$ $\displaystyle\ \ \ \
R_{W}^{-\ast/2}$ (138) $\displaystyle C_{2}(z)$
$\displaystyle=-zR_{M}^{1/2}K_{M}(zI-F_{E})^{-1}U_{3}G_{A}.$ (139)

First, we write each of the (partial) products in (137) explicitly

$\displaystyle M^{-1/2}(z)C_{1}(z)R_{W}^{-/2}$
$\displaystyle=-zG_{2}^{\ast}U_{1}F_{S}(zI-
F_{S})^{-1}U_{2}H^{\ast}\times\operatorname*{}$ $\displaystyle\ \ \ \
R_{W}^{-1}\operatorname*{}$ $\displaystyle M^{-1/2}(z)C_{2}(z)R_{W}^{-/2}$
$\displaystyle=-zK_{M}(zI-F_{M})^{-1}U_{3}G_{A}R_{W}^{-/2}\operatorname*{}$
$\displaystyle M^{-1/2}(z)C_{N}(z)R_{W}^{-/2}$ $\displaystyle=z(I-K_{M}(zI-
F_{M})^{-1}G_{E})\times\operatorname*{}$ $\displaystyle\ \ \
R_{M}^{-/2}H_{A}\Pi(zI-F_{N})^{-1}K_{N}R_{W}^{-/2}.$ (140)

The sum of the terms in (B) can be concisely expressed as

$\displaystyle
z\begin{pmatrix}R_{M}^{-/2}H_{A}\Pi&-K_{M}&-G_{2}^{\ast}U_{1}F_{S}\end{pmatrix}\times\operatorname*{}$
$\displaystyle\left(zI-\begin{pmatrix}F_{N}&0&0\\\
G_{E}R_{M}^{-/2}H_{A}\Pi&F_{M}&0\\\
0&0&F_{S}\end{pmatrix}\right)^{-1}\times\operatorname*{}$
$\displaystyle\begin{pmatrix}K_{N}R_{W}^{-/2}\\\ U_{3}G_{A}R_{W}^{-/2}\\\
U_{2}H^{\ast}R_{W}^{-1}\end{pmatrix}.$ (141)

Finally, we can use the general formula:

$\displaystyle A(zI-F)^{-1}B(I-C(zI-D)^{-1}E)\operatorname*{}$ $\displaystyle\
=\begin{pmatrix}0&A\end{pmatrix}(zI-\begin{pmatrix}D&0\\\
-BC&F\end{pmatrix})^{-1}\begin{pmatrix}E\\\ B\end{pmatrix},$ (142)

to multiply (B) with $(I-H(zI-F_{W})^{-1}K_{W})$ to obtain (137):

$\displaystyle
Q_{c}(z)=z\begin{pmatrix}0&R_{M}^{-/2}H_{A}\Pi&-K_{M}&-G_{2}^{\ast}U_{1}F_{S}\end{pmatrix}\times\operatorname*{}$
$\displaystyle\left(zI-\begin{pmatrix}F_{W}&0&0&0\\\
-K_{N}R_{W}^{-/2}H&F_{N}&0&0\\\ -U_{3}G_{A}R_{W}^{-/2}H&\tilde{F}&F_{M}&0\\\
-U_{2}H^{\ast}R_{W}^{-1}H&0&0&F_{S}\end{pmatrix}\right)^{-1}\times\operatorname*{}$
$\displaystyle\begin{pmatrix}K_{W}\\\ K_{N}R_{W}^{-/2}\\\
U_{3}G_{A}R_{W}^{-/2}\\\ U_{2}H^{\ast}R_{W}^{-1}\end{pmatrix},$ (143)

with

$\tilde{F}=G_{E}R_{M}^{-/2}H_{A}\Pi$ (144)

The proof is completed by applying the time-unit shift

$zA(zI-B)^{-1}C=A(I+(zI-B)^{-1}B)C$ (145)

to (B) in order to obtain the regret-optimal controller in Theorem 2. ∎


# Origin of ferroelectric domain wall alignment with surface trenches in
ultrathin films

Jack S. Baker1,2 David R. Bowler1,2,3 1London Centre for Nanotechnology, UCL,
17-19 Gordon St, London WC1H 0AH, UK  2Department of Physics & Astronomy, UCL,
Gower St, London WC1E 6BT, UK  3International Centre for Materials
Nanoarchitectonics (MANA) National Institute for Materials Science (NIMS), 1-1
Namiki, Tsukuba, Ibaraki 305-0044, Japan

###### Abstract

Engraving trenches on the surfaces of ultrathin ferroelectric (FE) films and
superlattices promises control over the orientation and direction of FE domain
walls (DWs). Through exploiting the phenomenon of DW-surface trench (ST)
parallel alignment, systems where DWs are known for becoming electrical
conductors could now become useful nanocircuits using only standard
lithographical techniques. Despite this clear application, the microscopic
mechanism responsible for the alignment phenomenon has remained elusive. Using
ultrathin PbTiO3 films as a model system, we explore this mechanism with large
scale density functional theory simulations on as many as 5,136 atoms.
Although we expect multiple contributing factors, we show that parallel DW-ST
alignment can be well explained by this configuration giving rise to an
arrangement of electric dipole moments which best restore polar continuity to
the film. These moments preserve the polar texture of the pristine film, thus
minimizing ST-induced depolarizing fields. Given the generality of this
mechanism, we suggest that STs could be used to engineer other exotic polar
textures in a variety of FE nanostructures as supported by the appearance of
ST-induced polar cycloidal modulations in this letter. Our simulations also
support experimental observations of ST-induced negative strains which have
been suggested to play a role in the alignment mechanism.

††preprint: APS/123-QED

Ferroelectric (FE) domain walls (DWs) have been observed in parallel alignment
with surface and substrate defects since the 2000’s [1, 2, 3, 4, 5]. While
this phenomenon was initially proposed to control polarization dependent
surface reactions [3], now that electrically conducting FE DWs [6, 7] have
been realized, new horizons for nanoelectronic devices approach. That is, the
phenomenon now promises control over the orientation of 2D conducting channels
on the nanoscale; a potential pathway for fabricating DW-mediated nanocircuits
with standard lithographical techniques. Presently, control over FE DWs and
polar textures is achieved using carefully directed electric fields. With this
technique, exotic topological phases including polar skyrmions [8] have been
stabilized and DW-based nanocomponents [9, 10, 11, 12] have been created; the
latter defining the emerging field of DW nanoelectronics via DW injection [13,
14, 15, 9, 16]. While directed electric fields offer the advantages of
nanocircuits written with reversibility [7], they can lack permanence once the
fields are released. Defect mediated alignment, however, offers enduring
control without external stimuli. In light of this, we envisage great
opportunities for these techniques to be applied in tandem helping to enable
the next generation of DW-nanoelectronic devices.

Notwithstanding these advances, the mechanism for DW alignment with surface
defects is far from a settled topic. One study used macroscopic time-dependent
Landau–Ginzburg–Devonshire theory to examine the interaction of PbTiO3
(PTO)/SrTiO3 (STO) superlattices with the milled edges of the nanostructure
[5]. They suggested that alignment arises from lowering the bulk and
electrostrictive energy terms driven by the release of lateral mechanical
restraints near the edge. Another study used a nanofocused X-ray beam to
reveal giant strain and strain gradients near milled surface defects on the
PTO/STO superlattice [4]. This opens the door to the possibility of
piezoelectric and/or flexoelectric contributions to the alignment mechanism.
Until the present letter, no microscopic origin for the phenomenon has been
proposed.

The feature of the domain structure pinned in the alignment is also disputed.
A density functional theory (DFT) study of steps on the PTO (001) surface [17]
found that DWs formed exactly at the step edges, suggesting that the DWs
themselves are pinned. In contrast, an effective Hamiltonian study [18] found
that film stability was enhanced when the domain centroid (DC, the area of
maximal out-of-plane polarization furthest from the DW) was pinned by the step
edge. While the DFT study [17], in principle, should offer higher accuracy
predictions, like-for-like comparisons cannot be made since the effective
Hamiltonian study [18] treats the underlying flux-closure domain structure
[19, 20] when the former does not. Indeed, providing a full treatment of FE
domains with DFT quickly becomes intractable due to the $\mathcal{O}(N^{3})$
(where $N$ is the number of atoms) scaling wall for computational time present
in conventional DFT codes [21, 22].

Figure 1: (a-c) Bird’s eye views of ST configurations. +AFD is bracketed in
(a) and (b) as they are treated with and without AFD modes. (a) Parallel to a
domain wall, positioned over a domain wall: $\parallel^{\text{DW}}$. (b)
Parallel to a domain wall, positioned over a domain centroid:
$\parallel^{\text{DC}}$. (c) Perpendicular to a domain wall: $\perp$ \+ AFD.
d) Looking down the axis (the [010] direction) of the $d=1$
$\parallel^{\text{DW}}$ film.

In this letter, we address these discrepancies and offer a microscopic
explanation of the DW alignment mechanism. Through performing state-of-the-art
large scale DFT simulations of ultrathin PTO films, we venture far beyond
previous works, able to treat the interaction between surface trenches (STs)
and FE domains with DFT accuracy. We propose that DW-ST parallel alignment can
be well explained by considering which ST orientations (relative to the DW)
are best able to support continuity of the polar texture. This continuity
minimizes new ST-induced depolarizing fields which would otherwise act to
destabilize the polydomain polarization. We suggest that this alignment is
only one consequence of a wider class of polar textures that can be produced
through minimizing ST-induced depolarizing fields. This idea is reinforced by
the appearance of engineered polar cycloidal modulations in our films.

To overcome the $\mathcal{O}(N^{3})$ scaling wall of conventional DFT, the
$\mathcal{O}(N)$ [22, 21] scaling algorithm [23, 24, 25] in the CONQUEST code
[26, 27] (v1.0.5 [28]) is used with a single-$\zeta$ plus polarization basis
set of pseudoatomic orbitals [29, 30, 31], norm-conserving Hamann
pseudopotentials [32, 33] and the local density approximation as parameterized
by Perdew & Wang [34]. Other simulation details and preliminary tests are
presented in the supplemental material [35].

We begin with a polydomain and free-standing PbO terminated PTO film seven
unit cells in thickness, with a 20 Å vacuum region in the out-of-plane
direction to prevent interactions between film images. An epitaxial strain of
-$1.2\%$ (relative to the the in-plane lattice parameters of bulk $P4mm$ PTO)
is imparted to represent the experimental PTO/STO lattice constant mismatch:
the conditions where DW alignment has been experimentally observed [1, 2, 3,
4, 5]. Explicit STO substrate is not used as the incipient broken inversion
symmetry in the out-of-plane directions is known to give rise to sizeable
perturbations to the domain structure [36, 20] which we wish to isolate from
our analysis. The equilibrium flux-closure domain period $\Lambda$ is used,
which we find to be $12$ unit cells. One unit cell wide STs are placed on both
surfaces of the film to (i) once again, limit the effect of inversion symmetry
breaking and (ii) to eliminate spurious electric fields resultant from
inequivalent surface dipole densities (otherwise requiring approximate
correction [37]). STs are laterally separated by 12 unit cells in one of three
positions: over the DC running parallel to the DW ($\parallel^{\text{DC}}$,
Fig. 1(a)), over the DW running parallel to the DW ($\parallel^{\text{DW}}$,
Fig. 1(b)) or running perpendicular to the DW (Fig. (1(c)). This last
orientation features antiferrodistortive (AFD) modes since the even in-plane
periodicity, broken surface symmetry and PbO termination will always invoke
the AFD $c(2\times 2)$ surface reconstruction [38] (enhanced surface
$a^{0}a^{0}c^{-}$ octahedral rotations). We label these films $\perp$ \+ AFD.
For fair comparison, we double the in-plane periodicity of
$\parallel^{\text{DW/DC}}$ to additionally consider $\parallel^{\text{DW/DC}}$
\+ AFD films. We are then able to evaluate the impact of AFD modes on the DW
alignment mechanism.

Each film listed so far is treated with ST depths of $d=1,2$ and $3$ unit
cells with PbO terminated trench floors (shown in Fig. 1(d)). To investigate
the effects of lateral interactions between STs, we perform two further
simulations of the $d=1$, $\parallel^{\text{DW/DC}}$ films with STs separated
by $4\Lambda$ ($\parallel^{\text{DW/DC}}_{4\Lambda}$). This sufficiently
limits the overlap of ST-induced surface strain fields. We relax each
structure until the magnitude of the force on each atom falls below $0.01$
eV/Å.

Figure 2: Local polarization vector fields calculated using the linear
approximation first noted by Resta [39]. Atom coloring is shared with Fig. 1
but O is removed for clarity. (a) At the DC of the pristine film. The Pb site
in the grey box is used in the scale bar positioned below (d). (b) At the DC
of the $d=1$ $\parallel^{\text{\text{DC}}}$ film. The purple dashed box is
discussed in the text. (c) At the DW of the pristine film. (d) At the DW of
the $d=1$ $\parallel^{\text{\text{DW}}}$ film. The orange dashed box is
discussed in the text. (e) A single polar vortex of the $d=2$ $\perp$ \+ AFD
film from the region colored on (f). (g) and (h) depict strings of Ti-centered
local polarizations along the [010] direction from the color-matched dashed
circles on (e).

The relative stability of each film is given in Table I. It is clear
immediately that the most stable ST arrangement treated in this letter is
$\parallel^{\text{DC}}$ \+ AFD, true for $d=1,2$ and $3$ supporting the
results of an effective Hamiltonian study [18]. The hierarchy of stabilities
is preserved independently of the treated $d$. In decreasing order we have
$\parallel^{\text{\text{DC}}}$ \+ AFD, $\parallel^{\text{DW}}$ \+ AFD, $\perp$
\+ AFD and then the films without AFD modes: $\parallel^{\text{DC}}$ and
$\parallel^{\text{DC}}$. This hierarchy demonstrates that parallel DW-ST
alignment is favoured, in good agreement with experimental observations [1, 2,
3, 4, 5]. For STs parallel to the DW, we see that part of this hierarchy is
preserved independently of AFD modes. While this strongly implies that AFD
modes play little-to-no role in the DW alignment mechanism, we do see that AFD
modes greatly lower the total energy and lead to strong antiphase tilts (of
$\approx 12^{\circ}$) at the surface which locally suppress polar modes in
agreement with a study using a similar method [20]. Remarkably, these modes
persist at a similar magnitude at the trench floor despite the broken
connectivity with the surface octahedral tilt network. The
$\parallel^{\text{DC}}_{4\Lambda}$ film is also more stable than
$\parallel^{\text{DW}}_{4\Lambda}$ by 0.359 eV/1760 atoms; part of the
hierarchy is unaffected by larger lateral ST separations, consistent with DW
alignment observed for isolated defects [4]. It is clear now that the DW
alignment mechanism is robust to the conditions tested in this letter.

Table 1: Film stabilities relative to $\parallel^{\text{DC}}$ \+ AFD per film
unit (FU): the number of atoms in the $\parallel$ arrangements. For $d=1,2$
and $3$, this is the energy per 428, 412 and 396 atoms, respectively.

| E - E($\parallel^{\text{DC}}$ \+ AFD) [eV/FU]
---|---
|  $\parallel^{\text{DC}}$ \+ AFD |  $\parallel^{\text{DW}}$ \+ AFD |  $\perp$ \+ AFD | $\parallel^{\text{DC}}$ | $\parallel^{\text{DW}}$
$d=1$ | 0 | +0.219 | +1.706 | +3.852 | +4.106
$d=2$ | 0 | +0.361 | +0.881 | +4.089 | +4.095
$d=3$ | 0 | +0.141 | +1.951 | +4.639 | +4.878

The stability of the different films can be best understood by considering how
the polar texture adapts to a trench. That is, regardless of orientation, a
trench must introduce new discontinuities to the polar texture of a pristine
film (the film without STs) and therefore new depolarizing fields. The most
stable arrangement must then be the one which best restores polar continuity,
thus minimizing ST-induced depolarizing fields. Using this principle, the
favorability of $\parallel^{\text{DC}}$ over $\parallel^{\text{DW}}$ can be
understood clearly from the local polarization vector fields shown in Fig.
2(a-d). Independent of depth, we find that the local polarization near the DC
for the $\parallel^{\text{DC}}$ film (Fig. 2(b)) is barely modulated in
comparison to the DC of the pristine film (Fig. 2(a)). This is made clear by
the two Ti-centered polarization vectors in the purple dashed box of Fig. 2(b)
which reproduce the two surface modes present for the pristine film in the
region for which the ST was inserted. This is because we have removed a unit
cell at a site of preexisting out-of-plane polar discontinuity. The resulting
polar texture arises from minimizing similar depolarizing fields to the
pristine film. This contrasts greatly with the $\parallel^{\text{DW}}$ films
(Fig. 2(d)) where there are large differences in the local polarization
compared with the pristine film (Fig. 2(c)). In this case, the trench removes
material in a region of continuous in-plane polarization at the cap of a polar
vortex. This creates a new and large discontinuity in the polarization, giving
rise to new in-plane depolarizing fields which are minimized by rotations of
the local polarization at the trench edges towards [010] (or [0$\bar{1}$0])
and [00$\bar{1}$]. While this minimization increases the stability of the
film, the resulting polar texture sees the two Ti-centered local polar modes
in the orange box of Fig. 2(d) stuck between a rock and a hard place. They are
arranged in an electrostatically unfavorable near-head-to-head configuration.
To compensate for this, the mean out-of-plane polarization of
$\parallel^{\text{DW}}$ films are reduced by $\approx 5\%$ compared with
$\parallel^{\text{DC}}$ films. All of these effects lower the stability of
$\parallel^{\text{DW}}$ films.

The low stability of $\perp$ \+ AFD films can also be understood using the
local polarization vector fields. This geometry is equivalent to interfacing
an axial slice of the flux-closure domain structure of a thinner film with a
thicker one. For example, at $d=2$, a single unit cell wide and 3 unit cell
thick axial slice is inserted (STs are present on both surfaces). This slice
has weaker out-of-plane polarization than the surrounding 7 unit cell thick
film; an experimentally observed effect for thinner pristine films [2]. In
spite of this, the mean out-of-plane polarization for the entire film is
greater than $\parallel^{\text{DW/DC}}$ \+ AFD films. The same is not true for
the axial polarization (along [010], the Bloch components [40]). These become
highly discontinuous giving rise to new depolarizing fields which are
compensated for by strongly reducing the axial polarization. Discontinuity
perpendicular to the direction of the polarization has striking effects. Fig.
2(e) shows strong rotations of the local polarization in the (010) plane along
the [010] direction within a single polar vortex (highlighted on Fig. 2(f)).
Fig. 2(g) shows a sharp rotation as we approach the trench edge while Fig.
2(h) shows a smooth rotation in order to create a continuous interface between
the different flux-closure domains of the 7 and 3 unit cell thick regions.
Although these modulations are cycloidal, close inspection reveals that the
films remain achiral. However, degenerate chiral and achiral phases have been
predicted in the PTO/STO system relating to the direction of the axial
polarization for each polar vortex [40]. Since $\perp$ \+ AFD films feature
suppressed axial polarization, these films actually suppress chirality.
Further, these modulations give rise to inhomogeneous strain and strain
gradients which interact with and disturb the underlying strain field of the
pristine film. The prediction of trench-induced cycloidal modulations
highlights that STs could be used to engineer exotic polar textures in
ferroelectric nanostructures. All of these effects compounded with one another
see the $\perp$ \+ AFD films most severely modulated from the pristine film
affirming their low stability compared with $\parallel^{\text{DW/DC}}$ \+ AFD
films.

Figure 3: Out-of-plane surface strain $\epsilon_{33}^{\text{surf}}$ along
[100] for: a) the $d=1$, $\parallel^{\text{DC}}_{4\Lambda}$ film and b) the
$d=1$, $\parallel^{\text{DW}}_{4\Lambda}$ film. Pristine film strain is shown
for means of comparison. Up domains are colored light grey, down domains are
colored a darker grey and the trench site is colored in red.

Motivated by the experimental observation of defect-induced out-of-plane
negative strains [4], we examined this effect within our simulations. We
define the out-of-plane surface strain $\epsilon_{33}^{\text{surf}}$ relative
to the average out-of-plane lattice parameter of the pristine film (4.051 Å).
Negative strains emerge for all films within $\approx 4-6$ unit cells of the
ST, weakly depending on $d$ apart from $d=3$ which gives rise to large
negative strains further from the ST which we postulate is a fictitious
artefact of the supercell geometry (the vertical trench-trench interaction is
too strong as at $d=3$ they become separated by only a single unit cell). When
increasing the ST separation to $4\Lambda$, we observe no change in the range
or character of the strain field. These fields are shown in Fig. 3, overlaid
with the underlying strain field of the pristine film. For the pristine film,
we see that the strain fields of the upper and lower surfaces are a pair of
sinusoids in antiphase with asymmetrical antinodes. The origin of this
asymmetry is beyond the scope of this letter but will be addressed in a
forthcoming publication. We see in Fig. 3(a) that near the ST, the ST-induced
negative strain cooperates with the negative strain of the pristine film at
the bottom surface (where polarization is directed along [001]) of
$\parallel^{\text{DC}}_{4\Lambda}$. The positive strain of the pristine film
is cancelled on the upper surface. Fig. 3(b) shows that
$\parallel^{\text{DW}}_{4\Lambda}$ brings slightly longer-range disorder to
the surface strain field than $\parallel^{\text{DW}}_{4\Lambda}$. While these
effects could contribute to the DW alignment mechanism (i.e. piezoelectric
and/or flexoelectric contributions), our method does not allow us to separate
these contributions from changes in the polarization emerging from the
aforementioned depolarizing field minimization. We remark that direct
comparison of our results with ref. [4] are limited by the drastically
different length scales. In [4], defects extend over many domain periods and
are less uniform than this study. It is then feasible that different
mechanisms could begin to contribute. That being said, in PTO/STO systems, the
in-plane axial components have been observed with much longer periods, robust
throughout entire samples [40]. Then, for larger surface defects, while it may
not be possible to discriminate between $\parallel^{\text{DC}}$ and
$\parallel^{\text{DW}}$ arrangements, perpendicular-to-wall geometries can be
precluded. So long as the defect sufficiently disturbs the axial components,
the resulting depolarizing field penalty can be large enough to prevent the
condensation of perpendicular-to-wall geometries and thus the ST-DW parallel
alignment phenomenon is still observed.

In summary, we have investigated the mechanism for experimentally observed
parallel DW-ST alignment with state-of-the art DFT simulations using as many
as 5,136 atoms. We assert that although the mechanism could have many
contributing components, it can be satisfactorily explained by how well the
polar texture of a film can adapt to a ST with a given orientation relative to
the DW. The preferred orientation features dipole moments in the vicinity of
the ST in an arrangement where energy penalties from ST-induced depolarizing
fields are most reduced. Subsequently, the preferred polar texture best
restores polar continuity and resembles the polar texture of the pristine
film. These conditions hold true for the $\parallel^{\text{DC}}$ films which
we find to be the most energetically stable arrangement, independent of the
presence of AFD modes, larger lateral ST separations and three depths (1, 2
and 3 unit cells). We find large negative strains in the vicinity of our STs
for all films. While it is possible that this contributes to the alignment
mechanism via piezoelectric and/or flexoelectric effects, we cannot decouple
these from other effects within our method. ST-DW parallel alignment is only
one consequence of ST engineering. Other exotic polar textures could be
engineered in a wide range of ferroelectric nanostructures as demonstrated by
the appearance of ST-induced polar cycloidal modulations in $\perp$ \+ AFD
films. Finally, we remark that the system sizes treated within this letter
greatly surpass standard ab initio studies. This demonstrates the power of
$\mathcal{O}(N)$ DFT methods [23, 24, 25] which we expect to become widely
adopted.

## Acknowledgements

We are grateful for use of the ARCHER/ARCHER2 UK National Supercomputing
Service funded by the UKCP consortium EPSRC Grant Ref. No. EP/P022561/1. The
authors also thank Pavlo Zubko and Marios Hadjimichael for their careful
review this letter.

## References

  * Streiffer  _et al._ [2002] S. K. Streiffer, J. A. Eastman, D. D. Fong, C. Thompson, A. Munkholm, M. V. R. Murty, O. Auciello, G. R. Bai, and G. B. Stephenson, Phys. Rev. Lett. 89, 067601 (2002).
  * Fong  _et al._ [2004] D. D. Fong, G. B. Stephenson, S. K. Streiffer, J. A. Eastman, O. Auciello, P. H. Fuoss, and C. Thompson, Science 304, 1650 (2004).
  * Thompson  _et al._ [2008] C. Thompson, D. D. Fong, R. V. Wang, F. Jiang, S. K. Streiffer, K. Latifi, J. A. Eastman, P. H. Fuoss, and G. B. Stephenson, Appl. Phys. Lett. 93, 182901 (2008).
  * Hadjimichael  _et al._ [2018] M. Hadjimichael, E. Zatterin, S. Fernandez-Peña, S. J. Leake, and P. Zubko, Phys. Rev. Lett. 120, 037602 (2018).
  * Park  _et al._ [2018] J. Park, J. Mangeri, Q. Zhang, M. H. Yusuf, A. Pateras, M. Dawber, M. V. Holt, O. G. Heinonen, S. Nakhmanson, and P. G. Evans, Nanoscale 10, 3262 (2018).
  * Seidel  _et al._ [2009] J. Seidel, L. W. Martin, Q. He, Q. Zhan, Y.-H. Chu, A. Rother, M. E. Hawkridge, P. Maksymovych, P. Yu, M. Gajek, N. Balke, S. V. Kalinin, S. Gemming, F. Wang, G. Catalan, _et al._ , Nat. Mater. 8, 229 (2009).
  * Hong  _et al._ [2021] Z. Hong, S. Das, C. Nelson, A. Yadav, Y. Wu, J. Junquera, L.-Q. Chen, L. W. Martin, and R. Ramesh, Nano Lett. (2021).
  * Das  _et al._ [2020] S. Das, Z. Hong, V. A. Stoica, M. A. P. Gonçalves, Y. T. Shao, E. Parsonnet, E. J. Marksz, S. Saremi, M. R. McCarter, A. Reynoso, C. J. Long, A. M. Hagerstrom, D. Meyers, V. Ravi, B. Prasad, H. Zhou, _et al._ , Nat. Mater. (2020).
  * Whyte and Gregg [2015] J. Whyte and J. Gregg, Nat. Commun. 6, 7361 (2015).
  * McConville  _et al._ [2020] J. P. V. McConville, H. Lu, B. Wang, Y. Tan, C. Cochard, M. Conroy, K. Moore, A. Harvey, U. Bangert, L.-Q. Chen, A. Gruverman, and J. M. Gregg, Adv. Funct. Mater. 30, 2000109 (2020).
  * Sharma  _et al._ [2017] P. Sharma, Q. Zhang, D. Sando, C. H. Lei, Y. Liu, J. Li, V. Nagarajan, and J. Seidel, Sci. Adv. 3, e1700512 (2017).
  * Jiang  _et al._ [2017] J. Jiang, Z. L. Bai, Z. H. Chen, L. He, D. W. Zhang, Q. H. Zhang, J. A. Shi, M. H. Park, J. F. Scott, C. S. Hwang, and A. Q. Jiang, Nat. Mater. 17, 49 (2017).
  * Maksymovych  _et al._ [2011] P. Maksymovych, A. N. Morozovska, P. Yu, E. A. Eliseev, Y.-H. Chu, R. Ramesh, A. P. Baddorf, and S. V. Kalinin, Nano Lett. 12, 209 (2011).
  * Catalan  _et al._ [2012] G. Catalan, J. Seidel, R. Ramesh, and J. F. Scott, Rev. Mod. Phys. 84, 119 (2012).
  * Whyte  _et al._ [2013] J. R. Whyte, R. G. P. McQuaid, P. Sharma, C. Canalias, J. F. Scott, A. Gruverman, and J. M. Gregg, Adv. Mater. 26, 293 (2013).
  * Werner  _et al._ [2017] C. S. Werner, S. J. Herr, K. Buse, B. Sturman, E. Soergel, C. Razzaghi, and I. Breunig, Sci. Rep. 7, 9862 (2017).
  * Shimada  _et al._ [2010a] T. Shimada, S. Tomoda, and T. Kitamura, J. Phys. Condens. Matter 22, 355901 (2010a).
  * Prosandeev and Bellaiche [2007] S. Prosandeev and L. Bellaiche, Appl. Phys. Lett. 91, 072901 (2007).
  * Tang  _et al._ [2015] Y. L. Tang, Y. L. Zhu, X. L. Ma, A. Y. Borisevich, A. N. Morozovska, E. A. Eliseev, W. Y. Wang, Y. J. Wang, Y. B. Xu, Z. D. Zhang, and S. J. Pennycook, Science 348, 547 (2015).
  * Baker and Bowler [2020] J. S. Baker and D. R. Bowler, Adv. Theory Simul. 3, 2000154 (2020).
  * Bowler and Miyazaki [2012] D. R. Bowler and T. Miyazaki, Rep. Prog. Phys. 75, 036503 (2012).
  * Goedecker [1999] S. Goedecker, Rev. Mod. Phys. 71, 1085 (1999).
  * Li  _et al._ [1993] X.-P. Li, R. W. Nunes, and D. Vanderbilt, Phys. Rev. B 47, 10891 (1993).
  * Palser and Manolopoulos [1998] A. H. R. Palser and D. E. Manolopoulos, Phys. Rev. B 58, 12704 (1998).
  * Bowler and Gillan [1999] D. Bowler and M. Gillan, Comput. Phys. Commun. 120, 95 (1999).
  * Bowler  _et al._ [2006] D. R. Bowler, R. Choudhury, M. J. Gillan, and T. Miyazaki, Phys. Status Solidi B 243, 989 (2006).
  * Nakata  _et al._ [2020] A. Nakata, J. S. Baker, S. Y. Mujahed, J. T. L. Poulton, S. Arapan, J. Lin, Z. Raza, S. Yadav, L. Truflandier, T. Miyazaki, and D. R. Bowler, J. Chem. Phys. 152, 164112 (2020).
  * Zen [2020] CONQUEST v1.0.5 (2020).
  * Torralba  _et al._ [2008] A. S. Torralba, M. Todorović, V. Brázdová, R. Choudhury, T. Miyazaki, M. J. Gillan, and D. R. Bowler, J. Phys. Condens. Matter 20, 294206 (2008).
  * Bowler  _et al._ [2019] D. R. Bowler, J. S. Baker, J. T. L. Poulton, S. Y. Mujahed, J. Lin, S. Yadav, Z. Raza, and T. Miyazaki, Jpn. J. Appl. Phys 58, 100503 (2019).
  * Baker  _et al._ [2020] J. S. Baker, T. Miyazaki, and D. R. Bowler, Electron. Struct. 2, 025002 (2020).
  * Hamann [2013] D. R. Hamann, Phys. Rev. B 88, 085117 (2013).
  * van Setten  _et al._ [2018] M. van Setten, M. Giantomassi, E. Bousquet, M. Verstraete, D. Hamann, X. Gonze, and G.-M. Rignanese, Comput. Phys. Commun. 226, 39 (2018).
  * Perdew and Wang [1992] J. P. Perdew and Y. Wang, Phys. Rev. B 45, 13244 (1992).
  * [35] See Supplemental Material [publisher url] for finer details of the simulation method and required preliminary tests/studies used to enable the main article, which includes Refs. [42, 43, 44, 45, 46, 47, 48, 49]. Additional raw data including basis sets and example inputs are hosted at https://doi.org/10.5281/zenodo.4699823.
  * Sai  _et al._ [2000] N. Sai, B. Meyer, and D. Vanderbilt, Phys. Rev. Lett. 84, 5636 (2000).
  * Bengtsson [1999] L. Bengtsson, Phys. Rev. B 59, 12301 (1999).
  * Munkholm  _et al._ [2001] A. Munkholm, S. K. Streiffer, M. V. R. Murty, J. A. Eastman, C. Thompson, O. Auciello, L. Thompson, J. F. Moore, and G. B. Stephenson, Phys. Rev. Lett. 88, 016101 (2001).
  * Resta  _et al._ [1993] R. Resta, M. Posternak, and A. Baldereschi, Phys. Rev. Lett. 70, 1010 (1993).
  * Shafer  _et al._ [2018] P. Shafer, P. García-Fernández, P. Aguado-Puente, A. R. Damodaran, A. K. Yadav, C. T. Nelson, S.-L. Hsu, J. C. Wojdeł, J. Íñiguez, L. W. Martin, E. Arenholz, J. Junquera, and R. Ramesh, PNAS 115, 915 (2018).
  * Shimada  _et al._ [2010b] T. Shimada, S. Tomoda, and T. Kitamura, Phys. Rev. B 81, 144116 (2010b).
  * Gonze  _et al._ [2016] X. Gonze, F. Jollet, F. A. Araujo, D. Adams, B. Amadon, T. Applencourt, C. Audouze, J.-M. Beuken, J. Bieder, A. Bokhanchuk, _et al._ , Comput. Phys. Commun. 205, 106 (2016).
  * Gonze  _et al._ [2009] X. Gonze, B. Amadon, P.-M. Anglade, J.-M. Beuken, F. Bottin, P. Boulanger, F. Bruneval, D. Caliste, R. Caracas, M. Côté, _et al._ , Comput. Phys. Commun. 180, 2582 (2009).
  * Bowler  _et al._ [2020] D. R. Bowler, J. S. Baker, J. T. L. Poulton, S. Y. Mujahed, L. Truflandier, S. Yadav, Z. Raza, T. Miyazaki, M. Gillan, A. Nakata, _et al._ , The CONQUEST code: public release at https://github.com/OrderN/CONQUEST-release (2020).
  * Zhang  _et al._ [2017] Y. Zhang, J. Sun, J. P. Perdew, and X. Wu, Phys. Rev. B 96, 035143 (2017).
  * Vanderbilt [1990] D. Vanderbilt, Phys. Rev. B 41, 7892 (1990).
  * Hotelling [1943] H. Hotelling, Ann. Math. Stat. 14, 1 (1943).
  * Aguado-Puente and Junquera [2012] P. Aguado-Puente and J. Junquera, Phys. Rev. B 85, 184105 (2012).
  * Pulay [1980] P. Pulay, Chem. Phys. Lett 73, 393 (1980).