# State-Output Risk-Constrained Quadratic Control
of Partially Observed Linear Systems

Nikolas Koumpis⋆, Anastasios Tsiamis† and Dionysios Kalogerias⋆ ⋆ Department
of Electrical Engineering, Yale University, New Haven, CT 06511 (email:
{nikolaos.koumpis, dionysis.kalogerias}@yale.edu). † Automatic Control
Laboratory, ETH Zürich, 8092, Switzerland (email:
atsiamis@control.ee.ethz.ch).

###### Abstract

We propose a methodology for performing risk-averse quadratic regulation of
partially observed Linear Time-Invariant (LTI) systems, disturbed by process
and output noise. To compensate against the induced variability due to both
types of noises, state regulation is subject to two risk constraints. The
latter render the resulting controller to be cautious of stochastic
disturbances, by restricting the statistical variability, namely, a simplified
version of the cumulative expected predictive variance, of both the _state_
and the _output_. It turns out that our proposed formulation results in an
optimal risk-averse policy that preserves favorable characteristics of the
classical Linear Quadratic (LQ) control. In particular, the optimal policy has
an affine structure with respect to the minimum mean square error (mmse)
estimates. The linear component of the policy regulates the state more
strictly in riskier directions, where the process and output noise covariance,
cross-covariance, and the corresponding penalties are simultaneously large.
This is achieved by “inflating" the state penalty in a systematic way. The
additional affine terms force the state against pure and cross third-order
statistics of the process and output disturbances. Another favorable
characteristic of our optimal policy is that it can be pre-computed off-line,
thus, avoiding limitations of prior work. Stability analysis shows that the
derived controller is always internally stable regardless of parameter tuning.
The functionality of the proposed risk-averse policy is illustrated through a
working example via extensive numerical simulations.

##  1 Introduction

Decision policies designed to be optimal on average are often inadequate in
many practical applications, especially where the system state transitions are
subject to unexpected, less probable, though with possibly catastrophic
consequences, random events. Critical applications where such risky events
should be accounted for are especially pronounced in many areas including
robotics [1, 2], wireless communications [3], networking [4, 5], control [6],
formal methods [7], health [8], and finance [9] to name a few. A major
challenge that arises is that we not only have to deal with stochastic
environments, but also with imperfect information, partial observability, and
noisy measurements. Although in some cases we can theoretically characterize
optimal risk-aware policies for general partially-observed systems, their
implementation can be really hard if not impossible [10]. In this paper, we
are interested in _implementable_ risk-averse control policies that counteract
against those less probable though catastrophic events for both the states and
the outputs, at the cost of slightly sacrificing performance on average.
Recently, a risk-constrained formulation for Linear Quadratic (LQ) control of
partially-observed linear systems was introduced in [10]. While the optimal
risk-averse policy of [10] can be characterized, it is only implementable in
the two simple settings of Gaussian noise and full state-observations. In the
non-Gaussian case, implementing the optimal policy, requires online
computation of a filtered version of the Riccati difference equation. Such a
computation is hard if not impossible to perform. Here, inspired by [10], we
aim to find a control policy that can be implemented efficiently while it
retains its risk-aversion properties. Our contributions are the following.

State and output risk constraints. We propose novel risk constraints, where we
restrict the variability of both the state and the output. In particular, we
restrict the cumulative expected predictive variance of the state and output,
where the prediction is conditioned on the previous state. By adjusting the
constraint specifications, we can control the tradeoff between average LQ cost
and statistical variability due to both process and output uncertainties.
State predictive variance constraints were also studied in [10], where the
prediction was conditioned on the past output information. Here, the
prediction is conditioned on the previous state, i.e. it discounts the
uncertainty of the estimation error. However, this allows us to obtain a
tractable and implementable risk-averse controller. In particular, the optimal
policy here can be pre-computed _off-line_ based only on second and third
order noise-statistics, overcoming the limitations of [10].
Optimal Risk-Aware policy & Stability. We show that the proposed risk-averse
formulation results in a twice quadratically constrained LQ problem, which
admits a closed-form solution. The optimal risk-averse strategy is affine with
respect to the state estimates, with the affine terms directly repelling the
pure and cross third order noise-statistics. Further, an inflated state
penalty matrix amplifies regulation by over-measuring directions where process
and output noise present (jointly and each separately) high risk.
Arbitrary noise model. Our results are applicable for all skew-symmetric,
heavy tailed, and skewed (process and/or output) models provided that the
corresponding fourth order moments remain finite. This is in contrast to the
classical Linear Exponential Quadratic Gaussian control [11], where the noise
distributions have to be Gaussian.
Separation. We show that separation holds, in the sense that the optimal
policy is affine with respect to the minimum mean square error (mmse)
estimate. Hence, the filter and the controller can be designed separately.
Computing efficiently the mmse estimate in the non-Gaussian is a very hard
problem of independent interest and outside of the scope of this paper. In the
simulations we use the Kalman Filter, which, despite being suboptimal, is the
best linear filter [12].

Related work. Having its origins in mathmatical finance and operations
research, risk-averse optimization has been lately naturally emerging in many
applications, and has been considered in a variety of contexts [13, 8, 14, 15,
16, 17, 18, 19, 20, 21, 22]. The main paradigm shift in risk-averse
optimization is to replace expectations by more general risk measures [23], as
an attempt to more effectively capture the tail behavior of the involved
random cost function, on top of or trading with mean performance. Typical
examples are mean-variance functionals [24, 23], mean-semideviations [16], and
Conditional Value-at-Risk (CVaR) [25]. In particular for the case of control
systems, apart form the well-known and studied exponential quadratic approach,
CVaR optimization techniques have also been considered for risk-aware
constraint satisfaction [22]. Although CVaR captures variability and tail
events well, CVaR optimization problems rarely admit closed-form expressions
[22], [26].

The predictive variance of the stage cost was introduced in [27], [28] for
risk-aware quadratic state regulation in the fully observable case and
subsequently it was applied in [10] for partially observed systems. Although
both formulations characterize the resulting policy in closed form, the latter
ends up being dependent on filtered information and as such it can be pre-
computed off-line only when both the process and the measurement noise are
Gaussian. Finally, a closely related problem is the celebrated Linear
Exponential Quadratic Gaussian (LEQG) control problem [11, 29]. Although it
admits a closed-from solution, it also requires Gaussian noise, which does not
capture distributions with asymmetric (skewed) structure. Moreover, tuning the
exponential parameter can be challenging, since certain values above a
threshold lead to unstable controllers (neurotic breakdown).

Notation. With $\mu_{s,o}$ we refer to the pair $(\mu_{s},\mu_{o})$, and by
$\mu_{s,o}>0$ we declare that both $\mu_{s}$ and $\mu_{o}$ are positive. We
denote a sequence of arrays $x_{0},\dots,x_{k}$, as $x_{0:k}$, and lastly the
$\sigma-$algebra generated by a random array $x$ as $\sigma(x)$. We use
$||x||^{2}_{Q}$ to denote the $Q-$weighted norm of $x\in\mathbb{R}^{n}$, with
the matrix $Q\in\mathbb{R}^{n\times{n}}$ being positive (semi) definite.

##  2 Risk-constrained Linear Quadratic formulation with partial observations

Consider a dynamical system described by the linear difference equation

$x_{t+1}=Ax_{t}+Bu_{t}+w_{t+1},$ (1)

with its output given by

$y_{t}=Cx_{t}+\epsilon_{t}.$ (2)

Here $x_{t}\in\mathbb{R}^{n}$ is the hidden state of the system,
$u_{t}\in\mathbb{R}^{m}$ is the control input, and $y_{t}\in\mathbb{R}^{q}$
are the outputs/observations of the system. In our study, the model
uncertainty $w_{t}$ and the output noise $\epsilon_{t}$ are assumed i.i.d
stochastic sequences, and not necessarily Gaussian, with
$\mathbb{E}\\{w_{t}\\}=\bar{w}$,
$\mathbb{E}\\{\epsilon_{t}\\}=\bar{\epsilon}$,
$\mathbb{E}\\{(w_{t}-\bar{w})(w_{t}-\bar{w})^{\top}\\}=W$, and
$\mathbb{E}\\{(\epsilon_{t}-\bar{\epsilon})(\epsilon_{t}-\bar{\epsilon})^{\top}\\}=E$,
$\mathbb{E}\big{\\{}(w_{t}-\bar{w})(\epsilon_{t}-\bar{\epsilon})^{\top}\big{\\}}=H$.
Let $\mathscr{F}_{t}\triangleq\sigma(y_{0:t},u_{0:t-1})$ be the sigma algebra
generated by all the observable quantities up to time $t$. Based on the above
notation, the state estimate and state prediction at time $t$ respectively
read

$\displaystyle\widehat{x}_{t|t-1}$
$\displaystyle=\mathbb{E}\\{x_{t}|\mathscr{F}_{t-1}\\},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \widehat{x}_{t|t}=\mathbb{E}\\{x_{t}|\mathscr{F}_{t}\\},$
(3)

with corresponding errors

$\displaystyle\rho_{t}$
$\displaystyle={x}_{t}-\widehat{x}_{t|t-1},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ e_{t}={x}_{t}-\widehat{x}_{t|t},$
(4)

where at each time we expect zero for both, i.e.,
$\mathbb{E}\\{e_{t}|\mathscr{F}_{t-1}\\}=\mathbb{E}\\{\rho_{t}|\mathscr{F}_{t-1}\\}=0$.
Lastly, we also define

$\displaystyle\Sigma_{t|t}$
$\displaystyle=\mathbb{E}\\{e_{t}e^{\top}_{t}|\mathscr{F}_{t}\\},$ (5)
$\displaystyle\Sigma_{t|t-1}$
$\displaystyle=\mathbb{E}\\{e_{t}e^{\top}_{t}|\mathscr{F}_{t-1}\\},$ (6)
$\displaystyle H_{t|t-1}$
$\displaystyle=\mathbb{E}\\{e_{t}\rho^{\top}_{t}|\mathscr{F}_{t-1}\\}.$ (7)

Throughout this paper, we assume that both the process and output noises have
finite fourth moments.

######  Assumption 1 (Noise Regularity).

The processes $w_{t}$, and $\epsilon_{t}$ have finite fourth-order moments,
i.e., for every $t\in\mathbb{N}$, $\mathbb{E}\\{||w_{t}||^{4}_{2}\\}<\infty$,
and $\mathbb{E}\\{||\epsilon_{t}||^{4}_{2}\\}<\infty$.

The above assumption guarantees that our risk-averse LQ control problem,
defined below, is well-posed. It covers general, potentially non-Gaussian,
distributions. Before we state the main problem, let us define the “extended"
sigma algebra
$\overline{\mathscr{F}}_{t}\triangleq\sigma(y_{0:t},u_{0,t-1},x_{0:t})$, which
is the sigma algebra of the observable quantities plus the hidden states. Let
also $\mathcal{L}_{2}(\mathscr{F}_{t})$ be the space of square integrable
vectors, which are $\mathscr{F}_{t}-$measurable. We pose the following
problem:

$\displaystyle J(u)=$
$\displaystyle\min_{u}\mathbb{E}\left\\{||x_{N}||^{2}_{Q}+\sum_{t=0}^{N-1}||x_{t}||^{2}_{Q}+||u_{t}||^{2}_{R}\right\\}$
$\displaystyle\text{ s.t. }J_{s}(u)=$
$\displaystyle\mathbb{E}\left\\{\sum_{t=1}^{N}\left[||x_{t}||^{2}_{Q_{s}}{-}\mathbb{E}\left(||x_{t}||^{2}_{Q_{s}}\mid\overline{\mathscr{F}}_{t-1}\right)\right]^{2}\right\\}\leq\epsilon_{s},$
$\displaystyle J_{o}(u)=$
$\displaystyle\mathbb{E}\left\\{\sum_{t=1}^{N}\left[||y_{t}||^{2}_{Q_{o}}{-}\mathbb{E}\left(||y_{t}||^{2}_{Q_{o}}\mid\overline{\mathscr{F}}_{t-1}\right)\right]^{2}\right\\}\leq\epsilon_{o},$
$\displaystyle x_{t+1}=Ax_{t}+Bu_{t}+w_{t+1},$ $\displaystyle
y_{t}=Cx_{t}+\epsilon_{t},$ $\displaystyle
u_{t}\in\mathcal{L}_{2}\left(\mathscr{F}_{t}\right),t=0,\ldots,N-1.$ (8)

We impose constraints on the cumulative expected predictive variance of the
state and the output. This essentially forces the optimal controller to not
only optimize average performance but also reduce the _variability_ of the
state and the output. Note that we _discount_ the risk of the estimation error
$e_{t}$ in the predictive variance. This is why the prediction in the
constraints is conditioned on the _extended_ sigma algebra
$\overline{\mathscr{F}}_{t-1}$ and not $\mathscr{F}_{t-1}$. Taking the
estimation error into account, would result in an intractable optimal control
problem [10]. By discounting $e_{t}$, we sacrifice some risk-aversion to
obtain an implementable controller. However, we still account for the
immediate process and output noises. In any case, the control input only has
access to the partial observations $\mathscr{F}_{t-1}$ as captured by the
requirement $u_{t}\in\mathcal{L}_{2}(\mathscr{F}_{t})$. Finally, the positive
semi-definite weighting matrices $Q_{s},Q_{o}$ are some additional tuning
parameters that we can tweak, offering more flexibility to our LQ formulation.

##  3 Optimal Risk-Averse LQR Controllers

The derivation of the optimal risk-averse policy tracks [27] and it is
summarized in the following steps: We first show that (8) is well defined and
that it can be recast to a sequential twice quadratically constrained
quadratic program (TQCQP). Then, we utilize Lagrangian duality to solve (8) in
closed form.

######  Proposition 1 (TQCQP Reformulation).

Let Assumption 1 be in effect, and define

$\displaystyle p_{t}\triangleq{C}(w_{t}-\bar{w})+\epsilon_{t}-\bar{\epsilon},$

along with the higher order weighted statistics

$\displaystyle\mathrm{M}_{w}$
$\displaystyle\triangleq\mathbb{E}\Big{\\{}(w_{t}-\bar{w})(w_{t}-\bar{w})^{\top}Q_{s}(w_{t}-\bar{w})\Big{\\}},$
$\displaystyle\mathrm{M}_{\epsilon}$
$\displaystyle\triangleq\mathbb{E}\Big{\\{}(\epsilon_{t}-\bar{\epsilon})(\epsilon_{t}-\bar{\epsilon})^{\top}Q_{o}(\epsilon_{t}-\bar{\epsilon})\Big{\\}},$
$\displaystyle m_{w}$
$\displaystyle\triangleq\mathbb{E}\Big{\\{}\Big{(}(w_{t}-\bar{w})^{\top}Q_{s}(w_{t}-\bar{w})-\mathrm{tr}(Q_{s}W)\Big{)}^{2}\Big{\\}},$
$\displaystyle\mathrm{M}$
$\displaystyle\triangleq\mathbb{E}\Big{\\{}p_{t}p_{t}^{\top}Q_{o}p_{t}\Big{\\}},$
(9) $\displaystyle m_{w\epsilon}$
$\displaystyle\triangleq\mathbb{E}\Big{\\{}\Big{(}p_{t}^{\top}Q_{o}p_{t}-\mathrm{tr}(Q_{o}P)\Big{)}^{2}\Big{\\}}.$

Then, the risk-constrained LQ problem (8) is well-defined and equivalent to
the sequential variational TQCQP

$\displaystyle\min_{u}$ $\displaystyle\leavevmode\nobreak\
\mathbb{E}\left\\{x_{N}^{\top}Qx_{N}+\sum_{t=0}^{N-1}x_{t}^{\top}Qx_{t}+u_{t}^{\top}Ru_{t}\right\\}$
s.t. $\displaystyle\leavevmode\nobreak\
\mathbb{E}\left\\{\sum^{N}_{t=1}4x_{t}^{\top}Q_{s}WQ_{s}x_{t}+4x^{\top}_{t}Q_{s}\mathrm{M}_{w}\right\\}<\bar{\epsilon}_{s},$
$\displaystyle\leavevmode\nobreak\
\mathbb{E}\left\\{\sum^{N}_{t=1}4x^{\top}_{t}C^{\top}Q_{o}PQ_{o}Cx_{t}+4x^{\top}_{t}C^{\top}Q_{o}\mathrm{M}\right\\}<\bar{\epsilon}_{o},$
$\displaystyle x_{t+1}=Ax_{t}+Bu_{t}+w_{t+1},$ $\displaystyle
u_{t}\in\mathcal{L}_{2}\left(\mathscr{F}_{t}\right),t=0,\ldots,N-1,$ (10)

where

$\displaystyle P$ $\displaystyle{=}CWC^{\top}+CH+(HC)^{\top}+E,$
$\displaystyle\bar{\epsilon}_{s}$
$\displaystyle{=}\epsilon_{s}-Nm_{w}+4N(\mathrm{tr}(Q_{s}W))^{2},$
$\displaystyle\bar{\epsilon}_{o}$
$\displaystyle{=}\epsilon_{o}{-}Nm_{w\epsilon}{-}4N\bar{\epsilon}^{\top}Q_{o}\mathrm{M}{-}8N\mathrm{tr}(C^{\top}Q_{o}PQ_{o}CW)-4N\mathrm{tr}(Q_{o}PQ_{o}Z),$

and
$Z=\mathbb{E}\big{\\{}(\bar{\epsilon}-C\delta_{t})(\bar{\epsilon}-C\delta_{t})^{\top}\big{\\}}$.

The proof is given in the Appendix 6.1. Proposition 1 allows us to utilize
duality theory and thus solve (8) in closed form via dynamic programming.

###  3.1 Lagrangian Duality

To tackle problem (8), we consider the variational Lagrangian
$\mathcal{L}:\mathcal{L}_{2}(\mathscr{F}_{0})\times\dots\times\mathcal{L}_{2}(\mathscr{F}_{N-1})\times\mathbb{R}_{+}\times\mathbb{R}_{+}\rightarrow\mathbb{R}$
of the sequential TQCQP (10), defined as

$\displaystyle\mathcal{L}(u;\mu_{s,o})\triangleq
J(u)+\mu_{s}J_{s}(u)+\mu_{o}J_{o}(u)-\mu_{s}\bar{\epsilon}_{s}-\mu_{o}\bar{\epsilon}_{o},$
(11)

where $\mu_{s,o}\geq 0$ are multipliers associated with the variational risk
constraints of (10). The dual function
$\mathcal{D}:\mathbb{R}^{+}\times\mathbb{R}^{+}\rightarrow[-\infty,+\infty)$
of the primal problem (10) is defined as

$\displaystyle\mathcal{D}(\mu_{s,o})=\inf_{u\in\mathcal{U}}\mathcal{L}(u,\mu_{s,o}),$

where

$\displaystyle\mathcal{U}\triangleq\left\\{u_{0:N-1}\in\prod_{t=k}^{N-1}\mathcal{L}_{2}\left(\mathscr{F}_{t}\right)\Bigg{|}x_{t+1}=Ax_{t}+Bu_{t}+w_{t+1}\right\\}$

refers to the constraints that remain not dualized. The optimal value of the
always concave dual problem

$\displaystyle\sup_{\mu_{s,o}\geq
0}\mathcal{D}(\mu_{s,o})\equiv\sup_{\mu_{s,o}\geq
0}\inf_{u\in\mathcal{U}}\mathcal{L}(u,\mu_{s,o}),$ (12)

with $\mathcal{D}^{*}=\sup_{\mu_{s,o}\geq
0}\mathcal{D}(\mu_{s,o})\in[-\infty,\infty]$ is the tightest under-estimate of
the optimal value of the primal problem $J^{*}$, when knowing only
$\mathcal{D}$. Leveraging Lagrangian duality, we may now state the following
result, which provides sufficient optimality conditions for the TQCQP (10).

######  Theorem 1 (Optimality Conditions).

With Assumption 1 being in effect, suppose there exist a feasible policy-
multiplier pair
$(u^{*},\mu^{*}_{s,o})\in\mathcal{U}\times\mathbb{R}_{+}\times\mathbb{R}_{+}$
such that

  1. 1)

$\mathcal{L}(u^{*}(\mu^{*}_{s,o}),\mu^{*}_{s,o})=\min_{u\in\mathcal{U}}\mathcal{L}(u,\mu^{*}_{s,o})=\mathcal{D}(\mu^{*}_{s,o})$

  2. 2)

$J_{s}(u^{*})\leq\bar{\epsilon}_{s}$, and
$J_{o}(u^{*})\leq\bar{\epsilon}_{o}$.

  3. 3)

$\mu_{s}(J_{s}(u^{*})-\bar{\epsilon}_{s})=0$, and
$\mu_{o}(J_{o}(u^{*})-\bar{\epsilon}_{o})=0$.

Then, $u^{*}$ is optimal for the primal (10) and the initial problem (8),
$\mu_{s,o}$ are optimal for the dual problem (12), and (10) exhibits zero
duality gap.

The proof of Theorem 1 is omitted as it follows as direct application of
Theorem 4.10 in [30]. Within our context, Theorem 1 provides justification of
the Lagrangian relaxation approach we take hereafter in regard to solving
Problem (8). In particular, by choosing certain values for the multipliers
$\mu_{s,o}$, evaluation of the constraints of (8) determine respective values
for $\bar{\epsilon}_{s,o}$, which in most cases satisfy the conditions of
Theorem 1. In other words, Theorem 1 may be used ‘in reverse’, as a
verification device, or even to tabulate certain tolerance/multiplier pairs
which lead to good performance. Also note that, for every optimal solution of
the corresponding Lagrangian (11), the resulting constraint values can be
efficiently evaluated recursively using standard LQ theory, along the lines of
[10].

###  3.2 Optimal risk-averse control policies

Let $\mu_{s,o}>0$ be arbitrary but fixed. First, we may simplify the
Lagrangian $\mathcal{L}$ and express it within a canonical dynamic programming
framework. In this respect, we have the following result.

######  Lemma 1 (Lagrangian Reformulation).

For every $u_{t}\in\mathcal{L}_{2}(\mathscr{F}_{t})$, $t\leq{N-1}$, the
Lagrangian function $\mathcal{L}$ can be expressed as

$\displaystyle\mathcal{L}(u;\mu_{s,o})$
$\displaystyle{=}\mathbb{E}\left\\{{g_{N}(x_{N};\mu_{s,o})}{+}\sum_{t=0}^{N-1}g_{t}(x_{t},u_{t};\mu_{s,o})\right\\}{+}g_{\mu_{s,o}}$
(13) s.t. $\displaystyle x_{t+1}=Ax_{t}+Bu_{t}+w_{t+1},$ $\displaystyle
u_{t}\in\mathcal{L}_{2}\left(\mathscr{F}_{t}\right),t=0,\ldots,N-1,$

where

$\displaystyle g_{N}(x_{N};\mu_{s,o})$ $\displaystyle\triangleq
x_{N}^{\top}Q_{\mu_{s,o}}x_{N}+x^{\top}_{N}\mathrm{M}_{\mu_{s,o}},$
$\displaystyle g(x_{t},u_{t};\mu_{s,o})$ $\displaystyle\triangleq
x_{t}^{\top}Q_{\mu_{s,o}}x_{t}+x^{\top}_{t}\mathrm{M}_{\mu_{s,o}}+u^{\top}_{t}Ru_{t},$
$\displaystyle g_{0}(x_{0};\mu_{s,o})$
$\displaystyle\triangleq-x^{\top}_{0}(Q_{\mu_{s,o}}-Q)x_{0}-x^{\top}_{0}\mathrm{M}_{\mu_{s,o}},$

and

$\displaystyle Q_{\mu_{s,o}}$ $\displaystyle\triangleq
Q+4\mu_{s}Q_{s}WQ_{s}+4\mu_{o}C^{\top}Q_{o}PQ_{o}C$
$\displaystyle\mathrm{M}_{\mu_{s,o}}$ $\displaystyle\triangleq
4\mu_{s}Q_{s}\mathrm{M}_{w}+4\mu_{o}(C^{\top}Q_{o}\mathrm{M}+2C^{\top}Q_{o}PQ_{o}\bar{\epsilon}),$
(14)

where

$\displaystyle\mathrm{M}=\mathrm{M}_{w\epsilon}+\mathrm{M}_{\epsilon}$ (15)

###### Proof.

It follows from Proposition 1 and the form of $\mathcal{L}$. ∎

######  Theorem 2 (Optimal Risk-averse Controls).

Let Assumption 1 be in effect, and opt $\mu_{s,o}>0$. Then for all
$t\in\mathscr{I}$, $\mathscr{I}=\\{1,\dots,N\\}$, $N\in\mathbb{N}$, the
optimal cost-to-go $\mathcal{L}^{*}(\mathscr{F}_{t};\mu_{s,o})$ can be
expressed as

$\displaystyle\mathcal{L}^{*}(\mathscr{F}_{t};\mu_{s,o})$
$\displaystyle=\widehat{x}^{\top}_{t|t}V_{t}\widehat{x}_{t|t}+2\widehat{x}^{\top}_{t|t}T_{t}\bar{w}+\widehat{x}^{\top}_{t|t}S_{t}\mathrm{M}_{\mu_{s,o}}+p_{t}.$

The risk-averse optimal controller reads

$\displaystyle u^{*}_{t-1}(\mu_{s,o})$
$\displaystyle=K_{t-1}\widehat{x}_{t-1|t-1}+h_{t-1}+l_{t-1},$ (16)

where

$\displaystyle K_{t-1}$
$\displaystyle=-\left(B^{\top}V_{t}B+R\right)^{-1}B^{\top}V_{t}A,$
$\displaystyle h_{t-1}$
$\displaystyle=-\left(B^{\top}V_{t}B+R\right)^{-1}B^{\top}\left(V_{t}+T_{t}\right)\bar{w}$
$\displaystyle l_{t-1}$
$\displaystyle=-\frac{1}{2}\left(B^{\top}V_{t}B+R\right)^{-1}B^{\top}S_{t}\mathrm{M}_{\mu_{s,o}}.$

The following backward recursions

$\displaystyle V_{t-1}$
$\displaystyle=A^{\top}V_{t}A+Q_{\mu_{s,o}}-A^{\top}V_{t}B(B^{\top}V_{t}B+R)^{-1}B^{\top}V_{t}A$
$\displaystyle K_{t-1}$
$\displaystyle=-(B^{\top}V_{t}B{+}R)^{-1}B^{\top}V_{t}A$ $\displaystyle
T_{t-1}$ $\displaystyle=(V_{t}+T_{t})(A+BK_{t-1})$ $\displaystyle S_{t-1}$
$\displaystyle=S_{t}(A+BK_{t-1})+I$ $\displaystyle p_{t-1}$
$\displaystyle=p_{t}-(h_{t-1}+l_{t-1})^{\top}(B^{\top}V_{t}B+R)(h_{t-1}+l_{t-1})$
$\displaystyle\hskip
10.0pt+\bar{w}^{\top}(V_{t}+2T_{t})\bar{w}+\mathrm{M}^{\top}S_{t}\bar{w}+\mathrm{tr}(Q_{\mu_{s,o}}{\Sigma}_{t|t})$
$\displaystyle\hskip
10.0pt-2\mathrm{tr}(V_{t}H_{t|t-1})+\mathrm{tr}(A^{\top}V_{t}A\Sigma_{t-1|t-1})+\mathrm{tr}(V_{t}W),$

hold, starting from $V_{N}{=}Q_{\mu_{s,o}}$, $S_{N}{=}I$, $T_{N}{=}0$, and
$p_{N}{=}\mathrm{tr}(Q_{\mu_{s,o}}\Sigma_{N})$.

Before sketching the proof of Theorem 2 we provide the following important
lemma

######  Lemma 2.

The estimation error $e_{t}$ and the prediction error $\rho_{t}$ are
statistically independent of the controls.

###### Proof.

(Lemma 2) See Appendix 6.2. ∎

###### Proof.

(Theorem 2) See Appendix 6.3. ∎

We note that the separation principle holds here, in the sense that the
optimal control policy is affine w.r.t. the mmse state estimates. The policy
parameters can thus be computed independently of the state estimate. In fact,
the term $l_{t}$ forces the state to follow $\mathrm{M}_{\mu_{s,o}}$ which,
according to (14), (15), combines $\mathrm{M}_{w}$, $\mathrm{M}_{w\epsilon}$,
and $\mathrm{M}_{\epsilon}$. While $\mathrm{M}_{w}$ accounts only for the
process noise risky events, $\mathrm{M}_{w\epsilon}$ pre-compensates for the
cross third order statistics between the output and process noise, the latter
as seen from the output. Further, $\mathrm{M}_{\epsilon}$ considers the pure
third order statistics of the output noise. At this point, we note that the
first constraint itself might not be sufficient to account for the variability
of the output. In particular, as seen by the form of the optimal controller,
the second constraint is required in order to counteract the output noise
(term $M_{\epsilon}$ in $M$). Both constraints induce an extra inflation to
the state penalty modifying the Riccati equation and subsequently the policy
gain accordingly. As a result, the state is regulated more strictly in riskier
directions, where the process and output noise covariance, cross-covariance,
and the corresponding penalties are simultaneously large. In the following, we
show that the optimal controller (16) is internally stable under the following
standard assumption.

######  Assumption 2.

The pair $(A,B)$ is stabilizable, the matrix $Q$ is positive semi-definite and
the matrix $R$ is positive definite.

######  Proposition 2 (Stability).

Let Assumptions 1 and 2 be in effect. For a fixed pair $\mu_{s,o}\geq 0$
consider the control policy $u^{*}(\mu_{s,o})$, as defined in (16). As
$N\rightarrow\infty$, $V_{t}$ converges exponentially fast to the unique
stabilizing solution of the algebraic Riccati equation.

$\displaystyle
V=A^{\top}VA+Q_{\mu_{s,o}}-A^{\top}VB(B^{\top}VB+R)^{-1}B^{\top}VA.$

Thus, for every $t\geq 0$ it is true that, as $N\rightarrow\infty$,

$\displaystyle K_{t}$
$\displaystyle\rightarrow{K}\triangleq-(B^{\top}VB+R)^{-1}B^{\top}VA,$
$\displaystyle S_{t}$ $\displaystyle\rightarrow{S}\triangleq(I-(A+BK))^{-1},$
$\displaystyle T_{t}$
$\displaystyle\rightarrow{T}\triangleq{V}(A+BK)(I-(A+BK))^{-1},$
$\displaystyle h_{t-1}$
$\displaystyle\rightarrow{h}\triangleq-\left(B^{\top}VB+R\right)^{-1}B^{\top}\left(V+T\right)\bar{w},$
$\displaystyle l_{t-1}$
$\displaystyle\rightarrow{l}\triangleq-\frac{1}{2}\left(B^{\top}VB+R\right)^{-1}B^{\top}S\mathrm{M}_{\mu_{s,o}}.$

###### Proof.

Since $Q>Q_{\mu_{s,o}}$, detectability of $(A,Q^{1/2})$ implies detectability
of $(A,Q_{\mu_{s,o}}^{1/2})$. That $A+BK$ is Hurwitz follows from standard LQR
theory [12]. ∎

Note that the above result only concerns the internal stability of the policy.
For the closed-loop system to be well-behaved, we also need the mmse
estimator, or any other state estimator, to exhibit bounded mean square
estimation error. This can be achieved for example if $(A,Q^{1/2})$ is
detectable.

##  4 Simulations and results

Consider the inverting operational amplifier $S_{c}$ shown in Fig 1

$-$$+$$R_{\beta}$$R_{2}$$C_{2}$$+$$-$$\boldsymbol{x_{2,t}}$$R_{\alpha}$$C_{1}$$+$$-$$\boldsymbol{x_{1,t}}$$R_{1}$$-$$+$$\boldsymbol{u_{t}}$
Figure 1: Inverting operational amplifier

which may be described by the second order state space model

$\displaystyle x_{t+1}$ $\displaystyle=\left[\begin{array}[]{cc}0.172&0\\\
1.046&0.8869\end{array}\right]x_{t}+\left[\begin{array}[]{c}0.1882\\\
0.2762\end{array}\right]u_{t}+w_{t},$ (21)

where we followed [31, p. 92] with $R_{1}=0.5$, $R_{2}=1$, $R_{\alpha}=5$,
$R_{\beta}=0.25$, $C_{1}=0.5$, $C_{2}=0.3$, and subsequently discretized with
$T_{s}=0.4\leavevmode\nobreak\ \mathrm{sec}$. The system state
$x=(x_{1},x_{2})$ comprises the two capacitor voltages denoted in Fig. 1 by
forest green.

$S_{i}$$S$$+$$S_{c}$$+$$S_{o}$$u$$w$$v_{o}$$y$$\epsilon$ Figure 2:
Interconnection of inverting op-amp with two terminal stages. The input stage
$S_{i}$ refers to $S_{c}$ which has to follow $u$ and transmit $v_{o}$ despite
the voltage socks $w_{t}$ and the output disturbances $y_{t}$.

Based on the diagram in Fig. 2, $S_{c}$ is internally affected from $S_{i}$
through $u$ and it is the part of $S$ that interfaces $S_{o}$ through the
output $y$. The voltage of the resistor $R_{2}$ is set as an output and it is
also observed for state estimation according to

$\displaystyle y_{t}$
$\displaystyle=\left[\begin{array}[]{cc}0.05&-1\end{array}\right]x_{t}+\epsilon_{t},$
(23)

where $\epsilon_{t}$ expresses random events that occur peripherally around
the circuit, in this case in $R_{2}$.

The objective is to control the average energy storage in $S$ by regulating
the internal state to certain voltage levels $x^{*}{=}(0.2117,0.43995)$ while
accounting for unexpected (internal) voltage socks. On top of that, we are
interested in maintaining a low output variability subject to peripheral, rare
but catastrophic events. Roughly speaking, we are interested in controlling
the energy flow in $S$ in a bidirectional manner. On the one hand we safeguard
against risky events that act internally as disturbances in $u_{t}$, but also
keep $S$ externally shielded from the unexpected events that occur it the
output $y_{t}$, peripherally around $S$.

The stochastic disturbances $w_{t}$ model, random events within $S$. Assuming
nominal operating conditions for $S_{i}$, $S_{c}$ receives symmetric random
deviations around $u$, expressed by a Gaussian distribution
$w_{t}\sim\mathcal{N}(0,0.1)$. On the contrary, unexpected-risky events that
take place in $S$, are reflected on $w_{t}$ as ‘voltage socks’ thus modelled
as a mixture of two Gaussians $\mathcal{N}(0,0.01)$, and
$\mathcal{N}(10,0.001)$ with weights $p_{1}=0.8$ and $p_{2}=0.2$,
respectively. Further, we consider two different cases regarding the output
disturbance $\epsilon_{t}$. Nominal operating conditions are modeled by
$\epsilon_{t}\sim\mathcal{N}(0,0.01)$, while the Gaussian mixture
$0.7\mathcal{N}(0,0.01)+0.3\mathcal{N}(20,0.005)$ indicates rare but highly
undesirable events that reflect on $\epsilon_{t}$ as voltage shocks. The
reader may verify that the system (21)-(23) is controllable and observable.

Throughout the simulations we maintain $Q=\mathrm{diag}(1,1)$ and $R{=}1$ for
the LQ stage-cost, while we set $Q_{s}{=}Q_{o}{=}\mathrm{diag}(1,0.1)$ for the
state and output risk-constraint. In all cases state estimates are obtained
sequentially via a Kalman filter. Lastly, we compare the performance of our
controller with the LEQG controller-filter system [11]. Lastly, the tracking
objective may be recast as regulation objective w.r.t.

$\displaystyle\Delta{x}_{t+1}=A\Delta{x}_{t}+B\Delta{u},$ (24)

where $\Delta{x}_{t}=x_{t}-x^{*}$, $\Delta{u}_{t}=u_{t}-u^{*}$, and $u^{*}$ is
such that $x^{*}=Ax^{*}+Bu^{*}$. For the given $x^{*}$ we choose $Bu^{*}=B$,
while the output targets to $y^{*}=-0.4294$. In all cases, a Kalman filter
takes care of the mmse state estimation. The functionality of our controller
is demonstrated by the following three operating scenarios:

####  4.0.1 Case 1. (Skewed process noise-Gaussian output noise)

In this scenario our risk-averse controller safeguards against internal risky
events, while the surrounding conditions, are assumed to be nominal. Fig. 3
(top) depicts $||x_{t}-x^{*}||^{2}_{Q}$ over the first $100$ time-steps, while
at the bottom we plot the energy of the internal voltage socks.

Figure 3: State penalty variation $||x_{t}-x^{*}||^{2}_{Q}$ over time
considering skewed process noise prior and Gaussian output noise for $S$
(top). Energy of internal voltage shocks over time (bottom).

Note how the risk averse controller with $\mu_{s,o}=(10,0)$ drastically limits
the variability of the stored energy, thus protecting $S$ from the internal
voltage socks especially from those over
$t\in(15,25)\cup(45,50)\cup(60,70)\cup(85,95)$.

Although the LEQG controller performs better compared to the risk-neutral
control, it seems to be more susceptible to the incoming voltage socks but
with a slightly better average performance compared to the risk-averse
controller (black). Roughly speaking, the LEQG policy is linear; it does not
include affine terms related to third-order statistics and it renders the
controller cautious to risky events by only magnifying the Gaussian process
noise covariance. Thus, unexpected, risky events, normally modelled by the
distribution’s tail and mapped as the voltage socks that cause the state to
vary aggressively, are not considered by the LEQG controller. Worth noticing
that further improvements are not possible given that LEQG experiences the so-
called neurotic breakdown for $\theta<-0.05$.

Probably one of the main drawbacks of LEQG is that values of $\theta$ that
seem to be ‘stable’ for some noise realizations, turn out to be ‘unstable’ for
others (especially for those with large shocks). In other words, in order to
guarantee for a stable value of $\theta$ over all noise realizations, we have
to deteriorate the performance of the LEQG controller significantly. To put it
differently, compared to our state-output risk-averse controller, LEQG is not
of the ‘set and forget’ type. Lastly note that, on the one hand, state varies
more aggressively only when the system experiences successive double or triple
shocks. On the other, state experiences the internal shock with some
(constant) time delay.

####  4.0.2 Case 2. (Gaussian process noise-Skewed measurement noise)

This case demonstrates the functionality of the second constraint. In
particular, internal conditions in $S$ are assumed nominal, thus modelled with
a Gaussian process noise, while the output disturbance simulates unexpected-
risky events on the resistor voltage $y_{t}$.

Figure 4: Output variation w.r.t. the surrounding shocks $\epsilon_{t}$ for
the various control policies (top). Output voltage shocks in one-to-one
correspondence with the output variation (bottom).

Figure 4 (top) shows how the energy on the resistor $R_{2}$ alters with time.
First and foremost the resistor’s energy remains totally unguarded from the
upcoming voltage shocks under the risk-neutral policy. Note the similar
performance between our risk-averse controller with $\mu_{s,o}=(10^{6},0)$,
(black) and the LEQG (yellow), which without further improvement in between,
breaks down for $\theta=-0.08$. The Gaussian process noise kills the affine
terms of the risk-averse policy rendering our controller able to suppress
risky events only by the state penalty matrix inflation, recasting its
architecture similar to the one that, by default, LEQG enjoys. However, our
controller remains stable despite the extremes of $\mu_{s}$.

On the other hand, by activating the second constraint (forest green) with
$\mu_{s,o}=(0,5\times{10}^{-4})$ the controller counteracts risky events in a
direct manner by rendering the resistor’s energy inert to the surrounding
shocks in the cost of sacrificing average performance. Lastly, note the
instant, and more direct influence of the surrounding voltage disturbances
(bottom) to the output.

####  4.0.3 Case 3. (Skewed process noise-Skewed measurement noise)

Fig. 5 (top) shows how the various policies perform when skewed model prior
combines with skewed output prior. In order to clearly depict the
corresponding resistor’s energy variations due to the internal, and output
shocks, we separate the latter by choosing $(p_{1},p_{2})=(0.95,0.05)$ for
both types of noises. Thus, the first peak (top) is due to $\epsilon_{t}$
(bottom) while the rest are due to the inner ones $w_{t}$ (middle).

Figure 5: Output varriation due to internal and external disturbances (top).
Interrnal shocks (middle), and output shocks (bottom). Note how each type of
shock influences the output variation.

By construction, the first constraint (black) with $\mu_{s}=0.005$, reduces
the influence of the internal voltage shocks. On the contrary, the second
constraint (forest green) with $\mu_{s,o}=(0,0.05)$ counteracts for the
internal and external disturbances equally. Note that a larger value of
$\mu_{o}$ can completely flatten the effect of the external disturbance in the
cost of increasing effect of the internal disturbances. Lastly, LEQG with
$\theta=-0.0031$ achieves an adequate average performance while sufficiently
reduces variability. However, note that the specific value for $\theta$ might
be unstable for other noise realizations. Once again, simulations indicate
that guaranteed stable values of $\theta$ (over all noise realizations) comes
with significant performance deterioration.

##  5 Conclusion and future work

We proposed a new state-output risk-constrained optimal controller for
partially observed linear systems. Our formulation considers systems disrupted
by arbitrary process and/or output noise (with finite fourth order moments)
and the resulting policy admits closed form while it maintains favorable
characteristics of the classical LQ strategy, i.e. it can be pre-computed off-
line and stored. By constraining the squared error predictive variability of
the state and output penalties, the controller accounts for both internal
shocks and also shields against external (i.e., output-induced) risky events.
Our numerical results confirmed our theoretical analysis and corroborated the
usefulness of state-output constrained control via a realistic indicative
example.

In addition to those referred in [27], interesting extensions of the proposed
problem include but are not limited to risk-averse quadratic state regulation
where the risk constraints consider restricting the volatility of the distance
between the control system and an adversarial target, e.g.,

$\displaystyle\mathbb{E}\left\\{\sum_{t=1}^{N}\left[||x_{t}-g_{t}||^{2}_{Q_{s}}-\mathbb{E}\left(||x_{t}-g_{t}||^{2}_{Q_{s}}\big{|}\leavevmode\nobreak\
\overline{\mathscr{F}}_{t-1}\right)\right]^{2}\right\\}\leq\epsilon,$

where $g_{t}$ denotes the state of an adversarial target. Such extensions are
fruitful topics for further investigation.

##  6 Appendix

###  6.1 Proof of Proposition 1

####  6.1.1 State risk constraint re-formulation

Similarly to [27], recall Assumption 1, and the causality constraint
$u_{t}\in\mathcal{L}_{2}(\mathscr{F}_{t})$. We may write

$\displaystyle\tilde{x}_{t}=\mathbb{E}\\{x_{t}|\overline{\mathscr{F}}_{t-1}\\}$
$\displaystyle=\mathbb{E}\\{Ax_{t-1}+Bu_{t-1}+w_{t}|\overline{\mathscr{F}}_{t-1}\\}$
$\displaystyle=Ax_{t-1}+Bu_{t-1}+\bar{w}$ $\displaystyle=x_{t}-\delta_{t},$
(25)

where we declared $\delta_{t}=w_{t}-\bar{w}$. The quadratic term may be
written as

$\displaystyle
x^{\top}_{t}Q_{s}x_{t}=\tilde{x}^{\top}_{t}Q_{s}\tilde{x}_{t}+2\tilde{x}^{\top}Q_{s}\delta_{t}+\delta^{\top}_{t}Q_{s}\delta_{t}\leavevmode\nobreak\
,$ (26)

and therefore

$\displaystyle\mathbb{E}\\{x^{\top}_{t}Q_{s}x_{t}|\overline{\mathscr{F}}_{t-1}\\}=\tilde{x}^{\top}_{t}Q_{s}\tilde{x}_{t}+\mathrm{tr}(Q_{s}W),$
(27)

where we defined

$\displaystyle
W:=\mathbb{E}\\{(w_{t}-\bar{w})(w_{t}-\bar{w})^{\top}\\}=\mathbb{E}\\{\delta_{t}\delta^{\top}_{t}\\}$
(28)

as the process noise covariance. For the squared difference between the two we
obtain

$\displaystyle\Delta^{2}_{s,t}$
$\displaystyle=\Big{[}x^{\top}_{t}Q_{s}x_{t}-\mathbb{E}\big{\\{}x^{\top}_{t}Q_{s}x_{t}|\overline{\mathscr{F}}_{t-1}\big{\\}}\Big{]}^{2}$
$\displaystyle=\Big{[}2\tilde{x}^{\top}Q_{s}\delta_{t}+\delta^{\top}_{t}Q_{s}\delta_{t}-\mathrm{tr}(WQ_{s})\Big{]}^{2}$
$\displaystyle=(\delta^{\top}_{t}Q_{s}\delta_{t}-\mathrm{tr}(WQ_{s}))^{2}+4\tilde{x}^{\top}_{t}Q_{s}\delta_{t}(\delta^{\top}_{t}Q\delta_{t}-\mathrm{tr}(WQ_{s}))+4\tilde{x}^{\top}Q_{s}\delta_{t}\delta^{\top}_{t}Q_{s}\tilde{x}_{t}.$
(29)

At this point we define

$\displaystyle\mathrm{M}_{w}$
$\displaystyle\triangleq\mathbb{E}\Big{\\{}(w_{t}-\bar{w})(w_{t}-\bar{w})^{\top}Q_{s}(w_{t}-\bar{w})\Big{\\}},\leavevmode\nobreak\
\text{and}$ $\displaystyle m_{w}$
$\displaystyle\triangleq\mathbb{E}\Big{\\{}\Big{(}(w_{t}-\bar{w})^{\top}Q_{s}(w_{t}-\bar{w})-\mathrm{tr}(Q_{s}W)\Big{)}^{2}\Big{\\}}.$
(30)

Thus,

$\displaystyle\mathbb{E}\\{\Delta^{2}_{s,t}\\}$
$\displaystyle=m_{w}+\mathbb{E}\\{4\tilde{x}^{\top}_{t}Q_{s}\delta_{t}(\delta^{\top}_{t}Q_{s}\delta_{t}-\mathrm{tr}(WQ_{s}))+4\tilde{x}^{\top}Q_{s}\delta_{t}\delta^{\top}_{t}Q_{s}\tilde{x}_{t}\\}$
$\displaystyle=m_{w}+\mathbb{E}\\{4\tilde{x}^{\top}_{t}Q_{s}\delta_{t}\delta_{t}Q\delta_{t}\\}-\mathbb{E}\\{4\tilde{x}^{\top}_{t}Q_{s}\delta_{t}\mathrm{tr}(Q_{s}W)\\}+4\mathbb{E}\\{\tilde{x}_{t}Q_{s}\delta_{t}\delta^{\top}_{t}Q_{s}\tilde{x}_{t}\\}.$
(31)

By construction of $\tilde{x}_{t}$ and by first projecting onto
$\overline{\mathscr{F}}_{t-1}$, for the first term we obtain

$\displaystyle\mathbb{E}\\{4\tilde{x}^{\top}_{t}Q_{s}\delta_{t}\delta_{t}Q_{s}\delta_{t}\\}$
$\displaystyle=4\mathbb{E}\\{\mathbb{E}\\{\tilde{x}^{\top}_{t}Q_{s}\delta_{t}\delta_{t}Q_{s}\delta_{t}|\overline{\mathscr{F}}_{t-1}\\}\\}$
$\displaystyle=4\mathbb{E}\\{\tilde{x}^{\top}Q_{s}\mathrm{M}_{w}\\}$ (32)

Regarding the second term, $\mathbb{E}\\{\delta_{t}\\}=0$ implies
$\mathbb{E}\\{4\tilde{x}^{\top}_{t}Q\delta_{t}\mathrm{tr}(Q_{s}W)\\}=0$, while
the third one reads

$\displaystyle
4\mathbb{E}\\{\tilde{x}^{\top}_{t}Q_{s}\delta_{t}\delta^{\top}_{t}Q_{s}\tilde{x}_{t}\\}$
$\displaystyle=4\mathbb{E}\\{\tilde{x}^{\top}_{t}Q_{s}WQ_{s}\tilde{x}_{t}\\}.$
(33)

Thus,

$\displaystyle\mathbb{E}\\{\Delta^{2}_{s,t}\\}$
$\displaystyle=m_{w}+\mathbb{E}\\{4\tilde{x}^{\top}_{t}Q_{s}WQ_{s}\tilde{x}_{t}+4\tilde{x}_{t}Q_{s}\mathrm{M}_{w}\\}$
$\displaystyle=m_{w}+\mathbb{E}\\{4x_{t}^{\top}Q_{s}WQ_{s}x_{t}-8x^{\top}_{t}Q_{s}WQ_{s}\delta_{t}+4\delta^{\top}_{t}Q_{s}WQ_{s}\delta_{t}+4x_{t}Q_{s}\mathrm{M}_{w}-4\delta_{t}Q_{s}\mathrm{M}_{w}\\}$
$\displaystyle=m_{w}+\mathbb{E}\\{4x_{t}^{\top}Q_{s}WQ_{s}x_{t}+4x_{t}Q_{s}\mathrm{M}_{w}-8x^{\top}_{t}Q_{s}WQ_{s}\delta_{t}+4\delta^{\top}_{t}Q_{s}WQ_{s}\delta_{t}\\}$
$\displaystyle=m_{w}+4(\mathrm{tr}(Q_{s}W))^{2}-8(\mathrm{tr}(Q_{s}W))^{2}+\mathbb{E}\\{4x_{t}^{\top}Q_{s}WQ_{s}x_{t}+4x_{t}Q_{s}\mathrm{M}_{w}\\}$
$\displaystyle=m_{w}-4(\mathrm{tr}(Q_{s}W))^{2}+\mathbb{E}\\{4x_{t}^{\top}Q_{s}WQ_{s}x_{t}+4x_{t}Q_{s}\mathrm{M}_{w}\\}$
(34)

Therefore, the constraint recasts to

$\displaystyle\sum^{t=N}_{t=1}\mathbb{E}\\{\Delta^{2}_{s,t}\\}=\sum^{t=N}_{t=1}\mathbb{E}\\{4x_{t}^{\top}Q_{s}WQ_{s}x_{t}+4x_{t}Q_{s}\mathbb{M}_{w}\\}+Nm_{4}-4N(\mathrm{tr}(Q_{s}W))^{2}<\epsilon_{s}$
(35)

or

$\displaystyle\sum^{t=N}_{t=1}\mathbb{E}\\{4x_{t}^{\top}Q_{s}WQ_{s}x_{t}+4x_{t}Q_{s}\mathrm{M}_{w}\\}<\bar{\epsilon}_{s}$
(36)

whith $\bar{\epsilon}_{s}=\epsilon_{s}-Nm_{w}+4N(\mathrm{tr}(Q_{s}W))^{2}$.

####  6.1.2 Output risk constraint re-formulation

Tore-formulate the second constraint, recall Assumption 1, the causality
constraint $u_{t}\in\mathcal{L}_{2}(\mathscr{F}_{t})$, and consider (2), along
with (25). Therefore, we may write $\tilde{y}_{t}=y_{t}-p_{t}$, with
$p_{t}=C[w_{t}-\bar{w}]+\epsilon_{t}-\bar{\epsilon}$, and
$\mathbb{E}\\{p_{t}\\}=0$, and subsequently write the quadratic as

$\displaystyle
y^{\top}_{t}Q_{o}y_{t}=\tilde{y}^{\top}Q_{o}\tilde{y}_{t}+2\tilde{y}^{\top}_{t}Q_{o}p_{t}+p^{\top}_{t}Q_{o}p_{t}.$
(37)

Hence,

$\displaystyle\mathbb{E}\\{y^{\top}_{t}Q_{o}y_{t}|\overline{\mathscr{F}}_{t-1}\\}$
$\displaystyle=\tilde{y}^{\top}Q_{o}\tilde{y}_{t}+\mathbb{E}\\{p^{\top}_{t}Q_{o}p_{t}\\}$
$\displaystyle=\tilde{y}^{\top}Q_{o}\tilde{y}_{t}+\mathrm{tr}(Q_{o}P).$ (38)

where $P\triangleq\mathbb{E}\\{p_{t}p^{\top}_{t}\\}$. The squared difference
reads

$\displaystyle\Delta^{2}_{o,t}$
$\displaystyle=\Big{[}y^{\top}_{t}Q_{o}y_{t}-\mathbb{E}\big{\\{}y^{\top}_{t}Q_{o}y_{t}|\overline{\mathscr{F}}_{t-1}\big{\\}}\Big{]}^{2}$
$\displaystyle=\Big{[}2\tilde{y}_{t}^{\top}Q_{o}p_{t}+p^{\top}_{t}Q_{o}p_{t}-\mathrm{tr}(Q_{o}P)\Big{]}^{2}$
$\displaystyle=(p^{\top}_{t}Q_{o}p_{t}-\mathrm{tr}(Q_{o}P))^{2}$
$\displaystyle\hskip
10.0pt+4\tilde{y}^{\top}_{t}Q_{o}p_{t}(p^{\top}_{t}Q_{o}p_{t}-\mathrm{tr}(Q_{o}P))$
$\displaystyle\hskip
10.0pt+4\tilde{y}_{t}^{\top}Q_{o}p_{t}p^{\top}_{t}Q_{o}\tilde{y}_{t},$ (39)

and by taking expectation in (39) we obtain

$\displaystyle\mathbb{E}\big{\\{}\Delta^{2}_{o,t}\big{\\}}{=}m_{w\epsilon}+4\mathbb{E}\\{\tilde{y}^{\top}_{t}Q_{o}\mathrm{M}\\}+4\mathbb{E}\big{\\{}\tilde{y}_{t}^{\top}Q_{o}PQ_{o}\tilde{y}_{t}\big{\\}}.$
(40)

Further,
$\tilde{y}_{t}=C\tilde{x}_{t}+\bar{\epsilon}=Cx_{t}-C\delta_{t}+\bar{\epsilon}$
and therefore, (40) may be written w.r.t. the system’s state as

$\displaystyle\mathbb{E}\big{\\{}\Delta^{2}_{o,t}\big{\\}}$
$\displaystyle=m_{w\epsilon}+\mathbb{E}\\{4x_{t}^{\top}C^{\top}Q_{o}\mathrm{M}\\}+4\bar{\epsilon}^{\top}Q_{o}\mathrm{M}$
$\displaystyle+4\mathbb{E}\big{\\{}(x^{\top}_{t}C^{\top}+\zeta_{t}^{\top})Q_{o}PQ_{o}(Cx_{t}+\zeta_{t})\big{\\}},$
(41)

where we declared $\zeta_{t}=-C\delta_{t}+\bar{\epsilon}$. Thus,

$\displaystyle\mathbb{E}\big{\\{}\Delta^{2}_{o,t}\big{\\}}$
$\displaystyle=m_{w\epsilon}+\mathbb{E}\big{\\{}4x^{\top}_{t}C^{\top}Q_{o}\mathrm{M}\big{\\}}$
$\displaystyle\hskip
10.0pt+\mathbb{E}\big{\\{}4x^{\top}_{t}C^{\top}Q_{o}PQ_{o}Cx_{t}\big{\\}}$
$\displaystyle\hskip
10.0pt+4\bar{\epsilon}Q_{o}\mathrm{M}-8\mathrm{tr}(C^{\top}Q_{o}PQ_{o}CW)$
$\displaystyle\hskip
10.0pt+\mathbb{E}\big{\\{}8x^{\top}_{t}C^{\top}Q_{o}PQ_{o}\bar{\epsilon}\big{\\}}$
$\displaystyle\hskip 10.0pt+\mathrm{tr}(4Q_{o}PQ_{o}Z),$ (42)

where $Z=\mathbb{E}\big{\\{}\zeta_{t}\zeta_{t}^{\top}\big{\\}}$. Therefore,
the output risk constraint may be written as

$\displaystyle\sum^{N}_{t=1}\mathbb{E}\big{\\{}4x^{\top}_{t}C^{\top}Q_{o}PQ_{o}Cx_{t}+4x^{\top}_{t}C^{\top}Q_{o}(\mathrm{M}+2PQ_{o}\bar{\epsilon})\big{\\}}<\bar{\epsilon}_{o},$

where
$\bar{\epsilon}_{o}{=}{-}Nm_{w\epsilon}{-}4N\bar{\epsilon}^{\top}Q_{o}\mathrm{M}{-}8N\mathrm{tr}(C^{\top}Q_{o}PQ_{o}CW){-}4N\mathrm{tr}(Q_{o}PQ_{o}Z){+}\epsilon_{o}.$

###  6.2 Proof of Lemma 2

The solution of (1) may be written as

$\displaystyle x_{t}=x^{r}_{t}+x^{d}_{t}$ (43)

where $x^{r}_{t}$ transitions based on

$\displaystyle x^{r}_{t+1}=Ax^{r}_{t}+w_{t}$ (44)

with $x^{r}_{0}=x_{0}$, while $x^{d}_{t}$ according to

$\displaystyle x^{d}_{t+1}=Ax^{d}_{t}+u_{t}$ (45)

with $x^{d}_{0}=0$. Subsequently,

$\displaystyle y_{t}=y^{r}_{t}+y^{d}_{t},$ (46)

where $y^{r}_{t}=Cx^{r}_{t}+v_{t}$, and $y^{d}_{t}=Cx^{d}_{t}$. We will show
that

$\displaystyle\mathscr{F}_{t}\triangleq\sigma(y_{0:t},u_{0:t-1})\equiv\sigma(y^{r}_{0:t}),$
(47)

By construction, $x^{d}_{t}$ and $y^{d}_{t}$ are deterministic functions of
$u_{0:t}$. Since $y^{r}_{t}=y_{t}-y^{d}_{t}$, $y^{r}_{t}$ is a function of
$(y_{t},u_{0:t})$. Thus, $y^{r}_{0:t}$ is a function of $(y_{0:t},u_{0:t})$
and as a result
$\sigma(y^{r}_{0:t})\subseteq\sigma(y_{0:t},u_{0:t-1})=\mathscr{F}_{t}$.
To prove the reverse inclusion, fix a (deterministic) policy $\pi$, and
further write $y_{0:t}=y^{r}_{0:t}+y^{d}_{0:t}$. Note that

$\displaystyle y^{d}_{0}$ $\displaystyle=Cx_{0}$ $\displaystyle y_{1}^{d}$
$\displaystyle=C(Ax_{0}+B\pi(y^{r}_{0}+Cx_{0}))$
$\displaystyle=\varphi_{1}(y^{r}_{0})$ $\displaystyle y^{d}_{2}$
$\displaystyle=C(A^{2}x_{0}+AB\pi(y^{r}_{0}+Cx_{0})+B\pi(y^{r}_{0:1}+y^{d}_{0:1}))$
$\displaystyle=\varphi_{2}(y^{r}_{0:1})$ $\displaystyle\vdots$ $\displaystyle
y^{d}_{n}$ $\displaystyle=\varphi_{n}(y^{r}_{0:n-1}),$

where $\varphi_{n}:\mathbb{R}^{q\times{n}}\rightarrow\mathbb{R}^{q}$ are
deterministic functions. Thus,

$\displaystyle u_{0:t-1}$
$\displaystyle=\pi(y_{0:t-1})=\pi(y^{r}_{0:t-1}+\varphi_{t}(y^{r}_{0:t-1})).$

As a result,

$\displaystyle(y_{0:t},u_{0:t-1}){=}\big{(}y^{r}_{0:t}+\varphi_{t}(y^{r}_{0:t-1}),\pi(y^{r}_{0:t-1}+\varphi_{t}(y^{r}_{0:t-1}))\big{)}$

and therefore

$\displaystyle\sigma(y_{0:t},u_{0:t-1})\subseteq\sigma(y^{r}_{0:t})$

which concludes the proof.

###  6.3 Proof of Theorem 2

####  6.3.1 Bellman’s Equation

Consider the tail sub-problem

$\displaystyle\mathcal{L}^{*}(\mathscr{F}_{k};\mu_{s,o})$
$\displaystyle\triangleq\inf_{u_{k:N-1}}\mathbb{E}\Bigg{\\{}g(x_{N};\mu_{s,o})+\sum^{N-1}_{t=k}g(x_{t},u_{t};\mu_{s,o})\Bigg{|}\mathscr{F}_{k}\Bigg{\\}}$
$\displaystyle=\inf_{u_{k:N-1}}\mathbb{E}\Bigg{\\{}g(x_{N};\mu_{s,o})+\sum^{N-1}_{t=k+1}g(x_{t},u_{t};\mu_{s,o})+g(x_{k},u_{t};\mu_{s,o})\Bigg{|}\mathscr{F}_{k}\Bigg{\\}}$
$\displaystyle=\inf_{u_{k:N-1}}\mathbb{E}\Bigg{\\{}g(x_{N};\mu_{s,o})+\sum^{N-1}_{t=k+1}g(x_{t},u_{t};\mu_{s,o})+g(\widehat{x}_{k|k},u_{k};\mu_{s,o})+\phi(\widehat{x}_{k|k},e_{k};\mu_{s,o})\Bigg{|}\mathscr{F}_{k}\Bigg{\\}}$
$\displaystyle=\inf_{u_{k:N-1}}\mathbb{E}\Bigg{\\{}g(x_{N};\mu_{s,o})+\sum^{N-1}_{t=k+1}g(x_{t},u_{t};\mu_{s,o})+g(\widehat{x}_{k|k},u_{k};\mu_{s,o})+\mathrm{tr}(Q_{\mu_{s,o}}\Sigma_{k})\Bigg{|}\mathscr{F}_{k}\Bigg{\\}}.$
(48)

The latter rests on the fact that
$\mathbb{E}\\{\phi(\widehat{x}_{k|k},e_{k};\mu_{s,o})|\mathscr{F}_{k}\\}=\mathrm{tr}(Q_{\mu_{s,o}}\mathbb{E}\\{e_{k}e^{\top}_{k}|\mathscr{F}_{k}\\})$.
Thus

$\displaystyle\mathcal{L}^{*}(\mathscr{F}_{k};\mu_{s,o})$
$\displaystyle=\inf_{u_{k}}\leavevmode\nobreak\
\mathbb{E}\Bigg{\\{}\inf_{u_{k+1:N-1}}\Bigg{(}\mathbb{E}\Bigg{\\{}g(x_{N};\mu_{s,o})+\sum^{N-1}_{t=k+1}g(x_{t},u_{t};\mu_{s,o})\Bigg{|}\mathscr{F}_{k+1}\Bigg{\\}}\Bigg{)}$
$\displaystyle\hskip
113.81102pt+g(\widehat{x}_{k|k},u_{k};\mu_{s,o})+\mathrm{tr}(Q_{\mu_{s,o}}\mathbb{E}\\{e_{k}e^{\top}_{k}|\mathscr{F}_{k}\\})\Bigg{|}\mathscr{F}_{k}\Bigg{\\}}$
$\displaystyle=\inf_{u_{k}}\leavevmode\nobreak\
\mathbb{E}\Bigg{\\{}\mathcal{L}^{*}(\mathscr{F}_{k+1};\mu_{s,o})+g(\widehat{x}_{k|k},u_{k};\mu_{s,o})+\mathrm{tr}(Q_{\mu_{s,o}}\mathbb{E}\\{e_{k}e^{\top}_{k}|\mathscr{F}_{k}\\})\Bigg{|}\mathscr{F}_{k}\Bigg{\\}}$
$\displaystyle=\inf_{u_{k}}\leavevmode\nobreak\
\Big{(}\mathbb{E}\Big{\\{}\mathcal{L}^{*}(\mathscr{F}_{k+1};\mu_{s,o})\Big{|}\mathscr{F}_{k}\Big{\\}}+g(\widehat{x}_{k|k},u_{k};\mu_{s,o})+\mathrm{tr}(Q_{\mu_{s,o}}\mathbb{E}\\{e_{k}e^{\top}_{k}|\mathscr{F}_{k}\\})\Big{)}$
$\displaystyle=\inf_{u_{k}}\leavevmode\nobreak\
\Big{(}\mathbb{E}\Big{\\{}\mathcal{L}^{*}(\mathscr{F}_{k+1};\mu_{s,o})\Big{|}\mathscr{F}_{k}\Big{\\}}+g(\widehat{x}_{k|k},u_{k};{\mu_{s,o}})\Big{)}+\mathrm{tr}(Q_{\mu_{s,o}}\mathbb{E}\\{e_{k}e^{\top}_{k}|\mathscr{F}_{k}\\}),$
(49)

where the latter is based on Lemma 2. Thus, Bellman equation reads

$\displaystyle\mathcal{L}^{*}(\mathscr{F}_{k};\mu_{s,o})$
$\displaystyle=\inf_{u_{k}}\leavevmode\nobreak\
\Big{(}\mathbb{E}\Big{\\{}\mathcal{L}^{*}(\mathscr{F}_{k+1};\mu_{s,o})\Big{|}\mathscr{F}_{k}\Big{\\}}+g(\widehat{x}_{k|k},u_{k};{\mu_{s,o}})\Big{)}+\mathrm{tr}(Q_{\mu_{s,o}}\Sigma_{k}).$
(50)

####  6.3.2 Optimal Policy

The proof follows by induction. We demonstrate the proof for the root.
Bellman’s starts from $k=N$ with

$\displaystyle\mathcal{L}^{*}(\mathscr{F}_{N};\mu_{s,o})$
$\displaystyle=\mathbb{E}\Big{\\{}g(x_{N};\mu_{s,o})\leavevmode\nobreak\
\big{|}\leavevmode\nobreak\
\mathscr{F}_{N}\Big{\\}}=\mathbb{E}\Big{\\{}x^{\top}_{N}Q_{\mu_{s,o}}x_{N}+x^{\top}_{N}\mathrm{M}_{\mu_{s,o}}\leavevmode\nobreak\
\big{|}\leavevmode\nobreak\ \mathscr{F}_{N}\Big{\\}}$

At this point decompose $x_{N}=\widehat{x}_{N|N}+e_{N}$ to obtain

$\displaystyle\mathcal{L}^{*}(\mathscr{F}_{N};\mu_{s,o})$
$\displaystyle=\mathbb{E}\Big{\\{}\widehat{x}_{N|N}^{\top}Q_{\mu_{s,o}}\widehat{x}_{N|N}+\widehat{x}_{N|N}^{\top}\mathrm{M}_{\mu_{s,o}}+2\widehat{x}^{\top}_{N|N}Q_{\mu_{s,o}}e_{N}+e^{\top}_{N}Q_{\mu_{s,o}}e_{N}+e^{\top}_{N}\mathrm{M}_{\mu_{s,o}}\leavevmode\nobreak\
\big{|}\leavevmode\nobreak\ \mathscr{F}_{N}\Big{\\}}$
$\displaystyle=\widehat{x}_{N|N}^{\top}Q_{\mu_{s,o}}\widehat{x}_{N|N}+\widehat{x}_{N|N}^{\top}\mathrm{M}_{\mu_{s,o}}+\mathrm{tr}\Big{(}{Q}_{\mu_{s,o}}\Sigma_{N|N}\Big{)}$
(51)

The latter rests of the fact that
$\mathbb{E}\\{\widehat{x}_{N|N}^{\top}Q_{\mu_{s,o}}e_{N}|\mathscr{F}_{N}\\}=\widehat{x}_{N|N}^{\top}Q_{\mu_{s,o}}\mathbb{E}\\{e_{N}|\mathscr{F}_{N}\\}=0$
and
$\mathbb{E}\\{e^{\top}_{N}\mathrm{M}_{\mu_{s,o}}|\mathscr{F}_{N}\\}=\mathbb{E}\\{e^{\top}_{N}|\mathscr{F}_{N}\\}\mathrm{M}_{\mu_{s,o}}=0$.
Thus,

$\displaystyle\mathcal{L}^{*}(\mathscr{F}_{N};\mu_{s,o})$
$\displaystyle=\widehat{x}_{N|N}^{\top}Q_{\mu_{s,o}}\widehat{x}_{N|N}+\widehat{x}_{N|N}^{\top}\mathrm{M}_{\mu_{s,o}}+\mathrm{tr}\Big{(}Q_{\mu_{s,o}}\Sigma_{N|N}\Big{)}$
(52)

or

$\displaystyle\mathcal{L}^{*}(\mathscr{F}_{N};\mu_{s,o})$
$\displaystyle=\widehat{x}_{N|N}^{\top}V_{N}\widehat{x}_{N|N}+2\widehat{x}^{\top}_{N|N}T_{N}\bar{w}+\widehat{x}_{N|N}^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+p_{N},$
(53)

where $V_{N}=Q_{\mu_{s,o}}$, $S_{N}=I$, $T_{N}=0$, and
$p_{N}=\mathrm{tr}\Big{(}Q_{\mu_{s,o}}\Sigma_{N|N}\Big{)}$. Further,

$\displaystyle\mathcal{L}^{*}(\mathscr{F}_{N-1};\mu_{s,o})$
$\displaystyle=\inf_{u_{N-1}}\leavevmode\nobreak\
\Big{(}\mathbb{E}\Big{\\{}\mathcal{L}^{*}(\mathscr{F}_{N};\mu_{s,o})\Big{|}\mathscr{F}_{N-1}\Big{\\}}+g(\widehat{x}_{N-1|N-1},u_{N-1};{\mu_{s,o}})\Big{)}+\mathrm{tr}(Q_{\mu_{s,o}}\Sigma_{N-1|N-1}).$
(54)

Starting with
$\mathbb{E}\Big{\\{}\mathcal{L}^{*}(\mathscr{F}_{N};\mu_{s,o})\Big{|}\mathscr{F}_{N-1}\Big{\\}}$,
we obtain

$\displaystyle\mathbb{E}\Big{\\{}\mathcal{L}^{*}(\mathscr{F}_{N};\mu_{s,o})\Big{|}\mathscr{F}_{N-1}\Big{\\}}$
$\displaystyle=\mathbb{E}\Big{\\{}\widehat{x}_{N|N}^{\top}V_{N}\widehat{x}_{N|N}+\widehat{x}_{N|N}^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+p_{N}\Big{|}\mathscr{F}_{N-1}\Big{\\}},$
(55)

and after decomposing $\widehat{x}_{N|N}=x_{N}-e_{N}$ we get

$\displaystyle\mathbb{E}\Big{\\{}\mathcal{L}^{*}(\mathscr{F}_{N};\mu_{s,o})\Big{|}\mathscr{F}_{N-1}\Big{\\}}$
$\displaystyle=\mathbb{E}\Big{\\{}{x}_{N}^{\top}V_{N}{x}_{N}+{x}_{N}^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+p_{N}-2x^{\top}_{N}V_{N}e_{N}+e^{\top}_{N}V_{N}e_{N}-e^{\top}_{N}S_{N}\mathrm{M}_{\mu_{s,o}}\Big{|}\mathscr{F}_{N-1}\Big{\\}}$
$\displaystyle=\mathbb{E}\Big{\\{}{x}_{N}^{\top}V_{N}{x}_{N}+{x}_{N}^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+p_{N}\Big{|}\mathscr{F}_{N-1}\Big{\\}}-2\mathrm{tr}(V_{N}H_{N|N-1})+\mathrm{tr}(V_{N}\Sigma_{N|N}).$
(56)

Further, given that $x_{N}=Ax_{N-1}+Bu_{N-1}+w_{N}$ the first term reads:

$\displaystyle\mathbb{E}\Big{\\{}(Ax_{N-1}+Bu_{N-1}+w_{N})^{\top}V_{N}(Ax_{N-1}+Bu_{N-1}+w_{N})+(Ax_{N-1}+Bu_{N-1}+w_{N})^{\top}S_{N}\mathrm{M}(\mu_{s,o}+p_{N}\Big{|}\mathscr{F}_{N-1}\Big{\\}}=$
$\displaystyle=\mathbb{E}\Big{\\{}x^{\top}_{N-1}A^{\top}V_{N}Ax_{N-1}+2x^{\top}_{N-1}A^{\top}V_{N}Bu_{N-1}+2x^{\top}_{N-1}A^{\top}V_{N}w_{N}+u^{\top}_{N-1}B^{\top}V_{N}Bu_{N-1}+2u^{\top}_{N-1}B^{\top}V_{N}w_{N}$
$\displaystyle+w^{\top}_{N}V_{N}w_{N}+x^{\top}_{N-1}A^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+u^{\top}_{N-1}B^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+w^{\top}_{N}S_{N}\mathrm{M}_{\mu_{s,o}}\Big{|}\mathscr{F}_{N-1}\Big{\\}},$
(57)

and after decomposing $x_{N-1}=\widehat{x}_{N-1|N-1}+e_{N-1}$, the first term
in (56) yields

$\displaystyle\mathbb{E}\Big{\\{}{x}_{N}^{\top}V_{N}{x}_{N}+{x}_{N}^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+p_{N}\Big{|}\mathscr{F}_{N-1}\Big{\\}}=$
$\displaystyle=\widehat{x}^{\top}_{N-1|N-1}A^{\top}V_{N}A\widehat{x}_{N-1|N-1}+2\widehat{x}^{\top}_{N-1|N-1}A^{\top}V_{N}Bu_{N-1}+2\widehat{x}^{\top}_{N-1|N-1}A^{\top}V_{N}\bar{w}+u^{\top}_{N-1}B^{\top}V_{N}Bu_{N-1}$
$\displaystyle+2u^{\top}_{N-1}B^{\top}V_{N}\bar{w}+\bar{w}^{\top}V_{N}\bar{w}+\mathrm{tr}(V_{N}W)+\widehat{x}^{\top}_{N-1|N-1}A^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+u^{\top}_{N-1}B^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+\bar{w}^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}$
$\displaystyle+\mathrm{tr}(A^{\top}V_{N}A\Sigma_{N-1|N-1})$ (58)

Thus,

$\displaystyle\mathbb{E}\Big{\\{}\mathcal{L}^{*}(\mathscr{F}_{N};\mu_{s,o})\Big{|}\mathscr{F}_{N-1}\Big{\\}}$
$\displaystyle=\widehat{x}^{\top}_{N-1|N-1}A^{\top}V_{N}A\widehat{x}_{N-1|N-1}+2\widehat{x}^{\top}_{N-1|N-1}A^{\top}V_{N}Bu_{N-1}+2\widehat{x}^{\top}_{N-1|N-1}A^{\top}V_{N}\bar{w}+u^{\top}_{N-1}B^{\top}V_{N}Bu_{N-1}$
$\displaystyle+2u^{\top}_{N-1}B^{\top}V_{N}\bar{w}+\bar{w}^{\top}V_{N}\bar{w}+\mathrm{tr}(V_{N}W)+\widehat{x}^{\top}_{N-1|N-1}A^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+u^{\top}_{N-1}B^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+\bar{w}^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}$
$\displaystyle+\mathrm{tr}(A^{\top}V_{N}A\Sigma_{N-1|N-1})-2\mathrm{tr}(V_{N}H_{N|N-1})+\mathrm{tr}(V_{N}\Sigma_{N|N}).$
(59)

As a result, the minimization of

$\displaystyle\inf_{u_{N-1}}\leavevmode\nobreak\
\Big{(}\mathbb{E}\Big{\\{}\mathcal{L}^{*}(\mathscr{F}_{N};\mu_{s,o})\Big{|}\mathscr{F}_{N-1}\Big{\\}}+g(\widehat{x}_{N-1|N-1},u_{N-1};{\mu_{s,o}})\Big{)}$
(60)

in (54) rests on Lemma 2, and provides the optimal control

$\displaystyle u^{*}_{N-1}=$
$\displaystyle-\left(B^{\top}V_{N}B+R\right)^{-1}B^{\top}V_{N}A\widehat{x}_{N-1|N-1}$
$\displaystyle-\left(B^{\top}V_{N}B+R\right)^{-1}B^{\top}V_{N}\bar{w}$
$\displaystyle-\frac{1}{2}\left(B^{\top}V_{N}B+R\right)^{-1}B^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}$
(61)

or

$\displaystyle u^{*}_{N-1}=K_{N-1}\widehat{x}_{N-1|N-1}+h_{N-1}+l_{N-1}$ (62)

where

$\displaystyle K_{N-1}$
$\displaystyle=-\left(B^{\top}V_{N}B+R\right)^{-1}B^{\top}V_{N}A$ (63)
$\displaystyle h_{N-1}$
$\displaystyle=-\left(B^{\top}V_{N}B+R\right)^{-1}B^{\top}V_{N}\bar{w}$ (64)
$\displaystyle l_{N-1}$
$\displaystyle=-\frac{1}{2}\left(B^{\top}V_{N}B+R\right)^{-1}B^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}$
(65)

By substituting $u^{*}_{N-1}$, we obtain the optimal cost-to-go at $t=N-1$ as
follows

$\displaystyle\widehat{x}^{\top}_{N-1|N-1}\big{(}A^{\top}V_{N}+Q_{\mu_{s,o}}-A^{\top}V_{N}B(B^{\top}V_{N}B+R)^{-1}B^{\top}V_{N}A\big{)}\widehat{x}_{N-1|N-1}$
$\displaystyle+\widehat{x}^{\top}_{N-1|N-1}\big{(}2A^{\top}-2(B^{\top}V_{N}B+R)^{-1}B^{\top}V_{N}AB^{\top}V_{N}\big{)}\bar{w}$
$\displaystyle+\widehat{x}^{\top}_{N-1|N-1}\big{(}A^{\top}S_{N}-A^{\top}V_{N}B(B^{\top}V_{N}B+R)^{-1}B^{\top}S_{N}+I\big{)}\mathrm{M}_{\mu_{s,o}}$
$\displaystyle+(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(h_{N-1}+l_{N-1})+(h_{N-1}+l_{N-1})^{\top}\big{(}2B^{\top}V_{N}\bar{w}+B^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}\big{)}$
$\displaystyle+\bar{w}^{\top}V_{N}\bar{w}+\mathrm{tr}(V_{N}W)+\bar{w}^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+\mathrm{tr}(V_{N}\Sigma_{N|N-1})-2\mathrm{tr}(V_{N}H_{N|N-1})+\mathrm{tr}(V_{N}\Sigma_{N-1|N-1}).$
(66)

The second line reads

$\displaystyle\widehat{x}^{\top}_{N-1|N-1}\big{(}2A^{\top}-2(B^{\top}V_{N}B+R)^{-1}B^{\top}V_{N}AB^{\top}V_{N}\big{)}\bar{w}$
$\displaystyle+\widehat{x}^{\top}_{N-1|N-1}\big{(}A^{\top}S_{N}-A^{\top}V_{N}B(B^{\top}V_{N}B+R)^{-1}B^{\top}S_{N}\big{)}\mathrm{M}_{\mu_{s,o}}$
$\displaystyle=2\widehat{x}^{\top}_{N-1|N-1}(A^{\top}V_{N}+K^{\top}_{N-1}B^{\top}V_{N})\bar{w}$
$\displaystyle=2\widehat{x}^{\top}_{N-1|N-1}\big{(}A^{\top}+K^{\top}_{N-1}B^{\top}\big{)}V_{N}\bar{w}$
$\displaystyle=2\bar{w}^{\top}V_{N}\big{(}A+BK_{N-1}\big{)}\widehat{x}_{N-1|N-1}$
$\displaystyle=2\widehat{x}^{\top}_{N-1|N-1}\big{(}A+BK_{N-1}\big{)}^{\top}V_{N}\bar{w}.$
(67)

The third line reads

$\displaystyle\widehat{x}^{\top}_{N-1|N-1}\big{(}A^{\top}S_{N}-A^{\top}V_{N}BB^{\top}V_{N}B+R)^{-1}B^{\top}S_{N}+I\big{)}\mathrm{M}_{\mu_{s,o}}$
$\displaystyle=\widehat{x}^{\top}_{N-1|N-1}\big{(}(A^{\top}-K^{\top}_{N-1}B^{\top})S_{N}+I\big{)}\mathrm{M}_{\mu_{s,o}}.$

For the fourth line

$\displaystyle(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(h_{N-1}+l_{N-1})+(h_{N-1}+l_{N-1})^{\top}\big{(}2B^{\top}V_{N}\bar{w}+B^{\top}S_{N}\mathrm{M}_{\mu_{s,o}},\big{)},$

just multiply and divide by $(B^{\top}V_{N}B+R)$ to obtain

$\displaystyle(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(h_{N-1}+l_{N-1})$
$\displaystyle+(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(B^{\top}V_{N}B+R)^{-1}\big{(}2B^{\top}V_{N}\bar{w}+B^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}\big{)}$
$\displaystyle=(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(h_{N-1}+l_{N-1})$
$\displaystyle+(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(B^{\top}V_{N}B+R)^{-1}\big{(}2B^{\top}V_{N}\bar{w}+B^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}\big{)}$
$\displaystyle=(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(h_{N-1}+l_{N-1})$
$\displaystyle-2(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(B^{\top}V_{N}B+R)^{-1}\big{(}-B^{\top}V_{N}\bar{w}-\frac{1}{2}B^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}\big{)}$
$\displaystyle=(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(h_{N-1}+l_{N-1})$
$\displaystyle-2(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(h_{N-1}+l_{N-1}\big{)}$
$\displaystyle=-(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(h_{N-1}+l_{N-1}\big{)}.$
(70)

Thus, we may set

$\displaystyle
V_{N-1}=A^{\top}V_{N}+Q_{\mu_{s,o}}-A^{\top}V_{N}B(B^{\top}V_{N}B+R)^{-1}B^{\top}V_{N}A,$
$\displaystyle T_{N-1}=(A+BK_{N-1})^{\top}V_{N},$ $\displaystyle
S_{N-1}=(A+BK_{N-1})^{\top}S_{N}+I,$ $\displaystyle
p_{N-1}=p_{N}+\bar{w}^{\top}V_{N}\bar{w}+\mathrm{tr}(V_{N}W)+\bar{w}^{\top}S_{N}\mathrm{M}_{\mu_{s,o}}+\mathrm{tr}(V_{N}\Sigma_{N|N-1})-2\mathrm{tr}(V_{N}H_{N|N-1})+\mathrm{tr}(V_{N}\Sigma_{N-1|N-1})$
$\displaystyle-(h_{N-1}+l_{N-1})^{\top}(B^{\top}V_{N}B+R)(h_{N-1}+l_{N-1}\big{)},$
(71)

and write the optimal cost-to-go at $t=N-1$ as

$\displaystyle\mathcal{L}^{*}(\mathscr{F}_{N-1};\mu_{s,o})=\widehat{x}^{\top}_{N-1|N-1}V_{N-1}\widehat{x}_{N-1|N-1}+2\widehat{x}^{\top}_{N-1|N-1}T_{N-1}\bar{w}+\widehat{x}^{\top}_{N-1|N-1}S_{N-1}\mathrm{M}_{\mu_{s,o}}+p_{N-1}.$
(72)

to conclude the proof for the root. The transition of truthfulness follows
from similar steps.

## References

  * [1] S.-K. Kim, R. Thakker, and A.-A. Agha-Mohammadi, “Bi-directional value learning for risk-aware planning under uncertainty,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 3, pp. 2493–2500, 2019.
  * [2] A. A. Pereira, J. Binney, G. A. Hollinger, and G. S. Sukhatme, “Risk-aware path planning for autonomous underwater vehicles using predictive ocean models,” _Journal of Field Robotics_ , vol. 30, no. 5, pp. 741–762, 2013\.
  * [3] W.-J. Ma, C. Oh, Y. Liu, D. Dentcheva, and M. M. Zavlanos, “Risk-averse access point selection in wireless communication networks,” _IEEE Transactions on Control of Network Systems_ , vol. 6, no. 1, pp. 24–36, 2018.
  * [4] M. Bennis, M. Debbah, and H. V. Poor, “Ultrareliable and low-latency wireless communication: Tail, risk, and scale,” _Proceedings of the IEEE_ , vol. 106, no. 10, pp. 1834–1853, 2018.
  * [5] T. Iwaki, J. Wu, Y. Wu, H. Sandberg, and K. H. Johansson, “Multi-hop sensor network scheduling for optimal remote estimation,” _Automatica_ , vol. 127, p. 109498, 2021.
  * [6] D. D. Fan, J. Nguyen, R. Thakker, N. Alatur, A.-a. Agha-mohammadi, and E. A. Theodorou, “Bayesian learning-based adaptive control for safety critical systems,” in _2020 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2020, pp. 4093–4099.
  * [7] L. Lindemann, N. Matni, and G. J. Pappas, “STL Robustness Risk over Discrete-Time Stochastic Processes,” _arXiv preprint arXiv:2104.01503_ , 2021.
  * [8] A. R. Cardoso and H. Xu, “Risk-Averse Stochastic Convex Bandit,” in _International Conference on Artificial Intelligence and Statistics_ , vol. 89, Apr. 2019, pp. 39–47.
  * [9] D. Dentcheva, S. Penev, and A. Ruszczyński, “Statistical estimation of composite risk functionals and risk optimization problems,” _Annals of the Institute of Statistical Mathematics_ , vol. 69, no. 4, pp. 737–760, 2017\.
  * [10] A. Tsiamis, D. S. Kalogerias, A. Ribeiro, and G. J. Pappas, “Linear Quadratic Control with Risk Constraints,” _arXiv preprint arXiv:2112.07564_ , 2021.
  * [11] P. Whittle, “Risk-Sensitive Linear/Quadratic/Gaussian Control,” _Advances in Applied Probability_ , vol. 13, no. 4, pp. 764–777, 1981.
  * [12] B. D. Anderson and J. B. Moore, _Optimal filtering_. Courier Corporation, 2012.
  * [13] L. A. Prashanth and M. Fu, “Risk-Sensitive Reinforcement Learning: A Constrained Optimization Viewpoint,” _arXiv preprint, arXiv:1810.09126_ , Oct. 2018.
  * [14] W. Huang and W. B. Haskell, “Risk-Aware Q-learning for Markov Decision Processes,” in _2017 IEEE 56th Annual Conference on Decision and Control, CDC 2017_ , vol. 2018-Janua. IEEE, Dec. 2018, pp. 4928–4933.
  * [15] D. R. Jiang and W. B. Powell, “Risk-Averse Approximate Dynamic Programming with Quantile-Based Risk Measures,” _Mathematics of Operations Research_ , vol. 43, no. 2, pp. 554–579, Nov. 2018.
  * [16] D. S. Kalogerias and W. B. Powell, “Recursive Optimization of Convex Risk Measures: Mean-Semideviation Models,” _arXiv preprint, arXiv:1804.00636_ , Apr. 2018.
  * [17] A. Tamar, Y. Chow, M. Ghavamzadeh, and S. Mannor, “Sequential Decision Making with Coherent Risk,” _IEEE Transactions on Automatic Control_ , vol. 62, no. 7, pp. 3323–3338, Jul. 2017.
  * [18] C. A. Vitt, D. Dentcheva, and H. Xiong, “Risk-Averse Classification,” _Annals of Operations Research_ , Aug. 2019.
  * [19] L. Zhou and P. Tokekar, “An Approximation Algorithm for Risk-averse Submodular Optimization,” _arXiv preprint, arXiv:1807.09358_ , Jul. 2018\.
  * [20] A. Ruszczyński, “Risk-Averse Dynamic Programming for Markov Decision Processes,” _Mathematical Programming_ , vol. 125, no. 2, pp. 235–261, Oct. 2010.
  * [21] P. Sopasakis, D. Herceg, A. Bemporad, and P. Patrinos, “Risk-averse model predictive control,” _Automatica_ , vol. 100, pp. 281 – 288, 2019.
  * [22] M. P. Chapman, J. Lacotte, A. Tamar, D. Lee, K. M. Smith, V. Cheng, J. F. Fisac, S. Jha, M. Pavone, and C. J. Tomlin, “A Risk-Sensitive Finite-Time Reachability Approach for Safety of Stochastic Dynamic Systems,” in _2019 American Control Conference (ACC)_ , 2019, pp. 2958–2963.
  * [23] A. Shapiro, D. Dentcheva, and A. Ruszczynski, _Lectures on stochastic programming: modeling and theory_. SIAM, 2021.
  * [24] H. Markowitz, “Portfolio Selection,” _The Journal of Finance_ , vol. 7, no. 1, pp. 77–91, Mar. 1952.
  * [25] R. T. Rockafellar and S. Uryasev, “Optimization of Conditional Value-at-Risk,” _Journal of Risk_ , vol. 2, pp. 21–41, 1997.
  * [26] M. P. Chapman and K. M. Smith, “Classical risk-averse control for a finite-horizon borel model,” _IEEE Control Systems Letters_ , vol. 6, pp. 1525–1530, 2021.
  * [27] A. Tsiamis, D. S. Kalogerias, L. F. Chamon, A. Ribeiro, and G. J. Pappas, “Risk-Constrained Linear-Quadratic Regulators,” in _2020 59th IEEE Conference on Decision and Control (CDC)_. IEEE, 2020, pp. 3040–3047.
  * [28] M. Abeille, A. Lazaric, X. Brokmann _et al._ , “Lqg for portfolio optimization,” _Available at SSRN 2863925_ , 2016.
  * [29] J. L. Speyer and W. H. Chung, _Stochastic Processes, Estimation, and Control_. Siam, 2008, vol. 17.
  * [30] A. Ruszczynski, _Nonlinear Optimization_. Princeton university press, 2011.
  * [31] K. J. Åström and R. M. Murray, _Feedback systems_. Princeton university press, 2010.