1,$ $\displaystyle\delta_{\mathbf{x}_{0}}=1,$

$\alpha=\frac{c}{1+c}$ for a control regularization $c$.
$\gamma_{t}\vcentcolon=\max\limits_{s\leq
t}\big{(}1+\kappa(\overline{\boldsymbol{\theta}}_{s})^{2}\big{)}\lVert\mathbf{I}-\overline{\boldsymbol{\theta}}_{s}^{T}\overline{\boldsymbol{\theta}}_{s}\rVert_{2}$,

  * •

$\overline{\boldsymbol{\theta}}_{t}\vcentcolon=\boldsymbol{\theta}_{t-1}\cdots\boldsymbol{\theta}_{0},~{}t\geq
1$,

  * •

$\overline{\boldsymbol{\theta}}_{0}\vcentcolon=\mathbf{I},~{}t=0$.

Proof  we subtract both sides of Eq. (C.2) by
$\boldsymbol{\theta}_{t}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}$,
and recall the definition of linearization error $e_{t}$,

$e_{t+1}\leq\lVert\boldsymbol{\theta}_{t}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t}})\rVert_{2}\cdot
e_{t}+\lVert\boldsymbol{\theta}_{t}\rVert_{2}\cdot\lVert\mathbf{u}^{\mathcal{M}_{t}}(\overline{\mathbf{x}}_{\epsilon,t})-\mathbf{u}_{\mathbf{x}_{t}}^{P}(\overline{\mathbf{x}}_{\epsilon,t})\rVert_{2}+\frac{1}{2}\lVert
F^{\prime\prime}_{t}(\mathbf{p})\rVert_{\ast}\cdot\lVert\overline{\mathbf{x}}_{\epsilon,t}+\mathbf{u}^{\mathcal{M}_{t}}(\overline{\mathbf{x}}_{\epsilon,t})-\mathbf{x}_{t}\rVert_{2}^{2}.$

Let us simplify the above inequality.

  * •

The orthogonal projection admits
$\lVert\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t}}\rVert_{2}\leq 1$.

  * •

Recall Proposition 8,

$\lVert\mathbf{u}^{\mathcal{M}_{t}}(\overline{\mathbf{x}}_{\epsilon,t})-\mathbf{u}_{\mathbf{x}_{t}}^{P}(\overline{\mathbf{x}}_{\epsilon,t})\rVert_{2}\leq
4\sigma_{t}(1+2\sigma_{t})\cdot\lVert\overline{\mathbf{x}}_{\epsilon,t}-\mathbf{x}_{t}\rVert_{2}^{2},$

where $\sigma_{t}$ is the uniform upper bound on $\lVert
f_{t}^{\prime\prime}(\mathbf{x})\rVert_{\ast}$. We denote

$\displaystyle k_{t}=4\sigma_{t}(1+2\sigma_{t}),$
$\displaystyle\lVert\mathbf{u}^{\mathcal{M}_{t}}(\overline{\mathbf{x}}_{\epsilon,t})-\mathbf{u}_{\mathbf{x}_{t}}^{P}(\overline{\mathbf{x}}_{\epsilon,t})\rVert_{2}\leq
k_{t}\cdot\lVert\overline{\mathbf{x}}_{\epsilon,t}-\mathbf{x}_{t}\rVert_{2}^{2}.$

  * •

$F_{t}(\cdot)$ admits an uniform upper bound $\beta_{t}$ such that
$\sup_{\mathbf{x}\in\mathbb{R}^{d}}\lVert
F^{\prime\prime}_{t}(\mathbf{x})\rVert_{\ast}\leq\beta_{t}$.

  * •

Recall the inequality in Eq. (• ‣ C.1),
$\lVert\mathbf{u}^{\mathcal{M}_{t}}(\overline{\mathbf{x}}_{\epsilon,t})\rVert_{2}\leq\lVert\overline{\mathbf{x}}_{\epsilon,t}-\mathbf{x}_{t}\rVert_{2}$,

$\displaystyle\lVert\overline{\mathbf{x}}_{\epsilon,t}+\mathbf{u}^{\mathcal{M}_{t}}(\overline{\mathbf{x}}_{\epsilon,t})-\mathbf{x}_{t}\rVert_{2}^{2}$
$\displaystyle\leq
2\cdot\lVert\overline{\mathbf{x}}_{\epsilon,t}-\mathbf{x}_{t}\rVert_{2}^{2}+2\cdot\lVert\mathbf{u}^{\mathcal{M}_{t}}(\overline{\mathbf{x}}_{\epsilon,t})\rVert_{2}^{2},$
$\displaystyle\leq
4\cdot\lVert\overline{\mathbf{x}}_{\epsilon,t}-\mathbf{x}_{t}\rVert_{2}^{2}.$

.

Therefore,

$e_{t+1}\leq\lVert\boldsymbol{\theta}_{t}\rVert_{2}e_{t}+(k_{t}\lVert\boldsymbol{\theta}_{t}\rVert_{2}+2\beta_{t})\cdot\lVert\overline{\mathbf{x}}_{\epsilon,t}-\mathbf{x}_{t}\rVert_{2}^{2}.$

Furthermore,

$\displaystyle\;\;\;\;\;\lVert\overline{\mathbf{x}}_{\epsilon,t}-\mathbf{x}_{t}\rVert_{2}^{2}$
$\displaystyle=\lVert\overline{\mathbf{x}}_{\epsilon,t}-\mathbf{x}_{t}-\boldsymbol{\theta}_{t-1}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t-1}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}+\boldsymbol{\theta}_{t-1}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t-1}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}\rVert_{2}^{2},$
$\displaystyle\leq
e_{t}^{2}+\lVert\boldsymbol{\theta}_{t-1}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t-1}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}\rVert_{2}^{2}.$

Then, the linearization error can be bounded as follows,

$e_{t+1}\leq\lVert\boldsymbol{\theta}_{t}\rVert_{2}e_{t}+(k_{t}\lVert\boldsymbol{\theta}_{t}\rVert_{2}+2\beta_{t})e_{t}^{2}+(k_{t}\lVert\boldsymbol{\theta}_{t}\rVert_{2}+2\beta_{t})\cdot\lVert\boldsymbol{\theta}_{t-1}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t-1}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}\rVert_{2}^{2}.$

We can express the initial perturbation as $\mathbf{z}=\epsilon\mathbf{v}$,
where $\epsilon$ is perturbation magnitude and $\mathbf{v}$ is a unit vector
that represents the perturbation direction. The perturbation direction
$\mathbf{v}$ admits a direct sum such that
$\mathbf{v}=\mathbf{v}^{\parallel}\oplus\mathbf{v}^{\perp}$, where
$\mathbf{v}^{\parallel}\in\mathcal{T}_{\mathbf{x}_{0}}\mathcal{M}_{0}$ and
$\mathbf{v}^{\perp}$ lies in the orthogonal complement of
$\mathcal{T}_{\mathbf{x}_{0}}\mathcal{M}_{0}$.

Recall Theorem 1,

$\displaystyle\;\;\;\;\;\lVert\boldsymbol{\theta}_{t-1}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t-1}})\boldsymbol{\theta}_{t-2}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t-2}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}\rVert_{2}^{2},$
$\displaystyle\leq\lVert\boldsymbol{\theta}_{t-1}\cdots\boldsymbol{\theta}_{0}\rVert_{2}^{2}\cdot\bigg{(}\alpha^{2t}\lVert\mathbf{z}^{\perp}\rVert_{2}^{2}+\lVert\mathbf{z}^{\parallel}\rVert_{2}^{2}+\gamma_{t}\lVert\mathbf{z}\rVert_{2}^{2}\big{(}\gamma_{t}\alpha^{2}(1-\alpha^{t-1})^{2}+2(\alpha-\alpha^{t})\big{)}\bigg{)},$
$\displaystyle\leq\lVert\boldsymbol{\theta}_{t-1}\cdots\boldsymbol{\theta}_{0}\rVert_{2}^{2}\cdot\bigg{(}\alpha^{2t}\lVert\mathbf{v}^{\perp}\rVert_{2}^{2}+\lVert\mathbf{v}^{\parallel}\rVert_{2}^{2}+\gamma_{t}\lVert\mathbf{v}\rVert_{2}^{2}\big{(}\gamma_{t}\alpha^{2}(1-\alpha^{t-1})^{2}+2(\alpha-\alpha^{t})\big{)}\bigg{)}\epsilon^{2},$

where $\alpha=\frac{c}{1+c}$ for a control regularization $c$.
$\gamma_{t}\vcentcolon=\max\limits_{s\leq
t}\big{(}1+\kappa(\overline{\boldsymbol{\theta}}_{s})^{2}\big{)}\lVert\mathbf{I}-\overline{\boldsymbol{\theta}}_{s}^{T}\overline{\boldsymbol{\theta}}_{s}\rVert_{2}$,

  * •

$\overline{\boldsymbol{\theta}}_{t}\vcentcolon=\boldsymbol{\theta}_{t-1}\cdots\boldsymbol{\theta}_{0},~{}t\geq
1$,

  * •

$\overline{\boldsymbol{\theta}}_{0}\vcentcolon=\mathbf{I},~{}t=0$.

Let
$\delta_{\mathbf{x}_{t}}=\lVert\boldsymbol{\theta}_{t-1}\cdots\boldsymbol{\theta}_{0}\rVert_{2}^{2}\cdot\bigg{(}\alpha^{2t}\lVert\mathbf{v}^{\perp}\rVert_{2}^{2}+\lVert\mathbf{v}^{\parallel}\rVert_{2}^{2}+\gamma_{t}\lVert\mathbf{v}\rVert_{2}^{2}\big{(}\gamma_{t}\alpha^{2}(1-\alpha^{t-1})^{2}+2(\alpha-\alpha^{t})\big{)}\bigg{)}$
for $t\geq 1$, and $\delta_{\mathbf{x}_{0}}=1$, the linearization error
$e_{t+1}$ can be upper bounded by

$e_{t+1}\leq\lVert\boldsymbol{\theta}_{t}\rVert_{2}e_{t}+(k_{t}\lVert\boldsymbol{\theta}_{t}\rVert_{2}+2\beta_{t})e_{t}^{2}+(k_{t}\lVert\boldsymbol{\theta}_{t}\rVert_{2}+2\beta_{t})\cdot\delta_{\mathbf{x}_{t}}\cdot\epsilon^{2}.$

Since $e_{t}$ is defined for $t\geq 1$, the following derives a upper bound on
$e_{1}$. When $t=1$, recall the initial perturbation
$\overline{\mathbf{x}}_{\epsilon,0}-\mathbf{x}_{0}=\mathbf{z}$,

$\displaystyle\overline{\mathbf{x}}_{\epsilon,1}-\mathbf{x}_{1}$
$\displaystyle=F_{0}(\overline{\mathbf{x}}_{\epsilon,0}+\mathbf{u}_{0}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0}))-F_{0}(\mathbf{x}_{0}),$
$\displaystyle=\boldsymbol{\theta}_{0}(\overline{\mathbf{x}}_{\epsilon,0}+\mathbf{u}_{\mathbf{x}_{0}}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0})-\mathbf{x}_{0})+\frac{1}{2}F_{0}^{\prime\prime}(\mathbf{p})^{i,j,k}(\overline{\mathbf{x}}_{\epsilon,0}+\mathbf{u}_{\mathbf{x}_{0}}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0})-\mathbf{x}_{0})_{j}(\overline{\mathbf{x}}_{\epsilon,0}+\mathbf{u}_{0}^{\mathcal{M}}-\mathbf{x}_{0})_{k},$
$\displaystyle=\mbox{$\displaystyle\boldsymbol{\theta}_{0}(\mathbf{z}+\mathbf{u}_{\mathbf{x}_{0}}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0}))+\frac{1}{2}F_{0}^{\prime\prime}(\mathbf{p})^{i,j,k}(\mathbf{z}+\mathbf{u}_{0}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0}))_{j}(\mathbf{z}+\mathbf{u}_{0}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0}))_{k},$}$
$\displaystyle=\boldsymbol{\theta}_{0}(\mathbf{z}+\mathbf{u}_{0}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0})-\mathbf{u}_{0}^{P}(\overline{\mathbf{x}}_{\epsilon,0})+\mathbf{u}_{0}^{P}(\overline{\mathbf{x}}_{\epsilon,0}))+\frac{1}{2}F_{0}^{\prime\prime}(\mathbf{p})^{i,j,k}(\mathbf{z}+\mathbf{u}_{0}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0}))_{j}(\mathbf{z}+\mathbf{u}_{0}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0}))_{k},$
$\displaystyle=\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}+\boldsymbol{\theta}_{0}(\mathbf{u}_{0}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0})-\mathbf{u}_{0}^{P}(\overline{\mathbf{x}}_{\epsilon,0}))+\frac{1}{2}F_{0}^{\prime\prime}(\mathbf{p})^{i,j,k}(\mathbf{z}+\mathbf{u}_{0}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0}))_{j}(\mathbf{z}+\mathbf{u}_{0}^{\mathcal{M}}(\overline{\mathbf{x}}_{\epsilon,0}))_{k}.$

By following the same procedure as the derivation of $e_{t+1}$,

$\displaystyle
e_{1}\leq(k_{\mathbf{x}_{0}}\lVert\boldsymbol{\theta}_{0}\rVert_{2}+2\beta_{0})\cdot\delta_{\mathbf{x}_{0}}\cdot\epsilon^{2}.$

The following proposition solves the difference inequality of linearization
error.

###### Proposition 10

If the perturbation satisfies

$\epsilon^{2}\leq\frac{1}{\bigg{(}\sum_{i=0}^{T-1}\delta_{\mathbf{x}_{i}}(k_{\mathbf{x}_{i}}\lVert\boldsymbol{\theta}_{i}\rVert_{2}+2\beta_{i})\prod_{j=i+1}^{T-1}(\lVert\boldsymbol{\theta}_{j}\rVert_{2}+k_{\mathbf{x}_{j}}\lVert\boldsymbol{\theta}_{j}\rVert_{2}+2\beta_{j})\bigg{)}}.$

for $t\leq T$, the linearization error can be upper bounded by

$e_{t}\leq\bigg{(}\sum_{i=0}^{t-1}\delta_{\mathbf{x}_{i}}(k_{\mathbf{x}_{i}}\lVert\boldsymbol{\theta}_{i}\rVert_{2}+2\beta_{i})\prod_{j=i+1}^{t-1}(\lVert\boldsymbol{\theta}_{j}\rVert_{2}+k_{\mathbf{x}_{j}}\lVert\boldsymbol{\theta}_{j}\rVert_{2}+2\beta_{j})\bigg{)}\epsilon^{2}.$

Proof  We prove it by induction on $t$ up to some $T$, such that $t\leq T$. We
restrict the magnitude of initial perturbation
$\lVert\mathbf{z}\rVert_{2}^{2}\leq\epsilon_{T}$ for some constant
$\epsilon_{T}$, such that the error $e_{t}\leq 1$ for all $t\leq T$. The
expression of $\epsilon_{T}$ is derived later.

When $t=1$,

$\displaystyle
e_{1}\leq(k_{\mathbf{x}_{0}}\lVert\boldsymbol{\theta}_{0}\rVert_{2}+2\beta_{0})\cdot\delta_{\mathbf{x}_{0}}\cdot\epsilon^{2},$

which agrees with Proposition 9.

Suppose that it is true for some $t\leq T-1$, such that

$e_{t}\leq\bigg{(}\sum_{i=0}^{t-1}\delta_{\mathbf{x}_{i}}(k_{\mathbf{x}_{i}}\lVert\boldsymbol{\theta}_{i}\rVert_{2}+2\beta_{i})\prod_{j=i+1}^{t-1}(\lVert\boldsymbol{\theta}_{j}\rVert_{2}+k_{\mathbf{x}_{j}}\lVert\boldsymbol{\theta}_{j}\rVert_{2}+2\beta_{j})\bigg{)}\epsilon^{2}.$

Then at $t+1$, recall Proposition 9, given that $e_{t}\leq 1$ for all $t\leq
T$,

$\displaystyle e_{t+1}$
$\displaystyle\leq\lVert\boldsymbol{\theta}_{t}\rVert_{2}e_{t}+(k_{t}\lVert\boldsymbol{\theta}_{t}\rVert_{2}+2\beta_{t})e_{t}^{2}+(k_{t}\lVert\boldsymbol{\theta}_{t}\rVert_{2}+2\beta_{t})\cdot\delta_{\mathbf{x}_{t}}\cdot\epsilon^{2},$
$\displaystyle\leq(\lVert\boldsymbol{\theta}_{t}\rVert_{2}+k_{t}\lVert\boldsymbol{\theta}_{t}\rVert_{2}+2\beta_{t})e_{t}+(k_{t}\lVert\boldsymbol{\theta}_{t}\rVert_{2}+2\beta_{t})\cdot\delta_{\mathbf{x}_{t}}\cdot\epsilon^{2},$
$\displaystyle\leq(\lVert\boldsymbol{\theta}_{t}\rVert_{2}+k_{t}\lVert\boldsymbol{\theta}_{t}\rVert_{2}+2\beta_{t})\bigg{(}\sum_{i=0}^{t-1}\delta_{\mathbf{x}_{i}}(k_{\mathbf{x}_{i}}\lVert\boldsymbol{\theta}_{i}\rVert_{2}+2\beta_{i})\prod_{j=i+1}^{t-1}(\lVert\boldsymbol{\theta}_{j}\rVert_{2}+k_{\mathbf{x}_{j}}\lVert\boldsymbol{\theta}_{j}\rVert_{2}+2\beta_{j})\bigg{)}\epsilon^{2}$
$\displaystyle\;\;\;\;\;+(k_{t}\lVert\boldsymbol{\theta}_{t}\rVert_{2}+2\beta_{t})\cdot\delta_{\mathbf{x}_{t}}\cdot\epsilon^{2},$
$\displaystyle=\bigg{(}\sum_{i=0}^{t}\delta_{\mathbf{x}_{i}}(k_{\mathbf{x}_{i}}\lVert\boldsymbol{\theta}_{i}\rVert_{2}+2\beta_{i})\prod_{j=i+1}^{t}(\lVert\boldsymbol{\theta}_{j}\rVert_{2}+k_{\mathbf{x}_{j}}\lVert\boldsymbol{\theta}_{j}\rVert_{2}+2\beta_{j})\bigg{)}\epsilon^{2}.$

We have restricted the initial perturbation
$\lVert\mathbf{z}\rVert_{2}^{2}=\epsilon^{2}\leq\epsilon_{T}$, for some
constant $\epsilon_{T}$, such that $e_{t}\leq 1$, for all $t\leq T$.

For $t\leq T$,

$\displaystyle e_{t}$ $\displaystyle\leq e_{T},$
$\displaystyle\leq\bigg{(}\sum_{i=0}^{T-1}\delta_{\mathbf{x}_{i}}(k_{\mathbf{x}_{i}}\lVert\boldsymbol{\theta}_{i}\rVert_{2}+2\beta_{i})\prod_{j=i+1}^{T-1}(\lVert\boldsymbol{\theta}_{j}\rVert_{2}+k_{\mathbf{x}_{j}}\lVert\boldsymbol{\theta}_{j}\rVert_{2}+2\beta_{j})\bigg{)}\epsilon^{2},$
$\displaystyle\leq\bigg{(}\sum_{i=0}^{T-1}\delta_{\mathbf{x}_{i}}(k_{\mathbf{x}_{i}}\lVert\boldsymbol{\theta}_{i}\rVert_{2}+2\beta_{i})\prod_{j=i+1}^{T-1}(\lVert\boldsymbol{\theta}_{j}\rVert_{2}+k_{\mathbf{x}_{j}}\lVert\boldsymbol{\theta}_{j}\rVert_{2}+2\beta_{j})\bigg{)}\epsilon_{T},$
$\displaystyle=1,$

therefore,

$\epsilon_{T}=\frac{1}{\bigg{(}\sum_{i=0}^{T-1}\delta_{\mathbf{x}_{i}}(k_{\mathbf{x}_{i}}\lVert\boldsymbol{\theta}_{i}\rVert_{2}+2\beta_{i})\prod_{j=i+1}^{T-1}(\lVert\boldsymbol{\theta}_{j}\rVert_{2}+k_{\mathbf{x}_{j}}\lVert\boldsymbol{\theta}_{j}\rVert_{2}+2\beta_{j})\bigg{)}}.$

Proposition 10 provides several intuitions.

  * •

the linearization error is of $\mathcal{O}(\epsilon^{2})$ when the data
perturbation is small, where $\epsilon$ is the magnitude of the data
perturbation.

  * •

the linearization error becomes smaller when the nonlinear transformation
$F_{t}(\cdot)$ behaves more linearily ($\beta_{t}$ decreases), and the
curvature of embedding manifold is smoother ($k_{t}$ decreases). Specifically,
in the linear case, $\beta_{t}$ and $k_{t}$ become $0$, which results in no
linearization error.

  * •

the linearization becomes smaller when the initial perturbation lies in a
lower dimensional manifold ($\delta_{\mathbf{x}_{t}}$ decreases).

###  C.3 Error Estimation

Now we reach the main theorem on the error estimation of
$\lVert\overline{\mathbf{x}}_{\epsilon,t}-\mathbf{x}_{t}\rVert$.

See 2

Proof  recall that
$e_{t+1}=\lVert(\overline{\mathbf{x}}_{\epsilon,t+1}-\mathbf{x}_{t+1})-\boldsymbol{\theta}_{t}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}\rVert_{2}$,

$\displaystyle\;\;\;\;\;\lVert\overline{\mathbf{x}}_{\epsilon,t+1}-\mathbf{x}_{t+1}\rVert_{2}$
$\displaystyle=\lVert\overline{\mathbf{x}}_{\epsilon,t+1}-\mathbf{x}_{t+1}-\boldsymbol{\theta}_{t}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}+\boldsymbol{\theta}_{t}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}\rVert_{2},$
$\displaystyle\leq\lVert\boldsymbol{\theta}_{t}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}\rVert_{2}+\lVert\overline{\mathbf{x}}_{\epsilon,t+1}-\mathbf{x}_{t+1}-\boldsymbol{\theta}_{t}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}\rVert_{2},$
$\displaystyle=\lVert\boldsymbol{\theta}_{t}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}\rVert_{2}+e_{t+1}.$

Recall Theorem 1,

$\displaystyle\;\;\;\;\;\lVert\boldsymbol{\theta}_{t}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{t}})\cdots\boldsymbol{\theta}_{0}(\mathbf{I}-\mathbf{K}_{\mathbf{x}_{0}})\mathbf{z}\rVert_{2}$
$\displaystyle\leq\bigg{(}\lVert\overline{\boldsymbol{\theta}}_{t+1}\rVert_{2}^{2}\cdot\bigg{(}\alpha^{2(t+1)}\lVert\mathbf{z}^{\perp}\rVert_{2}^{2}+\lVert\mathbf{z}^{\parallel}\rVert_{2}^{2}+\gamma_{t+1}\lVert\mathbf{z}\rVert_{2}^{2}\big{(}\gamma_{t+1}\alpha^{2}(1-\alpha^{t})^{2}+2(\alpha-\alpha^{t+1})\big{)}\bigg{)}\bigg{)}^{\frac{1}{2}},$
$\displaystyle\leq\lVert\overline{\boldsymbol{\theta}}_{t+1}\rVert_{2}\cdot\bigg{(}\alpha^{t+1}\lVert\mathbf{z}^{\perp}\rVert_{2}+\lVert\mathbf{z}^{\parallel}\rVert_{2}+\lVert\mathbf{z}\rVert_{2}\big{(}\gamma_{t+1}\alpha(1-\alpha^{t})+\sqrt{2\gamma_{t+1}(\alpha-\alpha^{t+1})}\big{)}\bigg{)},$

where
$\overline{\boldsymbol{\theta}}_{t+1}=\boldsymbol{\theta}_{t}\boldsymbol{\theta}_{t-1}\cdots\boldsymbol{\theta}_{0}$.

Recall Proposition 10 for the linearization error,

$e_{t+1}\leq\bigg{(}\sum_{i=0}^{t}\delta_{\mathbf{x}_{i}}(k_{\mathbf{x}_{i}}\lVert\boldsymbol{\theta}_{i}\rVert+2\beta_{i})\prod_{j=i+1}^{t}(\lVert\boldsymbol{\theta}_{j}\rVert_{2}+k_{\mathbf{x}_{j}}\lVert\boldsymbol{\theta}_{j}\rVert_{2}+2\beta_{j})\bigg{)}\epsilon^{2}.$

Therefore, for $t\geq 1$,

$\displaystyle\lVert\overline{\mathbf{x}}_{\epsilon,t+1}-\mathbf{x}_{t+1}\rVert_{2}$
$\displaystyle\leq\mbox{$\displaystyle\lVert\overline{\boldsymbol{\theta}}_{t+1}\rVert_{2}\bigg{(}\alpha^{t+1}\lVert\mathbf{z}^{\perp}\rVert_{2}+\lVert\mathbf{z}^{\parallel}\rVert_{2}+\lVert\mathbf{z}\rVert_{2}\big{(}\gamma_{t+1}\alpha(1-\alpha^{t})+\sqrt{2\gamma_{t+1}(\alpha-\alpha^{t+1})}\big{)}\bigg{)}$}$
$\displaystyle\;\;\;\;+\bigg{(}\sum_{i=0}^{t}\delta_{\mathbf{x}_{i}}(k_{\mathbf{x}_{i}}\lVert\boldsymbol{\theta}_{i}\rVert_{2}+2\beta_{i})\prod_{j=i+1}^{t}(\lVert\boldsymbol{\theta}_{j}\rVert_{2}+k_{\mathbf{x}_{j}}\lVert\boldsymbol{\theta}_{j}\rVert_{2}+2\beta_{j})\bigg{)}\epsilon^{2}.$

##  D Additional Numerical Experiments

###  D.1 Linear Closed-loop Control

Here, we consider the closed-loop control method in linear setting.
Specifically, the embedding manifolds are linear subspaces, and the embedding
functions are orthogonal projections onto those linear subspaces. We follow
the same experimental setting as in Sec 6 for baseline models, robustness
evaluations and PMP parameters. We perform principle component analysis on
clean training data to obtain embedding subspaces and embedding functions.

Table 6: CIFAR-10 accuracy measure: baseline model / linear control
$\ell_{\infty}:\epsilon=8/255$, $\ell_{2}:\epsilon=0.5$,
$\ell_{1}:\epsilon=12$
---
Standard models
|  None |  AA ($\ell_{\infty}$) |  AA ($\ell_{2}$) |  AA ($\ell_{1}$)
RN-18 |  94.71 / 87.04 |  0\. / 59.98 |  0\. / 73.01 |  0\. / 73.04
RN-34 |  94.91 / 87.13 |  0\. / 62.64 |  0\. / 75.28 |  0\. / 74.89
RN-50 |  95.08 / 87.83 |  0\. / 61.63 |  0\. / 75.37 |  0\. / 75.39
WRN-28-8 |  95.41 / 87.72 |  0\. / 68.11 |  0\. / 77.97 |  0\. / 78.33
WRN-34-8 |  94.05 / 88.06 |  0\. / 50.1 |  0\. / 67.16 |  0\. / 66.64
Robust models
RN-$18$ |  82.39 / 81.0 |  48.72 / 53.06 |  58.8 / 69.94 |  9.86 / 50.41
RN-$34$ |  84.45 / 83.0 |  49.31 / 52.75 |  57.27 / 70.51 |  7.21 / 51.44
RN-$50$ |  83.99 / 82.99 |  48.68 / 52.23 |  57.25 / 70.31 |  6.83 / 51.26
WRN-$28$-$8$ |  85.09 / 84.04 |  48.13 / 51.09 |  54.38 / 70.0 |  5.38 / 54.56
WRN-$34$-$8$ |  84.95 / 83.7 |  48.47 / 51.4 |  54.36 / 70.31 |  4.67 / 55.02

As shown in Table 6, on CIFAR-10 dataset, applying linear control on both
standard trained and robustly trained baseline models can consistently improve
the robustness against various perturbations. Notably, the controlled models
have more than $60\%$, $80\%$ and $70\%$ accuracies against autoattack
measured by $\ell_{\infty},\ell_{2}$ and $\ell_{1}$ norms respectively.

###  D.2 Robustness Improvement Under White-box Setting

Here, we test our method in a fully white-box setting, where an attacker has
full access to both pre-trained models and our control method. We summarize
our experimental settings below.

  * •

Baseline models. we use Pre-activated ResNet-$18$ (RN-18), -$34$ (RN-34),
-$50$ (RN-50) as the testing benchmarks. All baseline models are trained with
TRADES (Zhang et al., 2019a).

  * •

Robustness evaluations. In the white-box setting, the objective function of an
attack algorithm is defined as follows,

$\tilde{\mathbf{x}}_{\epsilon}=\arg\max\limits_{\tilde{\mathbf{x}}_{0}\sim\mathcal{B}(\mathbf{x}_{0},\epsilon)}{\rm
CE}(\tilde{\mathbf{x}}_{T},\mathbf{q}),\quad{\rm
s.t.}\;\tilde{\mathbf{x}}_{t+1}=F_{t}(\tilde{\mathbf{x}}_{t}+\mathbf{u}_{t}^{\mathcal{M}}).$

  * •

Embedding functions. We use a $2$-layer denoising auto-encoder to realize the
embedding function. In the white-box setting, we designing the embedding
function to have access to both model and attack information. The training
objective function of the embedding function at the $t^{th}$ layer is

$\displaystyle\mathcal{E}_{t}^{\ast}=\arg\min\limits_{\mathcal{E}_{t}}\max\limits_{\tilde{\mathbf{x}}_{0}\sim\mathcal{B}(\mathbf{x}_{0},\epsilon)}\frac{1}{N}\sum_{i=1}^{N}{\rm
CE}(\tilde{\mathbf{x}}_{i,T},\mathbf{q}_{i})+\lVert\mathcal{E}_{t}(\tilde{\mathbf{x}}_{t})-\mathbf{x}_{t}\rVert_{2}^{2},$
$\displaystyle{\rm
s.t.}\;\tilde{\mathbf{x}}_{t+1}=F_{t}\circ\mathcal{E}_{t}(\tilde{\mathbf{x}}_{t}),\;\;\;\;\mathbf{x}_{t+1}=F_{t}(\mathbf{x}_{t}),$

where $\mathcal{B}(\mathbf{x},\epsilon)$ is a set of points centered at
$\mathbf{x}$ with radius $\epsilon$ measured by the $\ell_{\infty}$ norm. The
above objective function enforces the embedding function to encode an
embedding manifold as a set of states that result in high robustness. The
second term penalizes the magnitude of applied control adjustment.

Table 7: Accuracy measure: baseline / controlled $\ell_{\infty}$ perturbation
with $\epsilon=8/255$
---
|  None |  PGD |  AA
RN-18 |  82.39 / 82.34 |  51.66 / 51.82 |  48.72 / 48.77
RN-34 |  84.45 / 84.12 |  51.35 / 51.87 |  49.31 / 49.96
RN-50 |  83.99 / 83.73 |  50.88 / 50.91 |  48.68 / 48.98

Although in practice, the control information is not released to the public,
we evaluate the performance of the proposed framework in this worst case.
Table 7 shows that applying closed-loop control method can consistently
improve the robustness of all adversarially pre-trained baseline models.
Specifically, on average, $0.5\%\sim 1\%$ accuracy improvements have been
achieved across all baseline models against auto-attack. When the control
information is released to an attacker, the robustness improvements are
significantly decreased compared with that in the oblivious setting. In this
experiment, we consider simple $2$-layer convolutional auto-encoders. In
future works, we will employ embedding functions that have stronger expressive
power, which allows the closed-loop control method to achieve better
performance.

## References

  * Allen-Zhu and Li (2022) Zeyuan Allen-Zhu and Yuanzhi Li.  Feature purification: How adversarial training performs robust deep learning.  In _2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS)_ , pages 977–988, 2022.
  * Althoff et al. (2011) Matthias Althoff, Soner Yaldiz, Akshay Rajhans, Xin Li, Bruce H Krogh, and Larry Pileggi.  Formal verification of phase-locked loops using reachability analysis and continuization.  In _International Conference on Computer-Aided Design_ , pages 659–666, 2011.
  * Andriushchenko et al. (2020) Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein.  Square attack: a query-efficient black-box adversarial attack via random search.  In _European Conference on Computer Vision_ , pages 484–501. Springer, 2020.
  * Antreich et al. (1994) Kurt J Antreich, Helmut E Graeb, and Claudia U Wieser.  Circuit analysis and optimization driven by worst-case distances.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 13(1):57–71, 1994.
  * Athalye et al. (2018) Anish Athalye, Nicholas Carlini, and David Wagner.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  In _International Conference on Machine Learning_ , pages 274–283. PMLR, 2018.
  * Badrinarayanan et al. (2017) Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  _IEEE transactions on pattern analysis and machine intelligence_ , 39(12):2481–2495, 2017.
  * Chen et al. (2018) Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  _Advances in neural information processing systems_ , 31, 2018.
  * Chen et al. (2021) Zhuotong Chen, Qianxiao Li, and Zheng Zhang.  Towards robust neural networks via close-loop control.  In _International Conference on Learning Representations_ , 2021.
  * Chernousko and Lyubushin (1982) FL Chernousko and AA Lyubushin.  Method of successive approximations for solution of optimal control problems.  _Optimal Control Applications and Methods_ , 3(2):101–114, 1982.
  * Chien et al. (2012) Charles Chien, Adrian Tang, Frank Hsiao, and Mau-Chung Frank Chang.  Dual-control self-healing architecture for high-performance radio SoCs.  _IEEE Design & Test of Computers_, 29(6):40–51, 2012.
  * Croce and Hein (2020a) Francesco Croce and Matthias Hein.  Minimally distorted adversarial examples with a fast adaptive boundary attack.  In _International Conference on Machine Learning_ , pages 2196–2205. PMLR, 2020a.
  * Croce and Hein (2020b) Francesco Croce and Matthias Hein.  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.  In _International conference on machine learning_ , pages 2206–2216. PMLR, 2020b.
  * Cui et al. (2020) Chunfeng Cui, Kaikai Liu, and Zheng Zhang.  Chance-constrained and yield-aware optimization of photonic ICs with non-gaussian correlated process variations.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 39(12):4958–4970, 2020.
  * Dang et al. (2004) Thao Dang, Alexandre Donzé, and Oded Maler.  Verification of analog and mixed-signal circuits using hybrid system techniques.  In _International Conference on Formal Methods in Computer-Aided Design_ , pages 21–36, 2004.
  * E (2017) E.  A proposal on machine learning via dynamical systems.  _Communications in Mathematics and Statistics_ , 5(1):1–11, 2017.
  * Fefferman et al. (2016) Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan.  Testing the manifold hypothesis.  _Journal of the American Mathematical Society_ , 29(4):983–1049, 2016.
  * Ganin et al. (2016) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ , 17(1):2096–2030, 2016.
  * Gehr et al. (2018) Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev.  AI2: Safety and robustness certification of neural networks with abstract interpretation.  In _2018 IEEE Symposium on Security and Privacy (SP)_ , pages 3–18. IEEE, 2018.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014.
  * Gowal et al. (2020) Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli.  Uncovering the limits of adversarial training against norm-bounded adversarial examples.  _arXiv preprint arXiv:2010.03593_ , 2020.
  * Goyal et al. (2011) Abhilash Goyal, Madhavan Swaminathan, Abhijit Chatterjee, Duane C Howard, and John D Cressler.  A new self-healing methodology for RF amplifier circuits based on oscillation principles.  _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , 20(10):1835–1848, 2011.
  * Guo et al. (2017) Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten.  Countering adversarial images using input transformations.  _arXiv preprint arXiv:1711.00117_ , 2017.
  * Gupta et al. (2006) Aarti Gupta, Malay K Ganai, and Chao Wang.  SAT-based verification methods and applications in hardware verification.  In _International School on Formal Methods for the Design of Computer, Communication and Software Systems_ , pages 108–143, 2006.
  * Haber and Ruthotto (2017) Eldad Haber and Lars Ruthotto.  Stable architectures for deep neural networks.  _Inverse Problems_ , 34(1):014004, 2017.
  * He and Zhang (2021) Zichang He and Zheng Zhang.  PoBO: A polynomial bounding method for chance-constrained yield-aware optimization of photonic ICs.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 2021.
  * Ho et al. (1975) Chung-Wen Ho, Albert Ruehli, and Pierce Brennan.  The modified nodal approach to network analysis.  _IEEE Transactions on circuits and systems_ , 22(6):504–509, 1975.
  * Jia and Rinard (2020) Kai Jia and Martin Rinard.  Efficient exact verification of binarized neural networks.  _Advances in neural information processing systems_ , 33:1782–1795, 2020.
  * Kang et al. (2021) Qiyu Kang, Yang Song, Qinxu Ding, and Wee Peng Tay.  Stable neural ODE with Lyapunov-stable equilibrium points for defending against adversarial attacks.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Keskin et al. (2010) Gokce Keskin, Jonathan Proesel, and Larry Pileggi.  Statistical modeling and post manufacturing configuration for scaled analog CMOS.  In _IEEE Custom Integrated Circuits Conference_ , pages 1–4, 2010\.
  * Lee et al. (2012) Jangjoon Lee, Srikar Bhagavatula, Swarup Bhunia, Kaushik Roy, and Byunghoo Jung.  Self-healing design in deep scaled CMOS technologies.  _Journal of Circuits, Systems, and Computers_ , 21(06):1240011, 2012.
  * Li et al. (2017) Qianxiao Li, Long Chen, Cheng Tai, and E Weinan.  Maximum principle based algorithms for deep learning.  _The Journal of Machine Learning Research_ , 18(1):5998–6026, 2017.
  * Li et al. (2004) Xin Li, Padmini Gopalakrishnan, Yang Xu, and T Pileggi.  Robust analog/RF circuit design with projection-based posynomial modeling.  In _IEEE/ACM International Conference on Computer Aided Design_ , pages 855–862, 2004.
  * Li et al. (2006) Xin Li, Padmini Gopalakrishnan, Yang Xu, and Lawrence T Pileggi.  Robust analog/RF circuit design with projection-based performance modeling.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 26(1):2–15, 2006.
  * Liao et al. (2018) Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu.  Defense against adversarial attacks using high-level representation guided denoiser.  In _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1778–1787, 2018.
  * Liu et al. (2011) Jenny Yi-Chun Liu, Adrian Tang, Ning-Yi Wang, Qun Jane Gu, Roc Berenguer, Hsieh-Hung Hsieh, Po-Yi Wu, Chewnpu Jou, and Mau-Chung Frank Chang.  A V-band self-healing power amplifier with adaptive feedback bias control in 65 nm cmos.  In _IEEE Radio Frequency Integrated Circuits Symposium_ , pages 1–4, 2011.
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3431–3440, 2015.
  * Madry et al. (2017) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ , 2017.
  * Mary et al. (2019) Jérémie Mary, Clément Calauzenes, and Noureddine El Karoui.  Fairness-aware learning for continuous attributes and treatments.  In _International Conference on Machine Learning_ , pages 4382–4391, 2019.
  * Moon et al. (2019) Suhong Moon, Kwanghyun Shin, and Dongsuk Jeon.  Enhancing reliability of analog neural network processors.  _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , 27(6):1455–1459, 2019.
  * Pontryagin (1987) Lev Semenovich Pontryagin.  _Mathematical theory of optimal processes_.  CRC press, 1987.
  * Qiao et al. (2019) Aurick Qiao, Bryon Aragam, Bingjing Zhang, and Eric Xing.  Fault tolerance in iterative-convergent machine learning.  In _International Conference on Machine Learning_ , pages 5220–5230, 2019.
  * Rajapakse and Groudine (2011) Indika Rajapakse and Mark Groudine.  On emerging nuclear order.  _Journal of Cell Biology_ , 192(5):711–721, 2011\.
  * Reagen et al. (2018) Brandon Reagen, Udit Gupta, Lillian Pentecost, Paul Whatmough, Sae Kyu Lee, Niamh Mulholland, David Brooks, and Gu-Yeon Wei.  Ares: A framework for quantifying the resilience of deep neural networks.  In _Design Automation Conference_ , pages 1–6, 2018.
  * Sadhu et al. (2013) Bodhisatwa Sadhu, Mark A Ferriss, Arun S Natarajan, Soner Yaldiz, Jean-Olivier Plouchart, Alexander V Rylyakov, Alberto Valdes-Garcia, Benjamin D Parker, Aydin Babakhani, Scott Reynolds, et al.  A linearized, low-phase-noise vco-based 25-ghz pll with autonomic biasing.  _IEEE Journal of Solid-State Circuits_ , 48(5):1138–1150, 2013.
  * Samangouei et al. (2018) Pouya Samangouei, Maya Kabkab, and Rama Chellappa.  Defense-gan: Protecting classifiers against adversarial attacks using generative models.  _arXiv:1805.06605_ , 2018.
  * Song et al. (2017) Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman.  Pixeldefend: Leveraging generative models to understand and defend against adversarial examples.  _arXiv preprint arXiv:1710.10766_ , 2017.
  * Sun et al. (2014) Shupeng Sun, Fa Wang, Soner Yaldiz, Xin Li, Lawrence Pileggi, Arun Natarajan, Mark Ferriss, Jean-Olivier Plouchart, Bodhisatwa Sadhu, Ben Parker, et al.  Indirect performance sensing for on-chip self-healing of analog and RF circuits.  _IEEE Transactions on Circuits and Systems I: Regular Papers_ , 61(8):2243–2252, 2014.
  * Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ , 2013.
  * Tang et al. (2012) Adrian Tang, Frank Hsiao, David Murphy, I-Ning Ku, Jenny Liu, Sandeep D’Souza, Ning-Yi Wang, Hao Wu, Yen-Hsiang Wang, Mandy Tang, et al.  A low-overhead self-healing embedded system for ensuring high yield and long-term sustainability of 60ghz 4gb/s radio-on-a-chip.  In _IEEE International Solid-State Circuits Conference_ , pages 316–318, 2012.
  * Wang et al. (2017) Mengshuo Wang, Fan Yang, Changhao Yan, Xuan Zeng, and Xiangdong Hu.  Efficient bayesian yield optimization approach for analog and sram circuits.  In _Design Automation Conference_ , pages 1–6, 2017.
  * Wang et al. (2021a) Ren Wang, Tianqi Chen, Stephen Lindsly, Cooper Stansbury, Indika Rajapakse, and Alfred Hero.  Immuno-mimetic deep neural networks (immuno-net).  _arXiv preprint arXiv:2107.02842_ , 2021a.
  * Wang et al. (2021b) Ren Wang, Tianqi Chen, Stephen Lindsly, Cooper Stansbury, Alnawaz Rehemtulla, Indika Rajapakse, and Alfred Hero.  RAILS: A robust adversarial immune-inspired learning system.  _arXiv preprint arXiv:2107.02840_ , 2021b.
  * Wang et al. (2020) Songtao Wang, Dan Li, Yang Cheng, Jinkun Geng, Yanshu Wang, Shuai Wang, Shutao Xia, and Jianping Wu.  A scalable, high-performance, and fault-tolerant network architecture for distributed machine learning.  _IEEE/ACM Transactions on Networking_ , 28(04):1752–1764, 2020.
  * Zhang et al. (2019a) Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.  Theoretically principled trade-off between robustness and accuracy.  In _International Conference on Machine Learning_ , pages 7472–7482. PMLR, 2019a.
  * Zhang and Styblinski (2013) Jian Cheng Zhang and MA Styblinski.  _Yield and variability optimization of integrated circuits_.  Springer Science & Business Media, 2013.
  * Zhang et al. (2019b) Wenbin Zhang, Xuejiao Tang, and Jianwu Wang.  On fairness-aware learning for non-discriminative decision-making.  In _2019 International Conference on Data Mining Workshops (ICDMW)_ , pages 1072–1079, 2019b.