affil0affil0affiliationtext: University of California San Diego, La Jolla, CA,
92093

#  Accelerating model evaluations in uncertainty propagation on tensor grids
using
computational graph transformations

Bingran Wang111Ph.D Candidate, Department of Mechanical and Aerospace
Engineering. Mark Sperry222Ph.D Student, Department of Mechanical and
Aerospace Engineering. Victor E. Gandarillas333Ph.D Candidate, Department of
Mechanical and Aerospace Engineering. and John T. Hwang444Assistant Professor,
Department of Mechanical and Aerospace Engineering.

###### Abstract

Methods such as non-intrusive polynomial chaos (NIPC), and stochastic
collocation are frequently used for uncertainty propagation problems.
Particularly for low-dimensional problems, these methods often use a tensor-
product grid for sampling the space of uncertain inputs. A limitation of this
approach is that it encounters a significant challenge: the number of sample
points grows exponentially with the increase of uncertain inputs. Current
strategies to mitigate computational costs abandon the tensor structure of
sampling points, with the aim of reducing their overall count. Contrastingly,
our investigation reveals that preserving the tensor structure of sample
points can offer distinct advantages in specific scenarios. Notably, by
manipulating the computational graph of the targeted model, it is feasible to
avoid redundant evaluations at the operation level to significantly reduce the
model evaluation cost on tensor-grid inputs. This paper presents a pioneering
method: Accelerated Model Evaluations on Tensor grids using Computational
graph transformations (AMTC). The core premise of AMTC lies in the strategic
modification of the computational graph of the target model to algorithmically
remove the repeated evaluations on the operation level. We implemented the
AMTC method within the compiler of a new modeling language called the
Computational System Design Language (CSDL). We demonstrate the effectiveness
of AMTC by using it with the full-grid NIPC method to solve three low-
dimensional UQ problems involving an analytical piston model, a
multidisciplinary unmanned aerial vehicle design model, and a multi-point air
taxi mission analysis model, respectively. For all of the test problems, AMTC
reduces the model evaluation cost by between 50% and 90%, making the full-grid
NIPC the most efficacious method to use among the UQ methods implemented.

## 1 Introduction

In practical scenarios, the numerical models inherently fall short of
achieving perfect fidelity with reality and consistently contend with inherent
uncertainties. To address this challenge, uncertainty propagation, also known
as forward uncertainty quantification (UQ), aims to undertake a rigorous
examination and analysis of how a system’s output is influenced by the
uncertainties inherent in its inputs. The resultant assessments of uncertainty
serve as a valuable foundation for facilitating informed decision-making and
thorough risk assessment. The applications of UQ can be found in many
scientific and engineering domains such as weather forecast [1, 2], structural
analysis [3, 4] and aircraft design [5, 6, 7].

Uncertainties within the inputs can stem from diverse origins and are
typically classified into two categories: aleatoric and epistemic. Aleatoric
uncertainties, recognized as irreducible uncertainties, are intrinsic to the
system and encompass factors such as fluctuations in operational conditions,
mission prerequisites, and model parameters. These uncertainties can often be
described within a probabilistic framework. In contrast, epistemic
uncertainties constitute reducible uncertainties arising from knowledge gaps.
These uncertainties may be induced by approximations utilized in computational
models or numerical solution methods, and are often difficult to be described
within a probabilistic framework. This paper focuses on UQ problems within a
probabilistic formalism where uncertain inputs are continuous random variables
with known probability density distributions. The goal is to estimate
statistical moments such as the mean and variance of the Quantity of Interest
(QoI), or more complex risk measures such as the probability of failure or the
conditional value at risk.

For this type of UQ problem, the most common non-intrusive methods include the
method of moments, polynomial chaos, kriging, and Monte Carlo. The method of
moments analytically computes the statistical moments of the QoI, using the
Taylor series approximation. It often only uses function evaluations and
derivative information at one or a small number of input points [8]. If the
first-order derivative is computed using the adjoint method, the cost of UQ
using the method of moments can be as low as two function evaluations [9].
While adept at efficiently estimating statistical moments, this approach
grapples with challenges in estimating other kinds of risk measures. Moreover,
it may lack accuracy when input variance is significant. In contrast, the
Monte Carlo approach employs random sampling of uncertain inputs to compute
the risk measures. Based on the law of large numbers, Monte Carlo’s
convergence rate is independent of the number of uncertain inputs, making it
particularly attractive for solving high-dimensional problems. Recent strides
in enhancing its efficiency encompass multi-fidelity [10, 11] and importance
sampling [12] techniques. However, in the context of low-dimensional problems,
the Monte Carlo method might demand substantially more model evaluations to
achieve the same level of accuracy as the alternative UQ methodologies.
Kriging, also known as Gaussian process regression, is a common statistical
technique employed in various fields. In UQ, kriging constructs a surrogate
response surface by leveraging input-output data pairs. The surrogate model
replaces the original expensive function to allow a large number of model
evaluations in order to perform reliability analysis [13, 14] or optimization
under uncertainty [15]. For low-dimensional UQ problems, if the objective
function is assumed to be smooth, the polynomial chaos-based techniques often
stand out as the most efficacious option. The generalized polynomial chaos
theory represents the QoI as orthogonal polynomials that are derived from the
distributions of the uncertain inputs. Benefiting from the inherent smoothness
of the random space, polynomial chaos-based methods exhibit rapid convergence
rates facilitated by sampling or integration techniques. Common polynomial
chaos-based methods include non-intrusive polynomial chaos (NIPC) [16, 17, 18]
and stochastic collocation (SC) [19, 20].

For both NIPC and SC methods, we can use a tensor-product grid to sample the
input space, but this approach is often not preferred and is only used for
very low-dimensional UQ problems, as the number of input points increases
exponentially with the number of uncertain inputs. The state-of-art methods
choose to use a sparse grid to sample the input space or have the sample
points randomly generated, in order to reduce the input points with a minimum
loss of accuracy. However, our recent findings show that maintaining a tensor
structure for the input points offers some unique advantages and for certain
problems, the total model evaluation cost can be reduced in a different
approach without changing the tensor structure of the input points.

When we evaluate a computational model on full-grid input points, if we view
the computational model as a computational graph with only elementary
operations, the current framework evaluates each operation the same number of
times at the tensor-product input points of all the uncertain inputs. However,
for each operation, the output data only has distinct values at the distinct
input points in their dependent input space, and a large portion of the
operations may not be dependent on all of the uncertain inputs. This means the
current framework of NIPC and SC could create many wasteful evaluations on a
large portion of the operations.

In this paper, we propose a computational graph transformation method called
Accelerated Model evaluations on Tensor grids using Computational graph
transformations (AMTC) that algorithmically modifies the computational graph
to eliminate the unnecessary evaluations incurred by the current framework of
integration-based NIPC and SC. AMTC reduces the model evaluation cost on full-
grid quadrature points by partitioning the computational graph into sub-graphs
using the operations’ dependency information and evaluating each sub-graph on
the distinct quadrature points. The proposed method is implemented in
conjunction with a new algebraic modeling language, the Computational System
Design Language (CSDL). We achieve cost reductions on UQ problems in a general
and automatic way by analyzing and manipulating the computational graph that
CSDL makes available.

This method has been applied to three different UQ problems with different
computational graph structures. For all of the problems, we observed a
significant acceleration in model evaluation time when solving the UQ problems
using the full-grid NIPC method, which makes this UQ method the most efficient
to use on these problems. Generally, for a wide range of UQ problems that
involve multi-point, multi-disciplinary, or other models that possess a sparse
graph structure, we expect our method to significantly reduce the model
evaluation time on tensor-grid input points, which can improve the time
scaling for, but not limited to NIPC and SC methods.

This paper is organized as follows. Section 2 gives some background on PCE-
related UQ methods, the current model evaluation framework, and how to view
tensor-grid evaluations through computational graphs. Section 3 presents the
details of the new computational graph transformation algorithm, AMTC, and how
it is implemented in CSDL. Section 4 shows the numerical results of three UQ
problems. Section 5 summarizes the work and offers concluding thoughts.

## 2 Background

### 2.1 Polynomial chaos expansion

Wiener initially introduced the concept of polynomial chaos expansion (PCE),
utilizing Hermite polynomials to model the response of a system affected by
Gaussian uncertain inputs [21]. This foundational concept was later advanced
into the generalized polynomial chaos (gPC) method by Xiu and Karniadakis
[22]. Within the framework of gPC, model responses are represented through
polynomial series of uncertain inputs. These polynomials are meticulously
chosen to exhibit orthogonality with respect to the probability distribution
of the uncertain inputs, thereby ensuring exponential convergence.

We address a general problem in uncertainty quantification (UQ) involving a
function defined as follows:

$f=\mathcal{F}(u),$ (1)

where $\mathcal{F}:\mathbb{R}^{d}\to\mathbb{R}$ represents the model
evaluation function, $u\in\mathbb{R}^{d}$ denotes the input vector, and
$f\in\mathbb{R}$ represents a scalar output. The uncertainties associated with
the inputs are expressed as a stochastic vector denoted by
$U:=[U_{1},\ldots,U_{d}]$, under the assumption that these random variables
are mutually independent. The stochastic input vector adheres to the
probability distribution $\rho(u)$ with its support defined by $\Gamma$. Our
focus centers on the evaluation of risk measures of the output random
variable, $f(U)$.

In gPC, the stochastic output, $f(U)$, is defined as an infinite series of
orthogonal polynomials with respect to the uncertain input variables:

$f(U)=\sum_{i=0}^{\infty}\alpha_{i}\Phi_{i}(U),$ (2)

where $\Phi_{i}(U)$ are the PCE basis functions that are generated based on
$\rho(U)$, and $\alpha_{i}$ are the corresponding weights that need to be
determined.

Choosing the PCE basis functions is important to ensure the rapid convergence
of PCE-based methods. These PCE basis functions have to satisfy the
orthogonality property,

$\left<\Phi_{i}(U),\Phi_{j}(U)\right>=\delta_{ij},$ (3)

where $\delta_{ij}$ is Kronecker delta and the inner product is defined as

$\left<\Phi_{i}(U),\Phi_{j}(U)\right>=\int_{\Gamma}\Phi_{i}(u)\Phi_{j}(u)\rho(u)du.$
(4)

In one-dimensional problems, we can directly use the univariate orthogonal
polynomials as PCE basis functions. In Table 1, we show the univariate
orthogonal polynomials for common types of continuous random variables. In
multi-dimensional problems, if the input variables are mutually independent,
the PCE basis functions can be formed as a tensor-product of the univariate
orthogonal polynomials corresponding to each uncertain input. To elucidate
this, we introduce the concept of multi-index notation and define a $p$-tuple
as

$i^{\prime}=(i_{1},\ldots,i_{p})\in\mathbb{N}^{p}_{0},$ (5)

with its magnitude determined as

$|i^{\prime}|=i_{1}+\ldots+i_{p}.$ (6)

Employing this notation, we denote the collection of univariate PCE basis
functions for variable $u_{k}$ as $\\{\phi_{i}(u_{k})\\}_{i=0}^{p}$. Then, the
multivariate PCE basis with an upper limit on the total degree of $p$ can be
expressed as

$\Phi_{\bm{i^{\prime}}}(u)=\phi_{i_{1}}(u_{1})\ldots\phi_{i_{d}}(u_{d}),\quad|i^{\prime}|\leq
p.$ (7)

In practice, one may use the PCE basis functions in (7) to truncate the
infinite series in (2), resulting in:

$f(U)\approx\sum_{i=0}^{q}\alpha_{i}\Phi_{i}(U).$ (8)

The resultant number of PCE basis functions, $q+1$, satisfies

$q+1=\frac{(d+p)!}{d!p!}.$ (9)

Once the PCE coefficients are estimated, it becomes straightforward to compute
statistical moments such as the mean and standard deviation of the model
output. These calculations can be expressed as follows:

$\mu_{f}=\alpha_{0},$ (10) $\sigma_{f}=\sum_{i=0}^{d}\alpha_{i}^{2}.$ (11)

For risk measures like the probability of failure, the truncated PCE model can
be treated as a surrogate model, and methods like Monte Carlo can be applied
on the surrogate model to compute the desired risk measure.

Table 1: Orthogonal polynomials for common types of continuous random variables Distribution | Orthogonal polynomials | Support range
---|---|---
Normal | Hermite | $(-\infty,\infty)$
Uniform | Legendre | $[-1,1]$
Exponential | Laguerre | $[0,\infty)$
Beta | Jacobi | $(-1,1)$
Gamma | Generalized Laguerre | $[0,\infty)$

#### 2.1.1 Non-intrusive polynomial chaos method

The non-intrusive polynomial chaos (NIPC) method solves the PCE coefficients
$\alpha_{i}$ in (8) by either integration or regression. The integration
approach uses the orthogonal property in (3) and projects the model output
onto each basis function:

$\alpha_{i}=\frac{\left<f(U),\Phi_{j}\right>}{\left<\Phi_{i}^{2}\right>}=\frac{1}{\left<\Phi_{i}^{2}\right>}\int_{u}f(u)\Phi_{i}(u)\rho(u)du.$
(12)

This requires solving a multi-dimensional integration problem, which is often
estimated by using the Gauss quadrature approach:

$\displaystyle\alpha_{i}$
$\displaystyle=\frac{1}{\left<\Phi_{i}^{2}\right>}\int_{u_{1}}\ldots\int_{u_{d}}f(u_{1},\ldots,u_{d})\Phi_{k}(u_{1},\ldots,u_{d})\rho(u_{1},\ldots,u_{d})du_{1}\ldots
du_{d}$ (13)
$\displaystyle\approx\frac{1}{\left<\Phi_{i}^{2}\right>}\sum_{i_{1}=0}^{k}\ldots\sum_{i_{d}=0}^{k}w_{1}^{(i_{1})}\ldots
w_{d}^{(i_{d})}f(u_{1}^{(i_{1})},\ldots,u_{d}^{(i_{d})})\Phi_{i}(u_{1}^{(i_{1})},\ldots,u_{d}^{(i_{d})}),$

where $(u_{i}^{(1)},\ldots,u_{i}^{(k)})$ and
$(w_{i}^{(1)},\ldots,w_{i}^{(k)})$ are the nodes and weights for the 1D Gauss
quadrature rule in the $u_{i}$ dimension, $k$ is the number of quadrature
points used in each dimension. The 1D quadrature rule with $k$ nodes is chosen
such that the quadrature rule can exactly integrate 1D polynomials up to
$(2k-1)$th order. In the multi-dimensional case, the Gauss quadrature rule
forms a tensor product of the 1D quadrature rule, which requires us to
evaluate the model output at the tensor-product quadrature points, defined as

$\bm{u}=\bm{u}_{1}^{k}\times\cdots\times\bm{u}_{d}^{k},$ (14)

where

$\bm{u}_{i}^{k}:=\\{u_{i}^{(j)}\\}_{j=1}^{k}.$ (15)

The total number of quadrature points is thus $k^{d}$, and this number
increases exponentially with respect to the number of uncertain inputs. For
high-dimensional problems, one way to mitigate the exponential growth of
quadrature points is to use the Smolyak sparse grid approach [23], which drops
the higher-order cross terms in the quadrature points with minimal loss of
accuracy. The sparse grid quadrature points can be expressed as:

$\bm{u}=\bigcup_{\ell-d+1\leq|i|\leq\ell}\left(\bm{u}^{i_{1}}_{1}\times\cdots\times\bm{u}^{i_{d}}_{d}\right),$
(16)

where $\ell$ is the level of construction. Another way to reduce the total
number of quadrature points is to use a designed quadrature approach. In [24],
Keshavarzzadeh et al. formulate an optimization problem to optimize for the
designed quadrature points and the corresponding weights that ensure the exact
integration on a polynomial subspace. The number of designed quadrature points
can be significantly smaller than the full-grid quadrature points for solving
the high-dimensional integration problems for the same level of accuracy.

The regression approach first generates $n$ sample points for $U$ and
evaluates the model for each sample point. The coefficients are determined by
solving a linear least-squares problem:

$\begin{bmatrix}\Phi_{1}(u^{(1)})&\ldots&\Phi_{q}(u^{(1)})\\\
\vdots&&\vdots\\\ \Phi_{1}(u^{(n)})&\ldots&\Phi_{q}(u^{(n)})\\\
\end{bmatrix}\begin{bmatrix}\alpha_{1}\\\ \vdots\\\
\alpha_{q}\end{bmatrix}=\begin{bmatrix}f(u^{(1)})\\\ \vdots\\\
f(u^{(n)})\end{bmatrix}.$ (17)

The rule of thumb is that the number of samples $n$ needs to be 2-3 times the
number of coefficients $q$. This means the cost of the regression-based NIPC
method can be reduced by using the sparse PCE basis, resulting in fewer
coefficients to be estimated. Recent advances in adaptive basis selection and
adaptive sampling can be found in [25, 26, 27].

#### 2.1.2 Stochastic collocation

Stochastic collocation (SC) is another PCE-related UQ method. It forms
Lagrange interpolation functions from a set of collocation points as follows:

$L_{j}(u)=\prod_{i=1,i\neq j}^{n}\frac{u-u^{(i)}}{u^{(j)}-u^{(i)}},$ (18)

where $L_{j}=1$ at $u=u_{j}$ and $L_{j}=0$ for all other collocation points.
The model output is approximated as the sum of the Lagrange interpolation
functions with the weights equal to the model output evaluations at each
collocation point:

$f(u)\approx\sum_{j}^{n}f(u^{(j)})L_{j}(u).$ (19)

SC typically chooses the set of collocation points in the same way as
quadrature points are chosen in integration-based NIPC by using the
full/sparse-grid strategy. The difference is that NIPC computes the PCE
coefficients by performing numerical integration, while SC forms Lagrangian
interpolation functions to approximate the model output. In [28], Eldred and
Burkardt compared the standard NIPC and SC methods and found similar
performance in most of the cases, although SC had a small edge over NIPC for
problems with a dimension greater than two.

### 2.2 Model evaluation cost for current non-intrusive UQ methods

If we consider only the non-adaptive approaches, the current framework for
non-intrusive UQ methods includes three steps:

  1. 1.

Generate sample/quadrature points based on the distribution of uncertain
inputs.

  2. 2.

Evaluate the target model for each of the sample points.

  3. 3.

Post-process the output points to compute the desired quantities of the
output.

Step one generates the input points based on the distribution of uncertain
inputs. For methods like regression-based NIPC, Monte Carlo and kriging, the
input points are often generated randomly (e.g. random sampling, Latin
hypercube sampling). For integration-based NIPC and SC methods, they can be
generated using the full/sparse-grid strategy. In step two, the target model
is repeatedly evaluated at each of the input points. For some models that
support vectorized input, the input points can be vectorized and evaluated at
the same time. However, most of the time, the target model needs to be run in
a for-loop by evaluating one input point at a time. In step three, the model
output points are used to estimate the PCE coefficients for NIPC, form
Lagrange interpolation functions for SC, or construct Gaussian process model
for kriging, before computing the desired quantity for the output.

Under this framework, the model evaluation cost for any UQ method can be
approximated as:

$\text{cost}\approx nO(\mathcal{F}(u)),$ (20)

where $n$ is the number of sample points and $O(\mathcal{F}(u))$ is the model
evaluation cost for one sample point. When we use the full-grid approach for
integration-based NIPC or SC to generate the input points, the number of input
points increases exponentially with the dimension of uncertainty inputs as
$n=k^{d}$. Under the current framework to evaluate the model, the evaluation
cost follows Eq. 20 and thus increases exponentially as the number of
uncertain inputs, thus suffers the curse of dimensionality.

The current methods (e.g. sparse-grid and designed quadrature) to overcome the
curse of dimensionality all focus on reducing the number of input points, $n$,
by breaking the full-tensor structure of the input points, so that the input
points do not increase exponentially as the number of the uncertain inputs,
but none tries to break the linear relationship in Eq. 20 to reduce the
evaluation cost on tensor-grid input points by using a computational graph
transformation approach.

### 2.3 Viewing tensor-grid evaluations via computational graph

Any computer program that describes a computational model is a computational
process that executes a sequence of elementary functions and operations. A
computational process can be described by a computational graph. A
computational graph is a directed acyclic graph in which the nodes represent
the operations and variables, and the directed edges represent how operations
and variables are connected to each other. Such a graph is also bipartite
because each operation is connected to an output variable, and likewise,
variables are connected to operations (as arguments). For example, the
computer program that evaluates a function $f=cos(u_{1})+exp(-u_{2})$ can be
decomposed into the following set of fundamental operations:

$\displaystyle\xi_{1}=\varphi_{1}(u_{1})=cos(u_{1});$ (21)
$\displaystyle\xi_{2}=\varphi_{2}(u_{2})=-u_{2};$
$\displaystyle\xi_{3}=\varphi_{3}(\xi_{2})=exp(\xi_{2});$ $\displaystyle
f=\varphi_{4}(\xi_{2},\xi_{3})=\xi_{2}+\xi_{3},$

The computational graph for this model is shown in Fig. 1 with the blue nodes
representing the variables and the orange nodes representing the operations.

Figure 1: Computational graph of function $f=cos(u_{1})+exp(-u_{2})$

A common use of computational graphs is in automatic differentiation (AD) [29,
30] to automatically compute the total derivatives. AD views the computer
program as a sequence of operations and functions and repeatedly applies the
chain rule to compute the total derivative. Another use of the computational
graph is to modify the computational graph to reduce the time of memory cost
of the model. In Tensorflow [31], a graph optimization method is implemented
to reduce memory usage and evaluation costs for neural network models. It
modifies the computational graph through constant folding, arithmetic
simplification, and optimization.

If we view the model as a computational graph, in the current UQ framework,
each operation and function is evaluated the same number of times as the
number of input points. However, this could create many unnecessary
evaluations when we evaluate the model on tensor-grid input points. For
example, consider a UQ problem involving the simple function we showed before,
given by

$f=\text{cos}(u_{1})+\text{exp}(-u_{2}).$ (22)

In this problem, $U_{1}$ and $U_{2}$ are independent uncertain inputs, and we
aim to compute the random variable $f(U_{1},U_{2})$. The computational process
that describes this function is in (21). If we choose the full-grid quadrature
points with $k$ quadrature points in each dimension, we need to evaluate this
function at a total of $k^{2}$ quadrature points which are

$\mathbf{u}=\begin{Bmatrix}(u_{1}^{(1)},u_{2}^{(1)})&\ldots&(u_{1}^{(k)},u_{2}^{(1)})\\\
\vdots&\ddots&\vdots\\\
(u_{1}^{(1)},u_{2}^{(k)})&\ldots&(u_{1}^{(k)},u_{2}^{(k)})\\\ \end{Bmatrix}.$
(23)

For both inputs $u_{1}$ and $u_{2}$, if we view their input points as vectors,
they are in a tensor-product form between the quadrature points in its
dimension and a vector of ones:

$\displaystyle
u_{1}=\text{vec}\left(\begin{bmatrix}u_{1}^{(1)}&\dots&u_{1}^{(k)}\\\
\vdots&\ddots&\vdots\\\ u_{1}^{(1)}&\ldots&u_{1}^{(k)}\\\
\end{bmatrix}\right)=\text{vec}\left([u_{1}^{(1)},\ldots
u_{1}^{(k)}]\otimes[1,\ldots,1]\right),$ (24) $\displaystyle
u_{2}=\text{vec}\left(\begin{bmatrix}u_{2}^{(1)}&\ldots&u_{2}^{(1)}\\\
\vdots&\ddots&\vdots\\\ u_{2}^{(k)}&\ldots&u_{2}^{(k)}\\\
\end{bmatrix}\right)=\text{vec}\left([1,\ldots,1]\otimes[u_{2}^{(1)},\ldots
u_{2}^{(k)}]\right).$

Even though both vectors have a size of $k^{2}$, there are only $k$ distinct
values, which are the values of the $k$ quadrature points in its dimension. In
the current framework, we can view the model evaluation step as propagating
the input vectors in (24) through the model’s computational graph, which is
shown in Fig. 1. When we view the propagation of input vectors from the
operation level, the current framework creates many repeated evaluations. For
example, when we evaluate the first operation node in the computational graph,
$\xi_{1}=\text{cos}(u_{1})$, we will evaluate this operation for $k^{2}$
points, since there are $k^{2}$ input points for $u_{1}$. However, out of the
$k^{2}$ evaluations, there are at most $k$ distinct evaluations and thus the
current framework creates $k^{2}-k$ wasteful repeated evaluations in this
case. Similarly, for $\xi_{2}=-1*u_{2}$ and $\xi_{3}=\text{exp}(\xi_{2})$,
both of the operations’ outputs are only dependent on $u_{2}$, and they also
have at most $k$ distinct values to be evaluated for. The current framework
also creates $k^{2}-k$ repeated evaluations for these operations. In contrast,
for the last operation node, $f=\xi_{1}+\xi_{3}$, there are $k^{2}$ distinct
values to be evaluated as $f$ is dependent on both $u_{1}$ and $u_{2}$.

In practical UQ problems, we could have a large computational graph with a
number of uncertain inputs, but some of the inputs may only affect a small
part of the computational graph. For example, for a multidisciplinary model,
we may have uncertain inputs coming from different disciplines. When we view
the computational graph for the whole model, there may be a small portion of
operations that are dependent on some uncertain inputs. In this case, we could
reduce the total computational cost on the tensor-grid inputs by eliminating
the repeated evaluation cost for each operation node but the output data still
carry the necessary results to compute the results of the QoI.

The logic here is simple and straightforward. One can manually modify the size
of the input and state variables in the model to achieve this reduction if the
model has a simple graph structure. However, for a complicated model with a
large number of uncertainties, it becomes impractical to manually change the
code to perform only necessary evaluations and match the shape of input for
each node in the computational process.

## 3 Methodology

Here, we present the Accelerated Model evaluations on Tensor grids using
Computational graph transformations (AMTC) method. The goal of our method is
to enable automatic generation of a modified computational graph, given a
computer program, that eliminates all wasteful evaluations due to tensor-
product sampling within a NIPC or SC algorithm. The modified computational
graph leverages the graph structure of the target model and enables performing
the minimal number of evaluations for each operation. To achieve this, we
modify the size of input data that are passed into the operations.

In the UQ context, when we evaluate the computational model on full-grid
quadrature points, we are forming a tensor product grid from the quadrature
points in each uncertain input dimension. If we use $k$ quadrature points in
each dimension, we end up having $k^{d}$ number of input points for $d$
uncertain inputs. For each dimension, these $k$ quadrature points are
determined based on the probability distribution of that uncertain input. When
we view the target model monolithically, the output is dependent on all of the
$d$ uncertain inputs. In the $d$-dimensional uncertain input space, we have a
total of $k^{d}$ quadrature points, and it is required for us to evaluate the
model output on all $k^{d}$ input points to gather all of the necessary
information to determine the effect of the uncertain inputs on the model
output for UQ purpose. However, when we view it as a computational graph and
consider the output of each operation in the graph, not all of the operations’
outputs are dependent on $d$ uncertain inputs. For an output dependent on
$d^{\prime}$ uncertain inputs with $d^{\prime}<d$, in its uncertain input
space, there are only $k^{d^{\prime}}$ number of quadrature points, and
accordingly this output only needs to be computed $k^{d^{\prime}}$ times to
compute the necessary information for UQ purpose. Drawing from this rationale,
we want to partition the model’s computational graph into discrete sub-graphs.
These sub-graphs are chosen such that the operations within each are dependent
on the same uncertain inputs and thus need to be evaluated on the same
quadrature points. We achieve this by generating the dependency information
for each operation, which shows whether the operation is dependent on an
uncertain input. The dependency information is stored as an influence matrix,
written as $D(\varphi_{i},u_{j})$, whose $(i,j)$th entry is 1 if the $i$th
operation depends on the $j$th uncertain input and 0 otherwise. For example,
the dependency information for the simple function in (16) is shown in Tab.2.

Table 2: Operations dependency information for function $f=cos(u_{1})+exp(-u_{2})$ $D(\varphi_{i},u_{j})$ | $u_{1}$ | $u_{2}$
---|---|---
$\varphi_{1}$ | 1 | 0
$\varphi_{2}$ | 0 | 1
$\varphi_{3}$ | 0 | 1
$\varphi_{4}$ | 1 | 1

From the dependency information, the operations that have the same dependency
information on all of the uncertain inputs can be grouped together to form a
sub-computational graph to be evaluated on the same quadrature points. Each
sub-computational graph should be evaluated the same number of times as the
number of quadrature points in its uncertain input space. However, the output
of one sub-graph may be the input of another sub-graph, since they have
different uncertain input spaces, it is required to extend the uncertain input
space of that output to match the uncertain input space of the second sub-
graph’s output. This requires us to perform a deliberate tensor-algebra
operation. In the Python language, this can be easily achieved using Numpy’s
Einstein summation (Einsum) operation. The Einsum operation can be used for
compactly performing summations involving indices that appear in multiple
tensors/matrices. In this case, we can treat the output of the sub-graph as a
tensor and use the Einsum operation to perform a summation on specific indices
between the output tensor and a tensor of ones, in order to extend its
uncertain input space to a desired higher-order input space. For example,
consider an addition operation in the computational graph

$f=\xi_{1}+\xi_{2}$

with $\xi_{1}(u_{1})\in\mathbb{R}^{k_{u_{1}}}$ and
$\xi_{2}(u_{2})\in\mathbb{R}^{k_{u_{2}}}$. In this case, $\xi_{1}$ is only
dependent on $u_{1}$ with a size of $k_{u_{1}}$, while $\xi_{2}$ is only
dependent on $u_{2}$ with a size of $k_{u_{2}}$. The values in $\xi_{1}$ and
$\xi_{2}$ correspond to their evaluations on the quadrature points in $u_{1}$
and $u_{2}$ dimensions, respectively. The output of this operation, $f$ is
dependent on both $u_{1}$ and $u_{2}$ and its vector should have a size of
$\mathbb{R}^{k_{u_{1}}k_{u_{2}}}$. What we need to do here is to insert
operations to modify the sizes of the inputs so that they have the same size
as the output $f$ is supposed to be. Thus we evaluate the addition operation,
it results in the correct vector for it. We show a demonstration of how to
extend the input size in a Python implementation using NumPy’s Einstein
summation function followed by a reshape operation. The specific code in
Python would be import numpy as np original_shape_xi_1 = (k_u1,1) #Original
size for xi_1 original_shape_xi_2 = (k_u2,1) #Original size for xi_2
modified_shape = (k_u1*k_u2, 1) #Target size for xi_1 and xi_2 xi_1 =
np.einsum(’i…,p…-¿pi…’, xi_1, np.ones((k_u2, 1)) #(k_u1,1) -¿ (k_u1,k_u2,1)
xi_2 = np.einsum(’i…,p…-¿ip…’, xi_2, np.ones((k_u1, 1)) #(k_u2,1) -¿
(k_u1,k_u2,1) xi_1 = np.reshape(xi_1, modified_shape) #(k_u1,k_u2,1)
-¿(k_u1*k_u2, 1) xi_2 = np.reshape(xi_2, modified_shape) #(k_u1,k_u2,1)
-¿(k_u1*k_u2, 1). The code above transforms both inputs into the same size of
the output with the correct order of values in the vectors. In fact, for any
dimension and size of the input, we can always use the Einstein summation
functions followed by reshape functions to transform the input data to the
correct size we need. In our method, we insert the $einsum$ operation nodes
for any connections between the sub-graphs we partitioned. Each $einsum$
operation node has an Einstein summation function followed by a reshape
function with the right arguments to ensure the correct data flow between the
sub-graphs. The outline for the AMTC method is shown in Alg.1.

Algorithm 1 Accelerated Model evaluations on Tensor grids using Computational
graph transformations (AMTC) algorithm

1:Specify a computational graph for a computational model

2:Run Kahn’s algorithm[32] to determine the correct order for the operations
to be evaluated

3:Generate the operations’ dependency information and store it as an influence
matrix

4:Group the operations that share the same dependency information and form the
sub-graphs

5:Insert the einsum operation nodes with the right argument between the sub-
graphs

6:Evaluate the modified computational graph with the correct input points

To demonstrate how our algorithm modifies the computational graph and reduces
the evaluation costs, we show a comparison of computational graphs with and
without using AMTC in Fig. 2. Assuming we evaluate the function in (22) at
full-grid quadrature points of the uncertain inputs, this figure shows the
computational graphs with and without using the AMTC method. The size of the
data flow is also labeled in the graphs and the partitioned sub-graphs are
indicated on the modified computational graph. In this case, without using the
AMTC method, both inputs $u_{1}$ and $u_{2}$ have the size of $k^{2}\times 1$,
and thus each operation in the computational graph is evaluated for $k^{2}$
number of times. However, with the AMTC method, the computational graph is
partitioned into three sub-graphs. The operations in sub-graph 1 are evaluated
on the $k$ quadrature points in $u_{1}$ dimension while operations in sub-
graph 2 are evaluated on the $k$ quadrature points in $u_{2}$ dimension. Two
einsum operations are inserted connecting sub-graph 1 and sub-graph 2 to sub-
graph 3, so that the outputs of the first two sub-graphs are modified into the
correct size and sub-graph 3 can be evaluated $k^{2}$ times to gather the same
output values as the current method.

(a)

(b)

Figure 2: Computational graphs with data size for full-grid quadrature points
evaluation on $f=cos(u_{1})+exp(-u_{2})$

### 3.1 Implementation in CSDL

The AMTC method can only be implemented from the data access layer of a
software package where we have access to the computational graph and data
structure of a computer program. We choose to implement this method within the
Computational System Design Language (CSDL) package555CSDL software
repository: https://lsdolab.github.io/csdl/ [33]. CSDL is an embedded domain-
specific language targeting large-scale multidisciplinary design analysis and
optimization problems. With CSDL, the user defines the model as a sequence of
operations by using a software interface that is highly expressive and natural
(CSDL code resembles regular Python).

We show a demonstration graph for the implementation of AMTC within CSDL in
Fig. 3. The CSDL package has a unique three-stage compiler, which includes
front-end, middle-end, and back-end. Using the front-end, the user defines the
model and specifies the problem that needs to be solved. Next, the middle-end
constructs a computational graph for that model, and the AMTC acts as a graph
transformation method to generate the modified computational graph that is
more efficient to evaluate. Finally, at the back-end, it creates an executable
script based on the modified computational graph using an automatic code
generation approach.

Below is a coarse outline of how CSDL and our proposed method interact:

  1. 1.

User defines the model and identifies the uncertain inputs.

  2. 2.

Generate the quadrature points according to the distributions of the input
variables.

  3. 3.

Generate the computational graph from the numerical model the user defines.

  4. 4.

Modify the computational graph as in Alg. 1.

  5. 5.

Generate an executable back-end.

  6. 6.

Post-process the output data to calculate the desired quantities of the
output.

Figure 3: Demonstration for the implementation of AMTC on CSDL

## 4 Numerical Results

We investigate the performance of AMTC by using it with the full-grid NIPC
method on three low-dimensional UQ problems involving different computational
models. The AMTC method has been implemented in the middle-end of CSDL
compiler as a graph transformation method, and all of the computational models
involved in the test problems are built in CSDL in order to use AMTC to
accelerate its tensor-grid evaluations.

### 4.1 Analytical piston model

We first consider a UQ problem involving an analytical non-linear model of the
cycle time of a piston. This problem is adapted from [34]. The piston cycle
time $C$ in seconds is expressed as

$C=2\pi\sqrt{\frac{M}{k+S^{2}\frac{P_{0}V_{0}T_{a}}{T_{0}V^{2}}}},$ (25)

with

$V=\frac{S}{2k}\left(\sqrt{A^{2}+4k\frac{P_{0}V_{0}}{T_{0}}T_{a}}-A\right)\text{
and }A=P_{0}S+19.62M-\frac{kV_{0}}{S}.$ (26)

In this problem, three of the input parameters are normal random variables,
and the rest of them are deterministic variables. The values and descriptions
for all of the variables are shown in Table 3.

Parameter | Range | Description
---|---|---
$M$ | $N(50,10)$ | Piston weight (kg)
$S$ | $N(0.01,0.005)$ | Piston surface area ($m^{2}$)
$V_{0}$ | $N(0.005,0.002)$ | Initial gas volume ($m^{3}$)
$k$ | 3000 | Spring coefficient ($N/m$)
$P_{0}$ | 100000 | Atmospheric pressure ($N/m^{2}$)
$T_{a}$ | 293 | Ambient temperature (K)
$T_{0}$ | 350 | Filling gas temperature (K)
Table 3: Input parameters and ranges for the piston problem

The objective of this problem is to compute the expectation value of the
piston cycle time, namely, $\mathbb{E}[C]$.

This UQ problem and those that follow are all solved using five methods:
integration-based NIPC method using the full-grid quadrature points (full-grid
NIPC), full-grid NIPC with the AMTC method (full-grid NIPC with AMTC);
integration-based NIPC method using the designed quadrature method (designed
quadrature NIPC); kriging and the Monte Carlo method.

In our numerical experiments, the full-grid NIPC and the full-grid NIPC with
AMTC methods always generate the same results. Fig. LABEL:fig:_piston_comp
shows the model evaluation time in terms of the number of equivalent model
evaluations for the full-grid NIPC method with and without AMTC. The results
show that the AMTC method brought a consistent 50%-60% reduction in evaluation
time. In Fig. LABEL:fig:_piston_results, the convergence plots of these five
methods are compared for the UQ result. The percentage errors are calculated
with respect to the results of the full-grid NIPC using 400 quadrature points.
For this UQ problem, the NIPC methods completely outperform the Monte Carlo
and kriging methods. This is because this UQ problem is fairly low-dimensional
(only three uncertain inputs), and the function is generally smooth. In this
kind of UQ problem, PCE-related methods are often the most efficient choices.
Among the NIPC methods, the designed quadrature NIPC performs better than the
full-grid NIPC, as the designed quadrature NIPC requires a smaller number of
quadrature points to achieve the same level of accuracy on integration.
However, we see a dramatic speed-up after applying the AMTC method to the
full-grid NIPC; the total model evaluation time is reduced by almost an order
of magnitude, making the full-grid NIPC with AMTC the most efficient method
among other methods.

The major speed-up that AMTC brought for full-grid NIPC can be explained by
the sparsity of the influence matrix that describes the dependency information
of the computational model in this UQ problem. We show the number of dependent
operations for each uncertain input in this computational model in Tab. 4.
From this table, we observe that out of the 65 operations in the computational
graph, 50% of the operations depend on the uncertain input $M$ and 60% of the
operations depend on uncertain inputs $S$ and $V_{0}$. This means, a large
portion of the operations are not dependent on all of the uncertain inputs and
a significant number of repeated evaluations are eliminated from the
operational level with the AMTC method. As this computational model only
comprises basic arithmetic operations, the repeated evaluations AMTC
eliminated significantly reduces the overall model evaluation time, making the
full-grid NIPC the most efficient UQ method.

(a)

(b)

Figure 4: UQ results on the piston model Uncertain input | No. of dependent operations | Total operations
---|---|---
$M$ | 31 | 65
$S$ | 43 | 65
$V_{0}$ | 45 | 65
Table 4: Number of dependent operations for each uncertain input in the piston
model

### 4.2 Low-fidelity multidisciplinary model

The second UQ problem we consider involves a low-fidelity multidisciplinary
model that computes the total energy stored for a laser-beam-powered unmanned
aerial vehicle (UAV) cruising around a ground station for one cycle while
being charged by the laser beam. The UQ problem aims to compute the
expectation of the total energy stored under three uncertain inputs. We show
the mission plot in Fig. 5 and the input parameters in Tab. 5. The
computational model involves five disciplines, comprising beam propagation,
weights, aerodynamics, and performance models. Further details can be found in
[35].

The performance comparison of the full-grid NIPC method with and without AMTC
is shown in Fig. 6(a), and the convergence results for all of the UQ methods
are shown in Fig. 6(b). We observe that the AMTC method provided approximately
a 90% speed-up in the model evaluation time for the full-grid NIPC method. As
a result, full-grid NIPC with AMTC is more efficient to use than all of the
other UQ methods we implemented. This is not surprising to see, as in the
multidisciplinary model, the parameter uncertainties typically come from
different disciplines and there may exist some uncertain inputs that only
affect a small portion of the operations. In this scenario, the acceleration
provided by the AMTC method can be extremely significant when evaluating on
the full-grid quadrature points. From the dependency information shown in
Tab.6, the uncertain input $\eta$ affects only less than 10% of the operations
in the computational model, while the other two operations affect roughly 50%
of the operations.

Figure 5: A circular cruise mission around a ground station Input parameters | Range | Description
---|---|---
$V$ | $N(100,20)$ |  Velocity (m/s)
$h$ | $N(10,000,2,000)$ |  Altitude(m)
$\eta$ | $N(0.2,0.1)$ | Atmospheric extinction
Table 5: Input parameters and ranges for the multidisciplinary model

(a)

(b)

Figure 6: UQ results on the multi-disciplinary model Uncertain input | No. of dependent operations | Total operations
---|---|---
$V$ | 162 | 285
$h$ | 141 | 285
$\eta$ | 21 | 285
Table 6:  Number of dependent operations for each uncertain input in the
multidisciplinary model

### 4.3 Medium-fidelity multi-point model

The third UQ problem we consider is a multi-point mission analysis problem
involving an electric vertical takeoff and landing (eVTOL) aircraft. The
problem is adapted from [36]. The UQ problem aims to compute the expectation
of the total energy consumption of a lift-plus-cruise eVTOL aircraft concept
(Fig.7) in a two-segment mission including climb and cruise. Two uncertain
inputs are considered in this problem, which are the flight speeds at two
mission segments. The details of the properties for the climb and cruise
segment are presented in Table 7. For each flight segment, we perform a
single-point analysis halfway through the stage and use the vortex-lattice
method (VLM), an incompressible and inviscid aerodynamic analysis method to
compute the lift and drag forces. The details of the computational model can
be found in [37].

Figure 7: Representation of the eVTOL aircraft [36], credit to NASA | Climb segment | Cruise segment
---|---|---
Initial altitude (ft) | 6,000 | 10,000
Final altitude (ft) | 10,000 | 10,000
Flight path angle $\gamma$ (deg) | 20 | 0
Range $R$ (nmi) | $R_{\text{climb}}$ | $37.5-R_{\text{climb}}$
Speed $V$ (Mach) | $N(0.3,0.03)$ | $N(0.5,0.05)$
Table 7:  Properties for climb and cruise segments of the flight mission

We show the performance comparison of the full-grid NIPC method with and
without AMTC in Fig. 8(a) and the UQ convergence plot comparing the five UQ
methods in Fig. 8(b). For this problem, AMTC provided a 70-90% percent
reduction in evaluation time, and the reduction gets more significant as we
increase the number of quadrature points. As a result, the full-grid NIPC with
AMTC method is significantly more efficient to use than the other UQ methods.
The dependency information in Tab. 8 shows roughly 40% of the operations
depend on each uncertain input. The sparsity of the influence matrix is
present because, for this multi-point problem, we have two uncertain inputs
each only affecting the aerodynamic analysis at one point. Although the output
of the model is based on the results from both aerodynamic analyses, each
aerodynamic analysis is only evaluated on the quadrature points in one-
dimensional input space by using the AMTC method. This dramatically
accelerated its model evaluation time on full-grid quadrature points. In fact,
the performance of AMTC can be more significant for many multi-point problems
involving multi-point analyses and multiple uncertain inputs.

(a)

(b)

Figure 8: UQ results on the multi-point model Uncertain input | No. of dependent operations | Total operations
---|---|---
$V_{\text{climb}}$ | 605 | 1505
$V_{\text{cruise}}$ | 604 | 1505
Table 8: Number of dependent operations for each uncertain input in the multi-
point model

## 5 Conclusion

This paper introduces a new method, Accelerated Model evaluations on Tensor
grids using Computational graph transformations (AMTC), designed to notably
reduce the model evaluation cost on tensor grids of input points via
computational graph transformation. It achieves the reduction of model
evaluation cost by using the computational graph of the model and
algorithmically removing the repeated evaluations on each data node so that it
generates the model evaluations on the full-grid input points in an efficient
way. This method can be used with the integration-based non-intrusive
polynomial chaos (NIPC) and stochastic collocation (SC) methods to solve
forward uncertainty quantification problems. To generate the numerical results
in this paper, we implemented AMTC as a graph transformation method within the
three-stage compiler for the Computational System Design Language, so that the
modeling language and its compiler automate the process of detecting and
capitalizing the model structure, achieving the UQ cost reduction in a general
way. We demonstrated the performance of this method on three test problems,
which include an analytical function model, a multidisciplinary model, and a
multi-point model. In each case, AMTC significantly reduces the model
evaluation cost for the full-grid NIPC method, showcasing a speed-up exceeding
50%. More importantly, full-grid NIPC with AMTC achieves the optimal
efficiency among the UQ methods implemented on all of the test problems.
However, we do not expect this method to achieve this level of performance for
any UQ problem, its application holds promise across a broad range of problems
that involve multi-disciplinary models, multi-point models, and other models
with sparse computational graph structures. These are the realms in which AMTC
is poised to markedly enhance the scalability of NIPC and SC methods.

One limitation of this method is that this method assumes a full-grid
structure for the input points. Looking ahead, a key direction for future work
lies in extending AMTC’s functionality to encompass partially tensor-
structured input points. Enabling users to determine the optimal tensor
structure based on an assessment of the computational graph structure could
lead to even lower model evaluation costs via AMTC. Another promising
direction is to enable optimization under uncertainty (OUU) with AMTC. As
common OUU frameworks perform a UQ analysis at each optimization iteration,
the speedup achieved by AMTC for the UQ analysis is vital to solve certain OUU
problems with high-fidelity computational models,

## Acknowledgments

The material presented in this paper is, in part, based upon work supported by
NASA under award No. 80NSSC21M0070 and DARPA under grant No. D23AP00028-00.

## References

## References

  * [1] Susan Joslyn and Sonia Savelli  “Communicating forecast uncertainty: Public perception of weather forecast uncertainty”  In _Meteorological Applications_ 17.2 Wiley Online Library, 2010, pp. 180–195  DOI: 10.1002/met.190
  * [2] F. Pappenberger et al.  “Cascading model uncertainty from medium range weather forecasts (10 days) through a rainfall-runoff model to flood inundation predictions within the European Flood Forecasting System (EFFS)”  In _Hydrology and Earth System Sciences_ 9.4, 2005, pp. 381–393  DOI: 10.5194/hess-9-381-2005
  * [3] Hua-Ping Wan, Zhu Mao, Michael D Todd and Wei-Xin Ren  “Analytical uncertainty quantification for modal frequencies with structural parameter uncertainty using a Gaussian process metamodel”  In _Engineering Structures_ 75 Elsevier, 2014, pp. 577–589  DOI: 10.1016/j.engstruct.2014.06.028
  * [4] Zhen Hu, Sankaran Mahadevan and Dan Ao  “Uncertainty aggregation and reduction in structure–material performance prediction”  In _Computational Mechanics_ 61.1 Springer, 2018, pp. 237–257  DOI: 10.1007/s00466-017-1448-6
  * [5] Leo WT Ng and Karen E Willcox  “Monte Carlo information-reuse approach to aircraft conceptual design optimization under uncertainty”  In _Journal of Aircraft_ 53.2 American Institute of AeronauticsAstronautics, 2016, pp. 427–438  DOI: 10.2514/1.C033352
  * [6] Bingran Wang, Nicholas C Orndorff, Mark Sperry and John T Hwang  “High-dimensional uncertainty quantification using graph-accelerated non-intrusive polynomial chaos and active subspace methods”  In _AIAA AVIATION 2023 Forum_ , 2023, pp. 4264  DOI: 10.2514/6.2023-4264
  * [7] Daejin Lim, Hyeongseok Kim and Kwanjung Yee  “Uncertainty propagation in flight performance of multirotor with parametric and model uncertainties”  In _Aerospace Science and Technology_ 122 Elsevier, 2022, pp. 107398  DOI: 10.1016/j.ast.2022.107398
  * [8] Jeffrey M Wooldridge  “Applications of generalized method of moments estimation”  In _Journal of Economic perspectives_ 15.4 American Economic Association, 2001, pp. 87–100  DOI: 10.1257/jep.15.4.87
  * [9] KB Fragkos, EM Papoutsis-Kiachagias and KC Giannakoglou  “pFOSM: An efficient algorithm for aerodynamic robust design based on continuous adjoint and matrix-vector products”  In _Computers & Fluids_ 181 Elsevier, 2019, pp. 57–66  DOI: 10.1016/j.compfluid.2019.01.016
  * [10] Benjamin Peherstorfer, Karen Willcox and Max Gunzburger  “Optimal model management for multifidelity Monte Carlo estimation”  In _SIAM Journal on Scientific Computing_ 38.5 SIAM, 2016, pp. A3163–A3194  DOI: 10.1137/15M1046472
  * [11] Benjamin Peherstorfer, Karen Willcox and Max Gunzburger  “Survey of multifidelity methods in uncertainty propagation, inference, and optimization”  In _Siam Review_ 60.3 SIAM, 2018, pp. 550–591  DOI: 10.1137/16M1082469
  * [12] Armin Tabandeh, Gaofeng Jia and Paolo Gardoni  “A review and assessment of importance sampling methods for reliability analysis”  In _Structural Safety_ 97 Elsevier, 2022, pp. 102216  DOI: 10.1016/j.strusafe.2022.102216
  * [13] Irfan Kaymaz  “Application of kriging method to structural reliability problems”  In _Structural safety_ 27.2 Elsevier, 2005, pp. 133–151  DOI: 10.1016/j.strusafe.2004.09.001
  * [14] Zhen Hu and Sankaran Mahadevan  “A single-loop kriging surrogate modeling for time-dependent reliability analysis”  In _Journal of Mechanical Design_ 138.6 American Society of Mechanical Engineers, 2016, pp. 061406  DOI: 10.1115/1.4033428
  * [15] Markus P Rumpfkeil  “Optimizations under uncertainty using gradients, Hessians, and surrogate models”  In _AIAA journal_ 51.2 American Institute of AeronauticsAstronautics, 2013, pp. 444–451  DOI: 10.2514/1.J051847
  * [16] Serhat Hosder, Robert Walters and Rafael Perez  “A non-intrusive polynomial chaos method for uncertainty propagation in CFD simulations”  In _44th AIAA aerospace sciences meeting and exhibit_ , 2006, pp. 891  DOI: 10.2514/6.2006-891
  * [17] Brandon A Jones, Alireza Doostan and George H Born  “Nonlinear propagation of orbit uncertainty using non-intrusive polynomial chaos”  In _Journal of Guidance, Control, and Dynamics_ 36.2 American Institute of AeronauticsAstronautics, 2013, pp. 430–444  DOI: 10.2514/1.57599
  * [18] Vahid Keshavarzzadeh, Felipe Fernandez and Daniel A Tortorelli  “Topology optimization under uncertainty via non-intrusive polynomial chaos expansion”  In _Computer Methods in Applied Mechanics and Engineering_ 318 Elsevier, 2017, pp. 120–147  DOI: 10.1016/j.cma.2017.01.019
  * [19] Dongbin Xiu and Jan S Hesthaven  “High-order collocation methods for differential equations with random inputs”  In _SIAM Journal on Scientific Computing_ 27.3 SIAM, 2005, pp. 1118–1139  DOI: 10.1137/040615201
  * [20] Ivo Babuška, Fabio Nobile and Raúl Tempone  “A stochastic collocation method for elliptic partial differential equations with random input data”  In _SIAM Journal on Numerical Analysis_ 45.3 SIAM, 2007, pp. 1005–1034  DOI: 10.1137/050645142
  * [21] Norbert Wiener  “The homogeneous chaos”  In _American Journal of Mathematics_ 60.4 JSTOR, 1938, pp. 897–936  DOI: 10.2307/2371268
  * [22] Dongbin Xiu and George Em Karniadakis  “The Wiener–Askey polynomial chaos for stochastic differential equations”  In _SIAM journal on scientific computing_ 24.2 SIAM, 2002, pp. 619–644  DOI: 10.1137/S1064827501387826
  * [23] Fabio Nobile, Raúl Tempone and Clayton G Webster  “A sparse grid stochastic collocation method for partial differential equations with random input data”  In _SIAM Journal on Numerical Analysis_ 46.5 SIAM, 2008, pp. 2309–2345  DOI: 10.1137/060663660
  * [24] Vahid Keshavarzzadeh, Robert M Kirby and Akil Narayan  “Numerical integration in multiple dimensions with designed quadrature”  In _SIAM Journal on Scientific Computing_ 40.4 SIAM, 2018, pp. A2033–A2061  DOI: 10.1137/17M1137875
  * [25] Géraud Blatman and Bruno Sudret  “Sparse polynomial chaos expansions and adaptive stochastic finite elements using a regression approach”  In _Comptes rendus mécanique_ 336.6 Elsevier, 2008, pp. 518–523  DOI: 10.1016/j.crme.2008.02.013
  * [26] Mishal Thapa, Sameer B Mulani and Robert W Walters  “Adaptive weighted least-squares polynomial chaos expansion with basis adaptivity and sequential adaptive sampling”  In _Computer Methods in Applied Mechanics and Engineering_ 360 Elsevier, 2020, pp. 112759  DOI: 10.1016/j.cma.2019.112759
  * [27] Nora Lüthen, Stefano Marelli and Bruno Sudret  “Sparse polynomial chaos expansions: Literature survey and benchmark”  In _SIAM/ASA Journal on Uncertainty Quantification_ 9.2 SIAM, 2021, pp. 593–649  DOI: 10.1137/20M1315774
  * [28] Michael Eldred and John Burkardt  “Comparison of non-intrusive polynomial chaos and stochastic collocation methods for uncertainty quantification”  In _47th AIAA aerospace sciences meeting including the new horizons forum and aerospace exposition_ , 2009, pp. 976  DOI: 10.2514/6.2009-976
  * [29] Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul and Jeffrey Mark Siskind  “Automatic Differentiation in Machine Learning: a Survey”  In _Journal of Machine Learning Research_ 18.153, 2018, pp. 1–43  URL: http://jmlr.org/papers/v18/17-468.html
  * [30] Mark Sperry, Kavish Kondap and John T Hwang  “Automatic adjoint sensitivity analysis of models for large-scale multidisciplinary design optimization”  In _AIAA AVIATION 2023 Forum_ , 2023, pp. 3721  DOI: 10.2514/6.2023-3721
  * [31] Martín Abadi  “TensorFlow: learning functions at scale”  In _Proceedings of the 21st ACM SIGPLAN international conference on functional programming_ , 2016, pp. 1–1  DOI: 10.1145/2951913.2976746
  * [32] Arthur B Kahn  “Topological sorting of large networks”  In _Communications of the ACM_ 5.11 ACM New York, NY, USA, 1962, pp. 558–562  DOI: 10.1145/368996.369025
  * [33] Victor Gandarillas et al.  “A graph-based methodology for constructing computational models that automates adjoint-based sensitivity analysis”  In _Structural and Multidisciplinary Optimization_ , (under revision)
  * [34] Einat Neumann Ben-Ari and David M Steinberg  “Modeling data from computer experiments: an empirical comparison of kriging with MARS and projection pursuit regression”  In _Quality Engineering_ 19.4 Taylor & Francis, 2007, pp. 327–338  DOI: 10.1080/08982110701580930
  * [35] Bingran Wang, Nicholas C Orndorff and John T Hwang  “Optimally tensor-structured quadrature rule for uncertainty quantification”  In _AIAA SCITECH 2023 Forum_ , 2023, pp. 0741  DOI: 10.2514/6.2023-0741
  * [36] Christopher Silva et al.  “VTOL urban air mobility concept vehicles for technology development”  In _2018 Aviation Technology, Integration, and Operations Conference_ , 2018, pp. 3847  DOI: 10.2514/6.2018-3847
  * [37] Bingran Wang, Mark Sperry, Victor E Gandarillas and John T Hwang  “Efficient uncertainty propagation through computational graph modification and automatic code generation”  In _AIAA AVIATION 2022 Forum_ , 2022, pp. 3997  DOI: 10.2514/6.2022-3997


# Zero-Shot Co-salient object detection Framework

###### Abstract

Co-salient Object Detection (CoSOD) endeavors to replicate the human visual
system’s capacity to recognize common and salient objects within a collection
of images. Despite recent advancements in deep learning models, these models
still rely on training with well-annotated CoSOD datasets. The exploration of
training-free zero-shot CoSOD frameworks has been limited. In this paper,
taking inspiration from the zero-shot transfer capabilities of foundational
computer vision models, we introduce the first zero-shot CoSOD framework that
harnesses these models without any training process. To achieve this, we
introduce two novel components in our proposed framework: the group prompt
generation (GPG) module and the co-saliency map generation (CMP) module. We
evaluate the framework’s performance on widely-used datasets and observe
impressive results. Our approach surpasses existing unsupervised methods and
even outperforms fully supervised methods developed before 2020, while
remaining competitive with some fully supervised methods developed before
2022.

Index Terms—  Zero-shot Co-saliency Detection, Foundational Computer Vision
Model.

##  1 Introduction

Co-salient object detection (CoSOD) is a task that seeks to replicate the
human visual system’s ability to identify common and salient objects from a
set of related images. One of the unique challenges in CoSOD is that co-
salient objects belong to the same semantic category, but their specific
category attributes remain unknown. These distinctive characteristics have
made CoSOD an emerging and demanding task that has gained rapid traction in
recent years [1, 2, 3].

Fig. 1: Left: The architecture of our proposed zero-shot CoSOD framework.
Right: The performance of our proposed zero-shot CoSOD framework. GWD [4],
RCAN [5], ICNet [6], CADC [7] and UFO [8] are five typical methods.

Detecting co-saliency fundamentally hinges on accurately modeling the inter
relation within an image group. To tackle this challenge, various meticulously
designed architectures have been proposed, including RNN-based methods [5, 9],
CNN-based methods [10, 6, 11], and Transformer-based methods [8, 12]. While
these methods have achieved impressive performance, they often rely on small-
scale datasets and require the integration of complex network modules. It’s
important to highlight that previous studies [1, 9] show that changing the
training data while keeping the same network architecture, or modifying the
backbone network while using the same training data can significantly impact
network performance. This suggests that improving network performance might be
attainable by employing a more robust backbone network or higher-quality
training datasets. These considerations prompt us to reevaluate whether
addressing the CoSOD task necessitates the design of intricate and diverse
modules, or if an alternative approach should be explored.

Recently, foundational computer vision (CV) models, such as SAM [13] and DINO
[14], have emerged. These models, once trained, can be seamlessly applied to
various downstream tasks in a zero-shot manner, eliminating the need for
dataset-specific fine-tuning. This prompts us to explore whether these CV
foundational models can be harnessed for CoSOD. However, existing CV
foundational models, like SAM, are tailored for single-image tasks and lack
the capacity to discern inter-saliency relations within an image group.
Moreover, manually supplying SAM with inter-saliency or group prompts, which
aid in co-saliency map generation, is impractical due to the nature of the
CoSOD task.

Fig. 2: The architecture of our proposed zero-shot CoSOD framework. Feature
extraction is accomplished by utilizing DINO and SD to extract both high-level
and low-level information. The CMP module employs SAM to generate the co-
saliency maps. Importantly, all parameters in the network remain frozen,
eliminating the need for additional training.

To tackle the aforementioned challenges, we present an innovative zero-shot
CoSOD framework that leverages foundational computer vision models. As
depicted in Fig. 1, our framework consists of two main components: group
prompt generation (GPG) and co-saliency map generation (CMP). In the GPG
module, we initially extract high-level semantic information from each image
using the CV foundational model. We also explore the supplementation of low-
level spatial details using Stable Diffusion (SD) [15], which may not be
captured by the foundational model. Subsequently, we combine these pieces of
information to generate group prompts. These prompts, created by the GPG
module, serve as input for the CMP module. As depicted in Fig. 1, our network
surpasses methods developed before 2021. Our key contributions are:

  * •

We take the pioneering step of introducing a zero-shot CoSOD framework,
potentially inspiring researchers to address the CoSOD from a fresh
perspective.

  * •

To address the limitations of existing CV foundational model when applied to
CoSOD task, we further design the GPG and CMP modules.

  * •

We validate our zero-shot CoSOD framework on three widely used datasets (CoCA
[16], CoSOD3k [2] and Cosal2015 [17]), and the performance shows the
effectiveness of our proposed zero-shot framework.

##  2 METHOD

In Fig. 2, since the CMP module directly utilizes SAM, our emphasis lies in
providing an in-depth description of the key components within GPG. GPG
encompasses feature extraction, group center proxy generation, and TopK
selection.

###  2.1 Feature Extraction

High-level Feature Extraction. Existing works [18, 14] demonstrate that self-
supervised ViT features, as exemplified by DINO, contain explicit information
for semantic segmentation and excel as KNN classifiers. In essence, DINO is
adept at accurately extracting the semantic content of each image, a vital
aspect in discerning the group features within an image set. Herein, we choose
the 11th layer feature $\mathcal{F}_{DINO}$ to represent the semantic
information of each image.

Fig. 3: The generated group features.

Low-level Feature Extraction. While DINO excels in providing substantial high-
level semantics, it lacks in delivering nuanced low-level spatial information.
In the second column of Fig. 3, the group feature generated solely through
DINO would lack low-level detailed information. As emphasized in previous
studies [19, 9], both low-level and high-level features are pivotal for
modeling group features. However, it’s noteworthy that there is a research gap
regarding the supplementation of low-level spatial information to features
extracted by DINO in a zero-shot manner.

In our proposed network, the inclusion of a pre-trained model that specializes
in low-level spatial information becomes crucial, particularly in scenarios
lacking strong texture cues. Such a model can effectively complement the low-
level spatial information extracted by DINO. Notably, SD [15] has recently
showcased its exceptional ability to generate high-quality images. This
underscores its potential for robustly representing images, encompassing both
content and spatial information. Consequently, our primary objective is to
explore whether SD features can enhance the establishment of inter-
relationships when combined with DINO.

The architecture of SD comprises three key components: an encoder
$\mathcal{E}$, a decoder $\mathcal{D}$, and a denoising U-Net $\mathcal{U}$
operating within the latent space. We begin by projecting an input image
$x_{0}$ into the latent space through the encoder $\mathcal{E}$, resulting in
a latent code $z_{0}$. Subsequently, we add Gaussian noise $\epsilon$ to the
latent code according to a predefined time step $t$. Lastly, with the latent
code $z_{t}$ at time step $t$, we then extract the SD features
$\mathcal{F}_{SD}$ utilizing the denoising U-Net:

$\displaystyle\mathcal{F}_{SD}=\mathcal{U}(z_{t},t),\
z_{t}=\sqrt{\bar{a}_{t}}+\sqrt{1-\bar{a}_{t}}\epsilon,\
z_{0}=\mathcal{E}(x_{0}).$ (1)

In accordance with the approach introduced in [20], we combine features
extracted from different decoder $\mathcal{D}$ layers, specifically layers 2,
5, and 8, to capture multi-scale features. However, a direct concatenation of
features from these three layers results in an excessively high-dimensional
feature vector, approximately 5440 dimensions. To address this issue, we
employ Principal Component Analysis (PCA) for each feature layer.
Subsequently, we upsample lower-resolution features (i.e., layers 2 and 5) to
match the resolution of the higher-resolution layer (i.e., layer 8) before
concatenation.

Feature Fusion. Expanding on the aforementioned discussions, we introduce a
simple yet remarkably effective fusion strategy aimed at harnessing the
advantages of both SD and DINO features. The core idea entails independently
normalizing both sets of features to ensure consistency in their scales and
distributions, subsequently concatenating them:

$\displaystyle\mathcal{F}_{FUSE}=Concat(\|\mathcal{F}_{SD}\|_{2},\|\mathcal{F}_{DINO}\|_{2}).$
(2)

In the third column of Fig. 3, the fused feature aids in generating a smoother
and more resilient group feature.

###  2.2 Group Center Proxy Generation and TopK Selecting

We acquire the feature $\mathcal{F}_{FUSE}\in\mathbb{R}^{C\times H\times W}$
for each image, a valuable asset for robust group information generation.
However, the subsequent challenge lies in disseminating the group information
to individual images. In existing CoSOD methods, group information is often
expressed as a feature map, directly concatenated with the original image.
This is followed by a trainable decoder for co-saliency map prediction. In our
proposed zero-shot framework, training a new decoder network using the CoSOD
training dataset is not feasible. To tackle this challenge, we transform the
representation of group information and introduce the pixel group center proxy
generation and TopK selecting mechanism. Through this innovative approach, we
can create group prompt points to represent co-salient objects in each images,
which, in turn, prompt SAM to generate the corresponding maps.

Assuming an image group comprises $N$ images, we concatenate these features to
generate the group feature $\mathcal{F}_{G}\in\mathbb{R}^{N\times C\times
H\times W}$. Subsequently, we reshape $\mathcal{F}_{G}$ into the shape
$\mathbb{R}^{NHW\times C}$, denoting each pixel embedding as
$\mathcal{F}_{G}^{ln}$, where $l\in[1,NHW]$ and $n$ means the $n$-th image.
$\mathcal{F}_{G}^{ln}$ means that the $l$-th pixel embedding belongs to $n$-th
image. Then, we use the easy averaging operation on these pixel embeddings to
generate the group center proxy $\mathcal{F}_{c}$. Moreover, to make the group
center proxy focus on salient regions, we use the unsupervised SOD method TSDN
[21] to filter pixels belong to non-salient regions in $\mathcal{F}_{G}^{ln}$,
generating the salient pixels $\mathcal{F}_{G}^{ln-s}$. The process of the
generation of $\mathcal{F}_{c}$ is written as:

$\displaystyle\mathcal{F}_{c}=Avg\left\\{\mathcal{F}_{G}^{ln-s}\right\\}\in\mathbb{R}^{C}.$
(3)

Concretely, for $N$-th image which contains $L$ salient pixels, we calculate
the correlation score between $\mathcal{F}_{c}$ and $\mathcal{F}_{G}^{LN}$,
and use TopK to select the point at position $P^{N}$ in the image $N$, which
can represent common co-salient objects in this image:

$\displaystyle
S^{LN}=\mathcal{F}_{c}\otimes\mathcal{F}^{LN},P^{N}={\text{TopK}}(S^{LN})\in\mathbb{R}^{K},$
(4)

where $\otimes$ means matrix multiplication. Finally, for the $N$-th image,
the generated prompts at position $P^{N}$ (Fig. 1 and Fig. 4) and the
corresponding original image is sent to CMP to generate the co-saliency maps.
We set $K=2$ in this paper.

##  3 EXPERIMENTS

###  3.1 Experimental Setup

Implementation Details. We employ the Stable Diffusion v1-5 and DINOv2 models
as our feature extractors, with the DDIM timestep $t$ in the denoising process
set to be 50 by default. We use the SAM with the smallest vit-b backbone. $N$
is the total image numbers of one certain image group. All experiments are
conducted on a single RTX 3090 GPU.

Fig. 4: Visual comparison between our method and other methods.  Table 1:
Quantitative comparison with SOTA on CoSOD datasets. The  red font means that
our proposed network achieves the competitive performance compared to these
methods.

|  |  |  | CoCA | CoSOD3k | CoSal2015
---|---|---|---|---|---|---
Performance Level | Methods | Year | Type | $S_{m}\uparrow$ | $F^{mean}_{\beta}\uparrow$ | $MAE\downarrow$ | $S_{m}\uparrow$ | $F^{mean}_{\beta}\uparrow$ | $MAE\downarrow$ | $S_{m}\uparrow$ | $F^{mean}_{\beta}\uparrow$ | $MAE\downarrow$
| TCNet | TCSVT 2023 | Supervised | 0.685 | 0.548 | 0.101 | 0.832 | 0.797 | 0.068 | 0.870 | 0.859 | 0.054
| UFO | TMM 2023 | Supervised | 0.697 | 0.555 | 0.095 | 0.819 | 0.783 | 0.073 | 0.860 | 0.848 | 0.064
Future | CoRP | TPAMI 2023 | Supervised | 0.732 | 0.603 | 0.093 | 0.850 | 0.824 | 0.057 | 0.884 | 0.884 | 0.044
| ICNet | NeurIPS 2020 | Supervised | 0.651 | 0.503 | 0.148 | 0.780 | 0.734 | 0.097 | 0.856 | 0.846 | 0.058
| CADC | ICCV 2021 | Supervised | 0.681 | 0.503 | 0.132 | 0.801 | 0.742 | 0.096 | 0.866 | 0.825 | 0.064
Competitive | GLNet | TCyb 2022 | Supervised | 0.591 | 0.435 | 0.188 | - | - | - | 0.855 | 0.849 | 0.060
| CSMG | CVPR 2019 | Supervised | 0.632 | 0.494 | 0.124 | 0.711 | 0.682 | 0.157 | 0.774 | 0.775 | 0.130
| PJO | TIP2018 | Unsupervised | 0.573 | 0.362 | 0.175 | 0.677 | 0.631 | 0.188 | 0.721 | 0.687 | 0.192
| GOMAG | TMM 2020 | Unsupervised | 0.587 | 0.387 | 0.170 | 0.687 | 0.642 | 0.180 | 0.734 | 0.698 | 0.187
| Ours (-SD) |  |  | 0.653 | 0.532 | 0.121 | 0.717 | 0.679 | 0.127 | 0.776 | 0.787 | 0.110
Outperform | Ours | 2023 | Zero Shot | 0.667 | 0.549 | 0.115 | 0.723 | 0.691 | 0.117 | 0.785 | 0.799 | 0.101

Table 2: Performance improvement of existing methods after adding group
information extracted by DINO and SD.

|  | CoCA | CoSOD3k | CoSal2015
---|---|---|---|---
Methods | Pub. &Year | $S_{m}\uparrow$ | $F^{mean}_{\beta}\uparrow$ | $MAE\downarrow$ | $S_{m}\uparrow$ | $F^{mean}_{\beta}\uparrow$ | $MAE\downarrow$ | $S_{m}\uparrow$ | $F^{mean}_{\beta}\uparrow$ | $MAE\downarrow$
TSCoSOD | TIP 2022 | 0.732 (0.741) | 0.613 (0.625) | 0.099 (0.093) | 0.843 (0.856) | 0.816 (0.831) | 0.062 (0.052) | 0.898 (0.905) | 0.897 (0.909) | 0.045 (0.039)
TCNet | TCSVT 2023 | 0.685 (0.697) | 0.548 (0.561) | 0.101 (0.092) | 0.832 (0.843) | 0.797 (0.814) | 0.068 (0.061) | 0.870 (0.877) | 0.859 (0.877) | 0.054 (0.046)
GCoNet+ | TPAMI 2023 | 0.738 (0.749) | 0.612 (0.624) | 0.081 (0.077) | 0.843 (0.855) | 0.813 (0.827) | 0.062 (0.054) | 0.881 (0.891) | 0.870 (0.884) | 0.056 (0.045)
Average Improvement | +1.46% | +2.05% | +6.63% | 1.41% | +1.86% | +13.11% | +2.69% | +1.63% | +15.93%

Datasets. We employ three benchmark datasets, including Cosal2015 [22],
CoSOD3k [2] and CoCA [16], to evaluate our approach. Cosal2015 comprises 50
groups with a total of 2015 images. It presents numerous challenges such as
complex environments. CoSOD3k, the largest-scale and most comprehensive
benchmark, offers 160 groups with a total of 3000 images. CoCA features 80
groups with a total of 1297 images, posing a challenge due to the presence of
multiple objects, including relatively small co-salient objects.

Evaluation Metrics. we employ three widely used criteria: (1) F-measure
$(F_{\beta}^{mean})$, representing the harmonic mean of precision and recall
values, is calculated using a self-adaptive threshold. (2) Structure Measure
$(S_{m})$, which is utilized to assess the spatial structural similarities of
saliency maps. (3) Mean Absolute Error $(MAE)$, which quantifies the average
L1 distance between ground truth maps and predictions.

###  3.2 Comparison Methods

We compare our method with 7 fully-supervised methods: CSMG [23], ICNet [6],
CADC [7], GLNet [24], TCNet [25], CoRP [1] and UFO [8]. The unsupervised CoSOD
models in barely explored, we only compare our method with 2 methods: PJO [26]
and GOMAG [27].

###  3.3 Quantitative and Qualitative Evaluation.

Table. 1 reveals that our proposed zero-shot CoSOD network consistently
outperforms all other state-of-the-art (SOTA) unsupervised CoSOD methods
across all evaluation metrics. These results underscore the efficacy of our
zero-shot CoSOD network. In comparison to supervised CoSOD methods, our
approach surpasses methods published in 2019 and achieves competitive
performance compared to those published from 2020 to 2023, as evidenced by
certain metrics. It’s worth noting that all modules in our network employ
basic design principles, such as simple averaging operations for center proxy
point feature extraction. In this configuration, our zero-shot framework
achieves such remarkable performance, instilling confidence in researchers to
explore CoSOD tasks from a novel zero-shot perspective. Fig. 4 presents
qualitative results from our proposed method, showcasing its ability to
accurately detect co-salient objects even in complex scenes.

###  3.4 Ablation Analysis

First, we propose that both high-level and low-level information are pivotal
for generating group features. To validate this assertion, we conduct
experiments. In the penultimate row of Table. 1, utilizing only the high-level
features extracted from DINO already achieves competitive performance within
the proposed network. However, it does not enable our zero-shot framework to
completely surpass the CSMG method in all metrics. Additionally, the
incorporation of low-level information from SD further enhances performance.

Another noteworthy contribution of this paper is the assertion that features
extracted from foundational models are valuable for generating group features.
Consequently, when we incorporate these group features into existing
frameworks, including TSCoSOD [9], TCNet [25] and GCoNet+ [28], we observe
further performance improvements, as demonstrated in Table. 2. The values in
parentheses represent results after retraining with the inclusion of new group
features. This underscores that, beyond the framework itself, this paper
contributes a zero-shot group feature generation approach.

##  4 conclusion

In this paper, we introduce an innovative zero-shot CoSOD framework.
Leveraging the feature extraction capabilities of established DINO and SD, we
have devised the GPG and CMP components, enabling the application of existing
foundational models to the zero-shot CoSOD task. Our experiments demonstrate
that these foundational models can effectively generate resilient group
features, and our proposed framework can reasonably address the zero-shot
CoSOD task. We envision that our work will serve as a cornerstone for the
zero-shot/unsupervised CoSOD task, inspiring researchers to approach CoSOD
from a novel perspective.

## References

  * [1] Ziyue Zhu, Zhao Zhang, Zheng Lin, Xing Sun, and Ming-Ming Cheng,  “Co-salient object detection with co-representation purification,”  TPAMI, 2023.
  * [2] Deng-Ping Fan, Tengpeng Li, Zheng Lin, Ge-Peng Ji, Dingwen Zhang, Ming-Ming Cheng, and Huazhu Fu,  “Re-thinking co-salient object detection,”  TPAMI, 2022.
  * [3] Dingwen Zhang, Huazhu Fu, Junwei Han, and Ali Borji,  “A review of co-saliency detection algorithms: Fundamentals, applications, and challenges,”  TIST, 2018.
  * [4] Lina Wei, Shanshan Zhao, Omar El Farouk Bourahla, Xi Li, and Fei Wu,  “Group-wise deep co-saliency detection,”  in IJCAI, 2017.
  * [5] Bo Li, Zhengxing Sun, Lv Tang, Yunhan Sun, and Jinlong Shi,  “Detecting robust co-saliency with recurrent co-attention neural network,”  in IJCAI, 2019.
  * [6] Wenda Jin, Jun Xu, Ming-Ming Cheng, Yi Zhang, and Wei Guo,  “Icnet: Intra-saliency correlation network for co-saliency detection,”  in NIPS, 2020.
  * [7] Ni Zhang, Junwei Han, Nian Liu, and Ling Shao,  “Summarize and search: Learning consensus-aware dynamic convolution for co-saliency detection,”  in ICCV, 2021.
  * [8] Yukun Su, Jingliang Deng, Ruizhou Sun, and Guosheng Lin,  “A unified transformer framework for group-based segmentation: Co-segmentation, co-saliency detection and video salient object detection,”  TMM, 2023.
  * [9] Bo Li, Lv Tang, and Senyun Kuang,  “Toward stable co-saliency detection and object co-segmentation,”  TIP, 2022.
  * [10] Qijian Zhang, Runmin Cong, Junhui Hou, Chongyi Li, and Yao Zhao,  “Coadnet: Collaborative aggregation-and-distribution networks for co-salient object detection,”  in NIPS, 2020.
  * [11] Kaihua Zhang, Mingliang Dong, Bo Liu, Xiao-Tong Yuan, and Qingshan Liu,  “Deepacg: Co-saliency detection via semantic-aware contrast gromov-wasserstein distance,”  in CVPR, 2021.
  * [12] Long Li, Junwei Han, Ni Zhang, Nian Liu, and Salman H. Khan,  “Discriminative co-saliency and background mining transformer for co-salient object detection,”  in CVPR, 2023.
  * [13] Alexander Kirillov, Eric Mintun, and etc at. Nikhila Ravi,  “Segment anything,”  CoRR, 2023.
  * [14] Maxime Oquab, Timothée Darcet, and etc at. Théo Moutakanni,  “Dinov2: Learning robust visual features without supervision,”  CoRR, 2023.
  * [15] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer,  “High-resolution image synthesis with latent diffusion models,”  in CVPR, 2022.
  * [16] Zhao Zhang, Wenda Jin, Jun Xu, and Ming-Ming Cheng,  “Gradient-induced co-saliency detection,”  in ECCV, 2020.
  * [17] Dingwen Zhang, Junwei Han, Chao Li, Jingdong Wang, and Xuelong Li,  “Detection of co-salient objects by looking deep and wide,”  IJCV, 2016.
  * [18] Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool,  “Warp consistency for unsupervised learning of dense correspondences,”  in ICCV, 2021.
  * [19] Kaihua Zhang, Tengpeng Li, Shiwen Shen, Bo Liu, Jin Chen, and Qingshan Liu,  “Adaptive graph convolutional network with attention graph clustering for co-saliency detection,”  in CVPR, 2020.
  * [20] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu,  “Unleashing text-to-image diffusion models for visual perception,”  CoRR, 2023.
  * [21] Huajun Zhou, Bo Qiao, Lingxiao Yang, Jianhuang Lai, and Xiaohua Xie,  “Texture-guided saliency distilling for unsupervised salient object detection,”  in CVPR, 2023.
  * [22] Dingwen Zhang, Junwei Han, Chao Li, and Jingdong Wang,  “Co-saliency detection via looking deep and wide,”  in CVPR, 2015.
  * [23] Kaihua Zhang, Tengpeng Li, and Bo Liu,  “Co-saliency detection via mask-guided fully convolutional networks with multi-scale label smoothing,”  in CVPR, 2019.
  * [24] Runmin Cong, Ning Yang, Chongyi Li, Huazhu Fu, and Yao Zhao,  “Global-and-local collaborative learning for co-salient object detection,”  TCyb, 2022.
  * [25] Yanliang Ge, Qiao Zhang, Tian-Zhu Xiang, and Cong Zhang,  “Tcnet: Co-salient object detection via parallel interaction of transformers and cnns,”  TCSVT, 2023.
  * [26] Chung-Chi Tsai, Weizhi Li, and Kuang-Jui Hsu,  “Image co-saliency detection and co-segmentation via progressive joint optimization,”  TIP, 2019.
  * [27] Bo Jiang, Xingyue Jiang, Jin Tang, and Bin Luo,  “Co-saliency detection via a general optimization model and adaptive graph learning,”  TMM, 2021.
  * [28] Peng Zheng, Huazhu Fu, Deng-Ping Fan, Qi Fan, Jie Qin, Yu-Wing Tai, Chi-Keung Tang, and Luc Van Gool,  “Gconet+: A stronger group collaborative co-salient object detector,”  TPAMI, 2023.