two utility functions.

In general, we found that the optimal time to retire the cell is most closely
related to the observed rate of capacity loss (essentially the slope of the
capacity vs cycle number curve). Typically, the optimal replacement time is
shortly after the cell’s knee point, which is the bend in the capacity fade
curve where more rapid capacity loss begins to occur. Intuitively, this makes
sense, as the total $Ah$ utility can remain high, because the cell is still
able to output a great deal of power even after its knee point, but the mean
time between charges begins to rapidly increase, because the capacity is
fading more quickly. As the capacity fades, the mean time between charges
decreases, and the customer has to charge their vehicle more frequently, which
is not desirable. In turn, the utility function for the mean time between
charges decreases in value to reflect this.

Lastly, in testing, we observed that the optimal retirement time was not
drastically affected by the particle filter’s projected shape of the capacity
fade trajectory. As long as the shape was fairly similar to that of the
measured curve, the results were consistent. What we did notice, however, was
that for cells with longer lifetimes, the particle filter would significantly
underestimate the capacity trajectory, and the optimal replacement time would
move closer to the current cycle to account for the perceived earlier EOL.

Figure 17: Optimal cycle to retire primary test cell 28 from its first life
application in an electric vehicle using the projected capacity from the
particle filter. Figure 18: Optimal cycle to retire secondary test cell 18
from its first life application in an electric vehicle using the projected
capacity from the particle filter. Figure 19: Optimal cycle to retire
secondary test cell 35 from its first life application in an electric vehicle
using the projected capacity from the particle filter.

###  3.4 Case study conclusion and ideas for future research

Altogether, this case study demonstrated how one could create a battery
digital twin for optimizing the retirement time of a Li-ion cell from its
first life use. The success of the proposed digital twin framework relies on
the integration of multiple individual pieces of software, which together,
form an intelligent model capable of informing engineers and practitioners of
optimal retirement times on a cell-by-cell basis.

The first key piece of software, the particle filter prognostic model, was
found to accurately predict the RUL of cells with varying lifetimes. However,
the particle filter used in this study was limited in that it only considered
a single capacity fade model, the power law model. In future research, it
would be worthwhile to investigate a multi-model particle filter, which
actively switches between different capacity fade models (Li et al., 2021b, Ye
et al., 2018, Boers and Driessen, 2003).

The second key piece of software, the multi-attribute utility optimization
model, was found to provide advanced notice of the optimal time to retire a
cell from its first life application using the projected future capacity from
the particle filter model. However, the optimization model struggled to
provide meaningful optimal replacement predictions for cells which have much
longer lifetimes. In future work, it would prove useful to investigate better
methods of defining the many utility functions so that they can be applied to
a wider range of cells. Additionally, this case study did not consider any
preferences (attributes) relevant to the second life application of the Li-ion
cell. The preferences of the second-life user were not considered for two main
reasons. First, this was done to simplify the case study, as the goal was to
demonstrate and explain in great detail the different software components that
ultimately comprise the proposed digital twin model. Second, not considering
the preferences of the second-life application simplified the optimization
model, making it much easier to visualize and explain how it works. In
reality, it is imperative to consider the preferences of all the involved
parties when determining the optimal time to remove a Li-ion cell from its
first life application. Both the electric vehicle manufacturer and second life
grid-scale energy storage management company would have input on the
attributes used in the optimization model, so that both parties’ preferences
are considered when determining the optimal time to switch from the first to
second life application. However, we leave this to future work.

Lastly, the main strength of the particle filter is that it can output a non-
parametric probabilistic prediction of a cell’s capacity trajectory and end of
life. However, this case study used a deterministic method to determine the
optimal time to retire a cell from its first life application. In practice, it
is desirable to present practitioners with confidence levels and intervals,
instead of point estimates. Therefore, in future work, it would prove
insightful to investigate probabilistic methods of evaluating multi-attribute
utility functions, so that an optimal replacement interval can be defined,
instead of a single-point prediction. Since the utility function approach to
optimization is relatively lightweight, it may prove tractable to use sampling
approaches to yield probabilistic results. Similar ideas are further discussed
in Sec. 5.2 of Part 1 and Sec. 2.2.3, a probabilistic decision-making
framework that considers model uncertainty (see Sec. 2.1) empowers maintenance
engineers and practitioners with more information to make the bests decisions
possible.

##  4 Demonstration and open source

###  4.1 Industry-scale demonstration of digital twin

Digital twins have become one of the key ingredients in improving the business
outcomes across a wide range of industries. The global digital twin market is
projected to grow from USD 3.1 billion in 2020 to USD 48.2 billion by 2026
(Markets and Markets, 2020). In many industries, digital twins have emerged as
an integral part of the digitization efforts that started almost two decades
ago. While the first wave of digitization focused on reducing physical
databases and interactions, the second wave happening now is when the digital
twin concept really kicked-off. Specifically, the second wave of digitization
focuses on creating a virtual model encompassing a physical asset or process
throughout its life cycle so as to analyze, predict and optimize the asset and
improve business outcomes. Most use cases of digital twins in industry fall
under the categories of predictive maintenance, asset life cycle management,
process planning & optimization, product design, virtual prototyping and more
(Sjarov et al., 2020). The benefit of incorporating digital twins is multi-
fold when solving various engineering and business problems, such as reducing
costs and risk, improving efficiency, security, reliability, resilience, and
supporting decision-making (VanDerHorn and Mahadevan, 2021). Digital twins
have been successfully applied to a broad range of industries, such as
manufacturing, software, aerospace, agriculture, automobiles, healthcare,
consumer goods, etc, and the usage of digital twins is still growing at a fast
speed (Durão et al., 2018, Augustine, 2020).

In the aerospace industry, digital twin effort and conceptualization was
pioneered and directed by NASA and the U.S. Air Force in early 2000s (see, for
example, Glaessgen and Stargel (2012)). Since then, digital twin has been
expanded to many areas, such as airframe, avionics, crack detection, feet
level health management, etc. Digital twin has also been one of critical
technologies that is listed as a part of future strategy for many leading
companies, such as Boeing and Airbus (Aydemir et al., 2020). In GE electric,
more than 500,000 digital twins have been deployed in various capacities
(i.e., parts twins, products twins, process twins and system twins) across its
production sites . In addition, GE aviation has reduced the amount of time for
anomaly detections based on sensor data by 15-30 days. The reduction of
anomaly detection time helps to achieve $44 million savings in the life cycle
management for turbine blade engines, and $10 million savings in dynamic
optimization to deliver optimal flight patterns and maintenance.

In the commercial sector, Boeing has reported 40% improvement in first-time
quality of the parts and systems (Bellamy III, 2018). They have been creating
and maintaining operational digital twins of components to track each
individual component’s unique characteristics and detect degradation rates.
Similarly, in the defense sector, Boeing is using digital twins to predict and
find possible fatigue maintenance hot spots in the F15 Eagle to reduce
operation and maintenance costs. More recently, Boeing is planning to build an
airplane in the metaverse using a digital twin (Careless, 2021).

In the automobile industry, digital twins have been playing an important role
across all the stages of vehicle life cycle development, such as concept
development, detailed design, and design verification (Sharma and George,
2018). In addition, digital twins have been used to improve the manufacturing
and production in the automobile manufacturing plants. For example, BMW has
recently used a virtual factory built upon digital twins and have produced 30%
more efficient planning processes (Caulfield, 2022). In the healthcare sector,
one of the main application areas lies in the development of technologies for
monitoring, digital surgery, remote surgical assistance, drug development,
medical treatment etc. using high-fidelity digital twin models of human bodies
(Skardal et al., 2016, Bruynseels et al., 2018, Liu et al., 2019). A digital
twin can also be used in building a virtual model of a physical city, also
known as “smart city”, to map the urban information system to into space and
time. It has been used to solve a broad range of problems, such as water
treatment, maintenance of facilities, and so on (Chen et al., 2018, Lehner and
Dorffner, 2020, Lu et al., 2020).

Additionally, digital companies like Siemens, Ansys, IBM, and others have been
playing a pivotal role in providing the infrastructure and platform to support
the adoptions of digital twin by both small and large-scale industries. For
example, Siemens has been one of the leading companies developing digital twin
technologies for products, production and performance of various mechanics,
electronics, software, or systems to minimize cost and optimize the
requirements (Siemens Newsletter, 2020). Some of Siemen’s product like Simatic
Real-time Locating Systems (RTLS), Teamcenter X software, and UltraSoC
(Markets and Markets, 2020) are representative examples of Siemen’s digital
twin product. Ansys has demonstrated applications of digital twins for Kärcher
(Advantage Magazine, 2021), ABB (Boscaglia et al., 2019), ENGIE (Digital
Engineering, 2021) and others. IBM has applied digital twins in different
sectors like life cycle management, healthcare, predictive maintenance,
manufacturing and more (IBM Newsletter, 2020, IBM Blog, 2019, 2020, IBM White
Paper, 2020).

###  4.2 Open-source tools and datasets related to digital twins

We have also identified publicly available, free-to-use tools and datasets
related to digital twins through our literature review. Tables 3 and 4 show,
respectively, the publicly available tools and datasets. The roles of these
tools and datasets in digital twins range from modeling, process mining,
control, data streaming, and digital twin projects. These open-source tools
and datasets could be helpful for production-level implementations of digital
twins in both academia and industry. In what follows, we describe a few
representative examples of open-source tools and datasets. Interested readers
are again invited to consult Tables 3 and 4 for an overview of the tools and
datasets we have identified.

Table 3: Publicly available tools related to digital twins (DS: digital system; P2V: physical to virtual; V2P: virtual to physical) Name (link) |  Organization |  Application(s) |  Role(s) in digital twins |  Brief description (paper ref)
---|---|---|---|---
TE Code (Ricker ) |  UW |  Manufacturing process control |  DS, P2V, V2P |  A simulation program for controlling chemical processes, known as the Tennessee Eastman Plant-wide Industrial Process Control Problem (Ricker and Lee (1995))
Chrono (Chrono ; Tasora et al. (2015)) |  UW-Madison, UNIPR |  Multiphysics simulation |  DS |  A modelling and simulation infrastructure for multiphysics simulations, including multibody simulation, finite element analysis, and fluid-solid interaction
ProM (Eindhoven ; Van Dongen et al. (2005), Verbeek et al. (2010)) |  TU/e |  Process Mining |  DS, P2V, V2P |  Business process mining software
PyMC3 (Team (a); Salvatier et al. (2016)) |  Collaborative |  Multiple |  P2V |  A Python package for Bayesian modeling and probabilistic machine learning
QUESO (Team (b); Prudencio and Schulz (2012)) |  UT-Austin |  Mechanical |  P2V |  A collection of algorithms and C++ classes that can be used for offline, probabilistic model updating
mFUSE (Team (2022);) |  LANL/UCSD Engineering Institute |  Mechanical, Civil, Structural |  P2V |  A set of MATLAB functions organized into modules according to the three primary stages of SHM
Digital Twin Open-Source Repository (Consortium ) |  Digital Twin Consortium |  Manufacturing, mechanical, healthcare, etc. |  DS, P2V, V2P |  Open-source code implementations, collaborative documents for guidance and training, open-source models, or other assets related to digital twins
Dito (Foundation ) |  Eclipse Foundation |  IoT middleware |  P2V, V2P |  IoT middleware, focusing on data modelling and IoT device connectivity
Unreal Engine (Epic Games ) |  Epic Games, Inc. |  Transportation, mixed reality, etc. |  DS, P2V |  Real-time 3D modeling software

##### Open-source software

  * •

Chrono: Chrono is an open-source multiphysics modeling and simulation
infrastructure capable of multibody dynamics simulation, finite element
analysis, and fluid-solid interaction. Its core is a C++ object-oriented
library called the Chrono::Engine, which can be embedded in third-party
applications, as part of a large digital twin project. As noted on the
PROJECTCHRONO web page, its multibody dynamics capability allows for vehicle
dynamics simulations where wheeled vehicles may operate on soft, deformable
soils. This effort started in 1998, and the current version, Chrono 7.0.1, was
released in November 2021. It is rare to see such a long-lasting open-source
effort, which has substantially pushed the boundary of open-source middleware
and fostered an ecosystem of multiphysics software tools.

  * •

Digital twin repositories: These open-source repositories are being managed by
a non-profit organization called Digital Twin Consortium as their “open
collaboration initiative”. The repositories are hosted on GitHub
(https://github.com/digitaltwinconsortium) and currently include several
industry-relevant projects focused on manufacturing digitatlization (e.g.,
ManufacturingDTDLOntologies and EcolCafe-Industrie-4.0). The open-source
nature of these code implementations, digital models, and training and
guidance documents will drive development, interoperability, and adoption of
digital twins in industry.

  * •

Commercial software tools: Commercially available software tools enabling the
realization of digital twins include (1) CAD software for geometric modeling,
such as CATIA, Autodesk AutoCAD, Siemens NX, PTC Creo, SolidWorks, to name a
few, (2) 3D manufacturing simulation software for production simulation and
layout design, such as DELMIA and Tecnomatix, and (3) modeling and data
analytics software for operations optimization and predictive maintenance,
such as MATLAB (the Simulink modeling environment and Predictive Maintenance
Toolbox), and (4) cloud-based IIoT platforms for predictive maintenance,
including Siemens Mindsphere, Azure IoT Edge, Google Cloud, and Azure IoT
Edge. Although not serving as digital twins on their own, these software tools
have created success stories in realizing the digital twin concept in industry
settings by working collaboratively with other enabling software tools. What
is now needed are frameworks and software platforms that cohesively integrate
modeling and simulation efforts of different forms and at various levels of
detail to create complete digital twins in various engineering applications
(Malik (2021), Malik and Brem (2021)). In this direction, some efforts have
been undertaken in the commercial design software domain. For example, ESTECO
modeFRONTIER is a commercially available design software tool that integrates
and automates multiple CAD/CAE tools to create digital twins for engineering
design. Another example is ….

##### Open-source data

  * •

Bosch Production Line Performance Dataset: this dataset was introduced in 2016
as part of a data challenge whose goal was to reduce quality control escapes.
Participants in the data challenge were to develop a model that could predict
which parts will fail quality control by using extensive measurements taken at
different locations along the assembly line. Large datasets like the one
provided by Bosch represent the first steps toward realizing digital twin
models in the production setting.

  * •

NASA Prognostics Data Repository: NASA’s Ames Research Center hosts 18 well-
documented datasets, collectively called the Prognostics Data Repository, on
their Prognostics Center of Excellence website. These datasets are mostly run-
to-failure time series data collected from an engineered system or component
starting at a healthy state and ending at a failed state. They were
contributed by universities (e.g., ETH Zurich, the FEMTO-ST Institute, UC
Berkeley, the University of Cincinnati, and Arizona State University),
companies (e.g., PARC and Sentient Corporation), and government agencies
(e.g., NASA Ames). This data repository intends to increase data availability
for prognostic algorithm development and validation. Among the 18 datasets is
(1) the first publicly available battery aging dataset Saha and Goebel (2007)
that has led to the production of the first few battery prognostics papers
Saha et al. (2008), Goebel et al. (2008), Saha et al. (2009) and (2) the first
two publicly available engine degradation datasets Saxena and Goebel (2008a,
b) that has led to the creation of numerous data-driven prognostics algorithms
in the literature Wang et al. (2008), Heimes (2008), Hu et al. (2012),
Moghaddass and Zuo (2014), Zhang et al. (2016), Zhao et al. (2017). This data
repository is actively being expanded, and its impact on the field of
predictive maintenance, which has been profound, will be increasingly
significant.

  * •

Data Challenges in PHM and SHM: In order to foster and encourage innovations
in the PHM and SHM communities, data challenges/competitions have been
organized in different PHM and SHM conferences. The released open-source
datasets after the competitions have been extensively used in the literature
as benchmark to test the performance of different diagnostics and prognostics
algorithms (i.e., P2V connection). For instance, the datasets from the PHM
Data Challenge of the PHM08 conference have been posted at NASA prognostics
data repository, and have become one of the first publicly available datasets
for failure prognostics. Since the data challenge in the PHM08 conference,
many similar data challenges for fault diagnostics and failure prognostics
have been organized by different PHM conferences including the annual PHM
conference (PHM, 2022), the European PHM conference (PHME, 2022), and the
Asia-Pacific PHM Conference (PHMAP, 2021). In the SHM community, the first
data compitition was organized in 2020 by the Asia-Pacific Network of Centers
for Research in Smart Structures Technology, Harbin Institute of Technology,
the University of Illinois at Urbana-Champaign (ICSHM, 2020). In this
competition, algorithms are compared in terms of accuracy for detecting
fatigue crack based on images, anomaly of bridge using acceleration data, and
damage of cables based on monitored cable tension data. Even though the
addressed fundamental fault diagnostics and failure prognostics problems are
similar, the data competition in the SHM community focuses more on fault
diagnostics, anomaly detection, and prognostics of large civil
infrastructures. The popularity of these data challenges shows the importance
of common datasets for the development of new algorithms in establishing the
P2V connection in digital twins.

  * •

Battery Archive: This web-based platform, launched in 2020, is the first
public platform that allows the battery community to visualize, analyze, and
compare battery aging data across different organizations, studies,
chemistries, and aging conditions. The site development is led by Sandia
National Laboratories (SNL), a U.S. Department of Energy (DOE) sponsored
Federally Funded Research and Development Center. The Battery Archive platform
is built based on an open-source platform called the Battery Life Cycle
Framework De Angelis et al. (2021) and uses an open-source extract-transform-
load engine called Redash. It has two unique properties: (1) all the battery
datasets shared on the platform share a standard format, making it easy to
compare performance across studies, battery chemistries, and aging conditions,
and (2) the platform has an open-source backend, opening the possibilities of
interoperating with existing software tools for more advanced analytics.
Compared to the NASA Prognostics Data Repository, Battery Archive goes one
step further by standardizing the data format and setting up open-source tools
for data visualization and basic analysis. This further step is expected to
have a long-last impact on future research studies surrounding battery
degradation modeling. Further, we think Battery Archive represents an
exemplary data-sharing effort that could be scaled up to benefit other digital
twin applications.

  * •

Sandia Thermal Challenge Problem: This thermal challenge problem was launched
by SNL in 2006 and has been used extensively as a benchmarking dataset to
evaluate the performance of methods and approaches for experimental validation
of computer simulation models and quantification of model uncertainty (Hills
et al., 2008). It is the first publicly available dataset to evaluate the
performance of different UQ methods for model calibration and validation. The
mathematical model of one-dimensional, linear heat conduction in a solid slab
and the associated experimental data are provided by Dowding et al. (2008).
The problem is concered about (1) evaluating whether the prediction accuracy
of a calibrated/validated model meets a regulatory requirement and (2)
quantifying the uncertainty in the accuracy assessment. In the past decades,
solutions to this challenge problem (model validation approaches and results)
have been developed using both Bayesian (e.g., the KOH framework discussed in
Sec. 2.1.2) and non-Bayesian methods. The problem is formulated in a way that
it is independent from specific applications and representative of the
complexities of realistic physical systems.

##### More collaborative, open-source efforts are needed.

The industry-scale adoption of digital twins will be accelerated by increasing
efforts in sharing tools (e.g., applications, libraries, and plug-ins), data,
and best practices (e.g., tutorials, implementation guides, and training
materials) between industry, academia, and government. Open-source efforts by
academic researchers are often directed to create rigorous, application-
agnostic solutions for specific technical components of digital twins (e.g.,
probabilistic model calibration, MPC). In contrast, open-source efforts by
industrial practitioners tend to be focused on integrating these technical
pieces into easy-to-use, well-documented software platforms thoughtfully
customized to individual applications. These two groups of open-source
efforts, representing two distinct ranges of technology readiness levels
(TRL), are complementary. The cross-sharing of knowledge among these
communities of interest will grow an open-source knowledge pool and promote
the development of high quality commercial-grade software for digital twins.
In this review, we want to call for the academic, industrial, and government
communities to work together and share their software tools and data. Such
collaborative, open-source efforts will go a long way in shaping the future of
the digital twin field.

Table 4: Publicly available datasets related to digital twin Name (link) |  Organization |  Application(s) |  Role(s) in digital twins |  Brief description (paper ref)
---|---|---|---|---
NASA Prognostics Data Repository (PCoE ) |  NASA |  Predictive maintenance |  P2V, V2P, OPT |  A collection of 18 prognostic datasets for prognostic algorithm development
Battery Archive (SNL ) |  DOE SNL |  Battery degradation modeling |  P2V, OPT |  A web-based platform for visualizing, analyzing, and comparing battery aging data across different studies (De Angelis et al. (2021))
Thermal Challenge Problem (Hills et al. (2008)) |  DOE SNL |  Model validation and UQ |  P2V, OPT |  A mathematical model and the solution of a one-dimensional, linear heat conduction in a solid slab. Experimental data related to the mathematical model are also provided (Dowding et al., 2008).
Kadi4Mat |  KIT |  Materials modeling |  P2V, OPT |  A web-based application facilitating easy and intuitive access, exchange, and integration of mostly battery materials data. https://demo-kadi4mat.iam-cms.kit.edu/
Bosch Production Line Performance Dataset |  Kaggle |  Production monitoring for quality control |  P2V, V2P, OPT |  https://www.kaggle.com/ competitions/bosch-production-line-performance/data
3D Data Hack Dublin (Graham ) |  Dublin City Council etc. |  3D modeling for urban planning, etc. |  DS, P3V, V2P |  3D datasets for creating a digital twin of the Docklands area in Dublin, Ireland White et al. (2021)

##  5 Perspectives on UQ and optimization for digital twins

###  5.1 Digital twins for structural life cycle management

The concept of digital twin for structural life cycle management is primarily
defined by its usage objectives, and the properties required to perform those
objectives. These digital representations or manifestations may be used for
several such goals including (but not limited to) visualization/curation,
structural state awareness quantification, and prediction of future structural
performance (e.g., when critical limit states are expected to occur).
Consequently, digital twins take on a variety of forms ranging from point
clouds graphical representations to machine learners to physics-based
simulators such as finite element models. Taken together, this collection of
capabilities empowers the possibility of structural life cycle emulation.
Initial designs (drawings, CAD, etc.) can be configured into baseline digital
realizations associated with nominal expected performance properties (e.g.,
material properties, connectivity); these realizations may be updated by
surveys (e.g., a UAV multi-mode lidar, thermal, visual data taken in the as-
built state), and/or deployed in-situ data streams that come from a variety of
IoT sources. Diagnostic analytic capability can be added by a variety of
supervised or unsupervised learning techniques and physical knowledge of
degradation or system state evolution. Stochastic load/demand and
environmental models can be added to evolve the structure through a variety of
operational scenarios, with uncertainty quantified by any of a variety of
techniques, to predict future performance variables all the way to the limit
states, thus completing the life cycle. This “cradle-to-grave” capability can
be further integrated with utility/cost models so that the consequences of all
future states (or decisions made because of these states) may be inferred,
leading to the ability to optimize the structure’s life cycle management,
including preventative maintenance, repair, etc. The digital twin would thus
become the primary tool of the structural owner to manage the structure,
leading to “smart structure”-informed decision-making that could potentially
mitigate total structural asset management risk. This vision becomes more and
more realizable as large-scale computing, data and predictive analytics, and
data management continually empower its execution.

###  5.2 Digital twins for sustainability

As discussed in Sec. 1 of our Part 1 paper (Thelen et al., 2022), a physical
system’s life cycle can be divided into four distinct phases: development,
manufacturing, service, and disposal. Most of the digital twin papers that we
reviewed cover one or multiple of the first three life cycle phases. We did
not find many studies that look at the disposal phase, where a physical system
or product which has reached its end of life in the first life cycle could be
reused, remanufactured, or recycled. Let us now look at two representative
examples where digital twins have the potential to make an enormous and long-
lasting impact at the disposal phase: (1) _remanufacturing_ in the equipment
manufacturing sector and (2) _battery repurposing_ in the energy sector.

Remanufacturing is a process where used products/subassemblies are returned to
an “as-new” condition (Ijomah et al., 2004) or even a “better-than-new”
condition (Zhao et al., 2010) with significantly lower energy and materials
consumption than original manufacturing (Li et al., 2021a). Remanufacturing
has been gaining popularity in recent years, as it plays an important role in
driving the global transition to a circular economy. Examples of physical
systems that can be remanufactured when approaching the end of design life
include a piece of industrial equipment on a production floor (Zacharaki et
al., 2021) and a high-value component of an agricultural machine in the field
(Li et al., 2021a). More research is needed to exploit the digital twin
concept to promote wide-scale adoption of remanufacturing in the manufacturing
industry. An interesting research topic along this line is the development of
ML methods for non-destructive damage assessment and RUL prediction of cores
(used products/subassemblies returned for remanufacturing) on a
remanufacturing floor. Although fault diagnostics and failure prognostics have
found successes in industrial, structural, and energy applications, as
discussed in Sec. 4.4 of our Part 1 paper, such success stories are rare in
remanufacturing mainly due to difficulties in accessing historical data from
returned products/subassemblies and in collecting faulty/run-to-failure data
representative of returned products/subassemblies. Physics-informed ML
approaches discussed in Sec. 3.4 of Part 1 can certainly be used to tackle
these data challenges. A recent attempt was made to quantify and forecast
accumulated fatigue damage in recycled materials (Hsu, 2021). This attempt can
be classified as the ML-assisted approach (Approach 6) shown in Fig. 10 of
Part 1. It is anticipated that remanufacturing will continue to grow in the
manufacturing industry. We call for industry-relevant studies that investigate
how to harness the power of digital twins and their building blocks, such as
ML, DL, and IIoT, to build rapid diagnostic, prognostic, and decision support
tools. Making these tools available to remanufacturers will lead to
opportunities to reduce the cost of remanufacturing and increase the core
reuse rate, eventually making remanufacturing more cost-competitive and
sustainable.

The other example is repurposing used electric vehicle batteries for
stationary storage applications (Zhu et al., 2021). To date, battery
repurposing has been mostly for small-scale commercial applications, such as
portable battery packs for camping and backpacking, waste disposal bins that
use batteries to store solar energy, and off-grid lighting, as reported in Zhu
et al. (2021), and small-scale recreational applications including
recreational vehicles and golf cars. Building on the digital twin concept, two
notable efforts are being pursued to support battery repurposing for larger-
scale commercial applications.

  * •

Rapid diagnostics: Similar to remanufacturing, battery repurposing typically
requires rapid testing/grading methods to estimate the state of health of used
electric vehicle batteries and predict their RUL. Estimating the state of
health at the end of the first life and predicting the RUL over the second
life is essential to determining the economic and technical viability of
repurposing any used battery pack/module for a specific second-life
application. Now we can look back at the proposed five-dimensional digital
twin model in Fig. 3 of the Part 1 paper and draw connections. The sensor data
(P2V or physical-to-virtual) would be voltage and current measurements from a
used battery pack/module (physical system or PS) during a rapid physical test.
The digital state would be the instantaneous state of health or RUL. This
digital state can be the output of an ML model (digital system or DS) built on
a training dataset (P2V, physical-to-virtual) to estimate either the
probabilities of predefined health grades (classification) or the probability
distribution of the state of health or RUL (regression). The resulting
estimate of the digital state (state of health or RUL) will enable rapid, near
real-time assessment of the economic value (e.g., expected revenue from a
simulated second-life scenario vs. associated costs) and technical feasibility
(e.g., expected remaining lifetime vs. desirable lifetime, and safety risks)
of repurposing the battery pack/module. The economic and technical assessment
will provide the battery repurposing company with actionable information on
optimally selecting (1) used battery modules with similar state of health
levels to create a repurposed pack and (2) the second-life application that a
repurposed pack has the best fit for (V2P or virtual-to-physical). Rapid
diagnostics for battery repurposing is an exciting application of digital
twins. It is probably the right time to invest serious and dedicated research
efforts in building easy-to-use and accurate tools for rapid battery
diagnostics. Such efforts could start with collecting and sharing performance
and degradation data from second-life applications, which we had difficulties
finding when populating the open-source datasets in Sec. 4.2.

  * •

Battery Passport: An emerging and attractive alternative to rapid physical
testing in a battery repurposing plant is the concept of Battery Passport,
initiated by a public-private collaboration platform called the Global Battery
Alliance (Alliance, 2020) in November 2020 and discussed extensively in the
battery modeling literature (Bai et al., 2020, Zhu et al., 2021, Ayerbe et
al., 2021). The Battery Passport can be viewed as a digital twin of a physical
battery that comprises all relevant information about the battery from cradle
(resource extraction and original manufacturing) to gate (repurposing or
recycling). It gives each battery a unique identity consisting of information
stored in the cloud or on a digital chip, for example, as part of a battery
management system that manages the power and health of the battery. An
achievable outcome of the Battery Passport is three-fold: (1) collecting state
of health estimates and use conditions of a battery over its first life, (2)
forecasting the battery’s state of health by comparing the collected first-
life information with the expected aging patterns for this battery chemistry,
and (3) predicting the cycle life over the second life based on the state of
health forecasting results. An obvious benefit would be the possibility of
eliminating physical testing in a battery repurposing facility, substantially
lowering the overall cost of repurposing. We believe that the value of digital
twins to battery repurposing is far greater than what has been realized. As
the Battery Passport continues to be adopted globally and across the entire
battery value chain, we expect to see increasing adoption of digital twins to
drive industry-scale battery repurposing that is both profitable and
environmentally responsible.

###  5.3 UQ of digital twins

As discussed in earlier sections, the methodologies within UQ can be grouped
within several broad categories: input uncertainty quantification, model
calibration (parameter and discrepancy estimation), model verification
(numerical error quantification), model validation, and uncertainty
aggregation towards prediction uncertainty quantification. Several methods are
available for each category, and an end-to-end UQ approach can be implemented
for static models. On the other hand, a digital twin, which is an evolving
model being updated when new data becomes available, presents new challenges,
especially in model validation and decision making. The model updated at one
time instant can only be validated with future data; thus validation is a
lagging indicator of digital twin model quality. However, decisions have to be
made with the currently updated model, whose validation is pending based on
future data.

Validation of a digital twin is not the same as validation of a general
purpose prediction model. In the case of the latter, validation experiments
can be performed to test the model’s performance. This is not possible in the
case of a digital twin, since the digital twin is tailored to a specific
specimen or realization of a physical system, not to a population of specimens
or realizations. Therefore validation data also has to be obtained from future
use of a particular specimen; a specimen used in laboratory validation
experiment is not the particular specimen for which the digital twin has been
constructed. This is why validation is a lagging indicator of a digital twin’s
quality.

Since online decision-making for a future time instant is based on a current
model which can only be validated in the future, there is additional
uncertainty due to lack of validation at the current time instant. How to
quantify this uncertainty is a new challenge. One option is to quantify the
validation performance and the corresponding model uncertainty in previous
time instants and extrapolate to future time instants; however, this raises
another question regarding how to quantify the confidence in the
extrapolation. In general, the uncertainty of the digital twin is expected to
decrease with time, but this is only true when the conditions of the system
operation remain similar. When aging (degrading) systems, such as aircraft and
civil infrastructure, are used beyond their certified design life or operated
outside the envelope of design conditions, newer methods are needed to
quantify the uncertainty in the system behavior and its prediction by the
digital twin.

##  6 Conclusion

In Part 1, we reviewed key digital twin modeling methods which connect the
virtual and physical worlds. However, in practice, measurements of the
physical world (P2V) and virtual model outputs (V2P) are never certain. In
this paper, we reviewed different methods to quantify the uncertainty of
machine learning models and the many digital twin modeling methods discussed
in Part 1. Additionally, we discussed various optimization techniques used to
improve accuracy, reduce uncertainty, and improve the usefulness of a digital
twin model.

In addition to a comprehensive review of digital twin technology, we
implemented and presented a battery digital twin model used for determining
the optimal time to retire a battery cell from its first life application. The
proposed battery digital twin serves as an example of how degradation modeling
and optimization can be combined to produce a digital twin model which
provides practitioners with actionable information regarding a cell’s status
in its overall life cycle.

The idea of a true digital twin which can accurately mirror all aspects of a
physical asset is rapidly becoming a possibility thanks to the many state-of-
the-art modeling techniques discussed in this two part review. In the future,
digital twin modeling stands to revolutionize the way manufacturers and end
users deliver, maintain, and retire high-value assets. Industry 4.0 is upon
us, and big data for digital twin modeling is leading the way.

## Acknowledgements

Adam Thelen and Chao Hu would like to thank the financial support from the
U.S. National Science Foundation under Grant No. ECCS-2015710. Xiaoge Zhang is
supported by a grant from the Research Committee of The Hong Kong Polytechnic
University under project code 1-BE6V. Sankaran Mahadevan acknowledges the
support of the National Institute of Science and Technology. Michael D. Todd
and Zhen Hu received financial support from the U.S. Army Corps of Engineers
through the U.S. Army Engineer Research and Development Center Research
Cooperative Agreement W912HZ-17-2-0024.

## Competing Interests

The authors have no relevant financial or non-financial conflicts of interest
to disclose.

## Replication of results

The Python code and preprocessed dataset used for the battery case study are
available for download on Github .

## Authors’ contributions

All authors read and approved the final manuscript. Hu, C. and Hu, Z. devised
the original concept of the review paper. Hu, Z, Thelen, A., and Zhang, X.
were responsible for the literature review. Thelen, A. was responsible for
geometric modeling. Hu, C., Thelen, A., and Hu, Z. were responsible for
physics-based modeling. Hu, Z. was responsible for data-driven modeling. Hu,
C. and Hu, Z. were responsible for physics-informed ML. Zhang, X. was
responsible for system modeling. Hu, C., and Hu, Z., were responsible for
probabilistic model updating. Zhang, X. was responsible for ML model updating.
Hu, C. and Hu., Z were responsible for fault diagnostics, failure prognostics,
and predictive maintenance. Lu, Y. was responsible for MPC. Fink, O. was
responsible for federated learning and domain adaptation. Zhang, X., Hu, Z.,
and Fink, O. were responsible for deep reinforcement learning. Hu, C. was
responsible for UQ of ML models. Hu, Z. was responsible for UQ of dynamic
system models, optimization for sensor placement, and optimization for
physical system modeling. Lu, Y. was responsible for the optimization of
additive manufacturing processes. Zhang, X. and Hu, Z. were responsible for
real-time mission planning. Thelen, A. and Hu, C. were responsible for the
case study and predictive maintenance scheduling. Hu, C. was responsible for
open-source software and data. Ghosh, S. was responsible for the industry
demonstration. Hu, C., Todd, M., and Mahadevan, S. were responsible for
perspectives. All authors participated in manuscript writing, review, editing,
and comment.

## References

  * Abdar et al. (2021) M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, et al.  A review of uncertainty quantification in deep learning: Techniques, applications and challenges.  _Information Fusion_ , 76:243–297, 2021.
  * Adnan et al. (2013) M. A. Adnan, M. A. Razzaque, I. Ahmed, and I. F. Isnin.  Bio-mimic optimization strategies in wireless sensor networks: A survey.  _Sensors_ , 14(1):299–345, 2013.
  * Advantage Magazine (2021) A. Advantage Magazine.  Kärcher cleans up with ansys twin builder, Feb 2021.  URL https://www.ansys.com/advantage-magazine/volume-xv-issue-2-2021/karcher-cleans-up-with-ansys-twin-builder.
  * Allah Bukhsh et al. (2019) Z. Allah Bukhsh, I. Stipanovic, G. Klanker, A. O’Connor, and A. G. Doree.  Network level bridges maintenance planning using multi-attribute utility theory.  _Structure and Infrastructure Engineering_ , 15(7):872–885, 2019.
  * Allemang (2003) R. J. Allemang.  The modal assurance criterion–twenty years of use and abuse.  _Sound and Vibration_ , 37(8):14–23, 2003.
  * Alliance (2020) G. B. Alliance.  The global battery alliance battery passport: giving an identity to the ev’s most important component.  _Glob. Batter. Alliance_ , 2020.
  * Alsheikh et al. (2015) M. A. Alsheikh, D. T. Hoang, D. Niyato, H.-P. Tan, and S. Lin.  Markov decision processes with applications in wireless sensor networks: A survey.  _IEEE Communications Surveys & Tutorials_, 17(3):1239–1267, 2015.
  * An et al. (2022a) H. An, B. D. Youn, and H. S. Kim.  A methodology for sensor number and placement optimization for vibration-based damage detection of composite structures under model uncertainty.  _Composite Structures_ , 279:114863, 2022a.
  * An et al. (2022b) H. An, B. D. Youn, and H. S. Kim.  Optimal sensor placement considering both sensor faults under uncertainty and sensor clustering for vibration-based damage detection.  _Structural and Multidisciplinary Optimization_ , 65(3):1–32, 2022b.
  * Anand et al. (2005) M. Anand, Z. Ives, and I. Lee.  Quantifying eavesdropping vulnerability in sensor networks.  In _Proceedings of the 2nd International Workshop on Data Management for Sensor Networks_ , pages 3–9, 2005.
  * Andrieu et al. (2003) C. Andrieu, N. De Freitas, A. Doucet, and M. I. Jordan.  An introduction to mcmc for machine learning.  _Machine Learning_ , 50(1):5–43, 2003.
  * Ao et al. (2017a) D. Ao, Z. Hu, and S. Mahadevan.  Design of validation experiments for life prediction models.  _Reliability Engineering & System Safety_, 165:22–33, 2017a.
  * Ao et al. (2017b) D. Ao, Z. Hu, and S. Mahadevan.  Dynamics model validation using time-domain metrics.  _Journal of Verification, Validation and Uncertainty Quantification_ , 2(1), 2017b.
  * ARC (2022) ARC.  Automotive research center at the university of michigan.  https://arc.engin.umich.edu/, 2022.  Last Accessed: 2022-05-06.
  * Arendt et al. (2012) P. D. Arendt, D. W. Apley, and W. Chen.  Quantification of model uncertainty: Calibration, model discrepancy, and identifiability.  2012\.
  * Asorey-Cacheda et al. (2017) R. Asorey-Cacheda, A.-J. Garcia-Sanchez, F. García-Sánchez, and J. García-Haro.  A survey on non-linear optimization problems in wireless sensor networks.  _Journal of Network and Computer Applications_ , 82:1–20, 2017.
  * Astroza et al. (2019) R. Astroza, A. Alessandri, and J. P. Conte.  A dual adaptive filtering approach for nonlinear finite element model updating accounting for modeling uncertainty.  _Mechanical Systems and Signal Processing_ , 115:782–800, 2019.
  * Attia et al. (2020) P. M. Attia, W. C. Chueh, and S. J. Harris.  Revisiting the t0. 5 dependence of sei growth.  _Journal of the Electrochemical Society_ , 167(9):090535, 2020.
  * Augustine (2020) P. Augustine.  The industry use cases for the digital twin idea.  In _Advances in Computers_ , volume 117, pages 79–105. Elsevier, 2020\.
  * Aydemir et al. (2020) H. Aydemir, U. Zengin, and U. Durak.  The digital twin paradigm for aircraft review and outlook.  In _AIAA Scitech 2020 Forum_ , page 0553, 2020.
  * Ayerbe et al. (2021) E. Ayerbe, M. Berecibar, S. Clark, A. A. Franco, and J. Ruhland.  Digitalization of battery manufacturing: Current status, challenges, and opportunities.  _Advanced Energy Materials_ , page 2102696, 2021.
  * Bai et al. (2020) Y. Bai, N. Muralidharan, Y.-K. Sun, S. Passerini, M. S. Whittingham, and I. Belharouak.  Energy and environmental aspects in recycling lithium-ion batteries: Concept of battery identity global passport.  _Materials Today_ , 41:304–315, 2020.
  * Bao and Wang (2015) N. Bao and C. Wang.  A monte carlo simulation based inverse propagation method for stochastic model updating.  _Mechanical Systems and Signal Processing_ , 60:928–944, 2015.
  * Barbehenn (1998) M. Barbehenn.  A note on the complexity of dijkstra’s algorithm for graphs with weighted vertices.  _IEEE Transactions on Computers_ , 47(2):263, 1998\.
  * Barzegar et al. (2022) V. Barzegar, S. Laflamme, C. Hu, and J. Dodson.  Ensemble of recurrent neural networks with long short-term memory cells for high-rate structural health monitoring.  _Mechanical Systems and Signal Processing_ , 164:108201, 2022.
  * Basagni et al. (2014) S. Basagni, L. Bölöni, P. Gjanci, C. Petrioli, C. A. Phillips, and D. Turgut.  Maximizing the value of sensed information in underwater wireless sensor networks via an autonomous underwater vehicle.  In _IEEE INFOCOM 2014-IEEE Conference on Computer Communications_ , pages 988–996. IEEE, 2014.
  * Beck and Katafygiotis (1998) J. L. Beck and L. S. Katafygiotis.  Updating models and their uncertainties. i: Bayesian statistical framework.  _Journal of Engineering Mechanics_ , 124(4):455–461, 1998.
  * Behmanesh et al. (2017) I. Behmanesh, B. Moaveni, and C. Papadimitriou.  Probabilistic damage identification of a designed 9-story building using modal data in the presence of modeling errors.  _Engineering Structures_ , 131:542–552, 2017.
  * Bellamy III (2018) W. Bellamy III.  Boeing ceo talks ’digital twin’ era of aviation.  https://www.aviationtoday.com/2018/09/14/boeing-ceo-talks-digital-twin-era-aviation/, Sep 2018.
  * Bing et al. (2000) L. Bing, Z. Meilin, and X. Kai.  A practical engineering method for fuzzy reliability analysis of mechanical structures.  _Reliability Engineering & System Safety_, 67(3):311–315, 2000.
  * Birkl et al. (2017) C. R. Birkl, M. R. Roberts, E. McTurk, P. G. Bruce, and D. A. Howey.  Degradation diagnostics for lithium ion cells.  _Journal of Power Sources_ , 341:373–386, 2017.
  * Bisdikian et al. (2013) C. Bisdikian, L. M. Kaplan, and M. B. Srivastava.  On the quality and value of information in sensor networks.  _ACM Transactions on Sensor Networks (TOSN)_ , 9(4):1–26, 2013.
  * Błachowski et al. (2020) B. Błachowski, A. Świercz, M. Ostrowski, P. Tauzowski, P. Olaszek, and Ł. Jankowski.  Convex relaxation for efficient sensor layout optimization in large-scale structures subjected to moving loads.  _Computer-Aided Civil and Infrastructure Engineering_ , 35(10):1085–1100, 2020.
  * Boers and Driessen (2003) Y. Boers and J. N. Driessen.  Interacting multiple model particle filter.  _IEE Proceedings-Radar, Sonar and Navigation_ , 150(5):344–349, 2003.
  * Boscaglia et al. (2019) L. Boscaglia, F. Bonsanto, A. Boglietti, S. Nategh, and C. Scema.  Conjugate heat transfer and cfd modeling of self-ventilated traction motors.  In _2019 IEEE Energy Conversion Congress and Exposition (ECCE)_ , pages 3103–3109. IEEE, 2019.
  * Bousdekis et al. (2019) A. Bousdekis, K. Lepenioti, D. Apostolou, and G. Mentzas.  Decision making in predictive maintenance: literature review and research agenda for industry 4.0.  _IFAC-PapersOnLine_ , 52(13):607–612, 2019.
  * Bruynseels et al. (2018) K. Bruynseels, F. Santoni de Sio, and J. Van den Hoven.  Digital twins in health care: ethical implications of an emerging engineering paradigm.  _Frontiers in Genetics_ , page 31, 2018.
  * Bukhsh et al. (2020) Z. A. Bukhsh, I. Stipanovic, and A. G. Doree.  Multi-year maintenance planning framework using multi-attribute utility theory and genetic algorithms.  _European Transport Research Review_ , 12(1):1–13, 2020.
  * Burns et al. (2014) J. A. Burns, E. M. Cliff, and K. Farlow.  Parameter estimation and model discrepancy in control systems with delays.  _IFAC Proceedings Volumes_ , 47(3):11679–11684, 2014.
  * Burns et al. (2018) J. A. Burns, E. M. Cliff, and T. L. Herdman.  Identification of dynamical systems with structured uncertainty.  _Inverse Problems in Science and Engineering_ , 26(2):280–321, 2018.
  * Camci (2009) F. Camci.  System maintenance scheduling with prognostics information using genetic algorithm.  _IEEE Transactions on reliability_ , 58(3):539–552, 2009.
  * Camci (2015) F. Camci.  Maintenance scheduling of geographically distributed assets with prognostics information.  _European Journal of Operational Research_ , 245(2):506–516, 2015.
  * Cantero-Chinchilla et al. (2020) S. Cantero-Chinchilla, J. Chiachío, M. Chiachío, D. Chronopoulos, and A. Jones.  Optimal sensor configuration for ultrasonic guided-wave inspection based on value of information.  _Mechanical Systems and Signal Processing_ , 135:106377, 2020.
  * Careless (2021) J. Careless.  Digital twinning: The latest on virtual models.  https://www.aerospacetechreview.com/digital-twinning-the-latest-on-virtual-models/, Aug 2021.
  * Carne and Dohrmann (1994) T. G. Carne and C. R. Dohrmann.  A modal test design strategy for model correlation.  Technical report, Sandia National Labs., Albuquerque, NM (United States), 1994.
  * Casals et al. (2019) L. C. Casals, B. A. García, and C. Canal.  Second life batteries lifespan: Rest of useful life and environmental analysis.  _Journal of environmental management_ , 232:354–363, 2019\.
  * Caulfield (2022) B. Caulfield.  Nvidia, bmw blend reality, virtual worlds to demonstrate factory of the future.  https://blogs.nvidia.com/blog/2021/04/13/nvidia-bmw-factory-future/, Apr 2022.
  * Cha (2007) S.-H. Cha.  Comprehensive survey on distance/similarity measures between probability density functions.  _City_ , 1(2):1, 2007.
  * Chadha et al. (2021) M. Chadha, Z. Hu, and M. D. Todd.  An alternative quantification of the value of information in structural health monitoring.  _Structural Health Monitoring_ , page 14759217211028439, 2021.
  * Chen et al. (1991) G.-S. Chen, R. J. Bruno, and M. Salama.  Optimal placement of active/passive members in truss structures using simulated annealing.  _AIAA Journal_ , 29(8):1327–1334, 1991.
  * Chen et al. (2018) X. Chen, E. Kang, S. Shiraishi, V. M. Preciado, and Z. Jiang.  Digital behavioral twins for safe connected cars.  In _Proceedings of the 21th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems_ , pages 144–153, 2018.
  * (52) P. Chrono.  Chrono.  https://projectchrono.org/.  Last Accessed: 2022-04-17.
  * (53) D. T. Consortium.  Digital twin open-source repository.  https://www.digitaltwinconsortium.org/initiatives/open-source.htm/.  Last accessed on 2022-04-17.
  * De Angelis et al. (2021) V. De Angelis, Y. Preger, and B. R. Chalamala.  Battery lifecycle framework: a flexible repository and visualization tool for battery data from materials development to field implementation.  2021\.
  * Der Kiureghian and Ditlevsen (2009) A. Der Kiureghian and O. Ditlevsen.  Aleatory or epistemic? does it matter?  _Structural safety_ , 31(2):105–112, 2009.
  * Diao et al. (2019) W. Diao, S. Saxena, and M. Pecht.  Accelerated cycle life testing and capacity degradation modeling of licoo2-graphite cells.  _Journal of Power Sources_ , 435:226830, 2019.
  * Digital Engineering (2021) M. Digital Engineering.  Engie lab crigen and ansys accelerate zero carbon energy, Mar 2021.  URL https://www.digitalengineering247.com/article/engie-lab-crigen-and-ansys-accelerate-zero-carbon-energy/.
  * Dodson et al. (2022) J. Dodson, A. Downey, S. Laflamme, M. D. Todd, A. G. Moura, Y. Wang, Z. Mao, P. Avitabile, and E. Blasch.  High-rate structural health monitoring and prognostics: An overview.  _Data Science in Engineering, Volume 9_ , pages 213–217, 2022.
  * Dowding et al. (2008) K. J. Dowding, M. Pilch, and R. G. Hills.  Formulation of the thermal problem.  _Computer Methods in Applied Mechanics and Engineering_ , 197(29-32):2385–2389, 2008.
  * Downey et al. (2018) A. Downey, C. Hu, and S. Laflamme.  Optimal sensor placement within a hybrid dense sensor network using an adaptive genetic algorithm with learning gene pool.  _Structural Health Monitoring_ , 17(3):450–460, 2018.
  * Duchoň et al. (2014) F. Duchoň, A. Babinec, M. Kajan, P. Beňo, M. Florek, T. Fico, and L. Jurišica.  Path planning with modified a star algorithm for a mobile robot.  _Procedia Engineering_ , 96:59–69, 2014.
  * Durão et al. (2018) L. F. Durão, S. Haag, R. Anderl, K. Schützer, and E. Zancul.  Digital twin requirements in the context of industry 4.0.  In _IFIP international conference on product lifecycle management_ , pages 204–214. Springer, 2018.
  * Ehsani and Afshar (2010) N. Ehsani and A. Afshar.  Optimization of contaminant sensor placement in water distribution networks: multi-objective approach.  In _Water Distribution Systems Analysis 2010_ , pages 338–346. 2010\.
  * (64) Eindhoven.  Prom tools, eindhoven university of technology.  http://www.promtools.org/doku.php/.  Last accessed on 2022-04-17.
  * Engel et al. (2019) H. Engel, P. Hertzke, and G. Siccardo.  Second-life ev batteries: The newest value pool in energy storage.  _McKinsey & Company_, 2019.
  * Engel and Wellman (2010) Y. Engel and M. P. Wellman.  Multiattribute auctions based on generalized additive independence.  _Journal of Artificial Intelligence Research_ , 37:479–525, 2010.
  * (67) I. Epic Games.  Unreal engine 5.  http://www.unrealengine.com/en-US/unreal-engine-5.  Last Accessed: 2022-05-27.
  * Eshghi et al. (2019) A. T. Eshghi, S. Lee, H. Jung, and P. Wang.  Design of structural monitoring sensor network using surrogate modeling of stochastic sensor signal.  _Mechanical Systems and Signal Processing_ , 133:106280, 2019.
  * Feng et al. (2018) X. Feng, J. Gu, and X. Guan.  Optimal allocation of hybrid energy storage for microgrids based on multi-attribute utility theory.  _Journal of Modern Power Systems and Clean Energy_ , 6(1):107–117, 2018.  10.1007/s40565-017-0310-3.
  * Ferson et al. (2008) S. Ferson, W. L. Oberkampf, and L. Ginzburg.  Model validation and predictive capability for the thermal challenge problem.  _Computer Methods in Applied Mechanics and Engineering_ , 197(29-32):2408–2430, 2008.
  * Flynn and Todd (2010) E. B. Flynn and M. D. Todd.  A bayesian approach to optimal sensor placement for structural health monitoring with application to active sensing.  _Mechanical Systems and Signal Processing_ , 24(4):891–903, 2010.
  * (72) E. Foundation.  Eclipse ditto: open-source framework for digital twins in the iot.  https://www.eclipse.org/ditto/.  Last accessed on 2022-04-23.
  * Frazier (2014) W. Frazier.  Metal additive manufacturing: A review.  _Journal of Materials Engineering and Performance_ , 23:1917–1928, 2014.
  * Froger et al. (2016) A. Froger, M. Gendreau, J. E. Mendoza, E. Pinson, and L.-M. Rousseau.  Maintenance scheduling in the electricity industry: A literature review.  _European Journal of Operational Research_ , 251(3):695–706, 2016.
  * Gal and Ghahramani (2015) Y. Gal and Z. Ghahramani.  Bayesian convolutional neural networks with bernoulli approximate variational inference.  _arXiv preprint arXiv:1506.02158_ , 2015.
  * Gal and Ghahramani (2016a) Y. Gal and Z. Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _International Conference on Machine Learning_ , pages 1050–1059. PMLR, 2016a.
  * Gal and Ghahramani (2016b) Y. Gal and Z. Ghahramani.  A theoretically grounded application of dropout in recurrent neural networks.  _Advances in neural information processing systems_ , 29, 2016b.
  * Gammell et al. (2014) J. D. Gammell, S. S. Srinivasa, and T. D. Barfoot.  Informed rrt*: Optimal sampling-based path planning focused via direct sampling of an admissible ellipsoidal heuristic.  In _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 2997–3004. IEEE, 2014.
  * Gao et al. (2010) W. Gao, C. Song, and F. Tin-Loi.  Probabilistic interval analysis for structures with uncertainty.  _Structural Safety_ , 32(3):191–199, 2010.
  * Garmabaki et al. (2016) A. Garmabaki, A. Ahmadi, and M. Ahmadi.  Maintenance optimization using multi-attribute utility theory.  In _Current trends in reliability, availability, maintainability and safety_ , pages 13–25. Springer, 2016.
  * Gawlikowski et al. (2021) J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscher, et al.  A survey of uncertainty in deep neural networks.  _arXiv preprint arXiv:2107.03342_ , 2021.
  * (82) Github.  Link to github repository where the preprocessed data and python scripts used to generate all the results and figures in the case study section reside.  https://github.com/acthelen/battery_digital_twin.
  * Glaessgen and Stargel (2012) E. Glaessgen and D. Stargel.  The digital twin paradigm for future nasa and us air force vehicles.  In _53rd AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics and materials conference 20th AIAA/ASME/AHS adaptive structures conference 14th AIAA_ , page 1818, 2012.
  * Goebel et al. (2008) K. Goebel, B. Saha, A. Saxena, J. R. Celaya, and J. P. Christophersen.  Prognostics in battery health management.  _IEEE instrumentation & measurement magazine_, 11(4):33–40, 2008.
  * Gomes et al. (2019) G. F. Gomes, F. A. de Almeida, P. da Silva Lopes Alexandrino, S. S. da Cunha, B. S. de Sousa, and A. C. Ancelotti.  A multiobjective sensor placement optimization for shm systems considering fisher information matrix and mode shape interpolation.  _Engineering with Computers_ , 35(2):519–535, 2019.
  * (86) N. Graham.  3d data model resources for dublin docklands sdz.  https://data.smartdublin.ie/dataset/3d-data-hack-dublin-resources/.  Last accessed on 2022-04-23.
  * Grall et al. (2002) A. Grall, L. Dieulle, C. Bérenguer, and M. Roussignol.  Continuous-time predictive-maintenance scheduling for a deteriorating system.  _IEEE Transactions on Reliability_ , 51(2):141–150, 2002.
  * Gupta et al. (2010) V. Gupta, M. Sharma, and N. Thakur.  Optimization criteria for optimal placement of piezoelectric sensors and actuators on a smart structure: a technical review.  _Journal of Intelligent Material Systems and Structures_ , 21(12):1227–1243, 2010.
  * Guratzsch and Mahadevan (2010) R. F. Guratzsch and S. Mahadevan.  Structural health monitoring sensor placement optimization under uncertainty.  _AIAA journal_ , 48(7):1281–1289, 2010.
  * He et al. (2011) W. He, N. Williard, M. Osterman, and M. Pecht.  Prognostics of lithium-ion batteries based on dempster–shafer theory and the bayesian monte carlo method.  _Journal of Power Sources_ , 196(23):10314–10321, 2011.
  * Heimes (2008) F. O. Heimes.  Recurrent neural networks for remaining useful life estimation.  In _2008 international conference on prognostics and health management_ , pages 1–6. IEEE, 2008.
  * Heydari et al. (2020) A. Heydari, M. Aghabozorgi, and M. Biguesh.  Optimal sensor placement for source localization based on rssd.  _Wireless Networks_ , 26(7):5151–5162, 2020.
  * Hills et al. (2008) R. Hills, K. Dowding, and L. Swiler.  Thermal challenge problem: summary.  _Computer Methods in Applied Mechanics and Engineering_ , 197(29-32):2490–2495, 2008.
  * Honkura et al. (2011) K. Honkura, K. Takahashi, and T. Horiba.  Capacity-fading prediction of lithium-ion batteries based on discharge curves analysis.  _Journal of power sources_ , 196(23):10141–10147, 2011.
  * Hsu (2021) M.-H. Hsu.  _Machine learning-based non-destructive evaluation of fatigue damage in metals_.  PhD thesis, 2021.
  * Hu et al. (2012) C. Hu, B. D. Youn, P. Wang, and J. T. Yoon.  Ensemble of data-driven prognostic algorithms for robust prediction of remaining useful life.  _Reliability Engineering & System Safety_, 103:120–135, 2012.
  * Hu et al. (2014) C. Hu, G. Jain, P. Tamirisa, and T. Gorka.  Method for estimating capacity and predicting remaining useful life of lithium-ion battery.  In _2014 International Conference on Prognostics and Health Management_ , pages 1–8. IEEE, 2014.
  * Hu et al. (2018) C. Hu, H. Ye, G. Jain, and C. Schmidt.  Remaining useful life assessment of lithium-ion batteries in implantable medical devices.  _Journal of Power Sources_ , 375:118–130, 2018.
  * Hu et al. (2020) X. Hu, L. Xu, X. Lin, and M. Pecht.  Battery lifetime prognostics.  _Joule_ , 4(2):310–346, 2020.
  * Hu and Mahadevan (2017) Z. Hu and S. Mahadevan.  Uncertainty quantification and management in additive manufacturing: current status, needs, and opportunities.  _The International Journal of Advanced Manufacturing Technology_ , 93(5):2855–2874, 2017.
  * Hu et al. (2017) Z. Hu, D. Ao, and S. Mahadevan.  Calibration experimental design considering field response and model uncertainty.  _Computer Methods in Applied Mechanics and Engineering_ , 318:92–119, 2017.
  * Hu et al. (2019) Z. Hu, C. Hu, Z. P. Mourelatos, and S. Mahadevan.  Model discrepancy quantification in simulation-based design of dynamical systems.  _Journal of Mechanical Design_ , 141(1), 2019.
  * Huan and Marzouk (2014) X. Huan and Y. Marzouk.  Gradient-based stochastic optimization methods in bayesian experimental design.  _International Journal for Uncertainty Quantification_ , 4(6), 2014.
  * Huan and Marzouk (2013) X. Huan and Y. M. Marzouk.  Simulation-based optimal bayesian experimental design for nonlinear systems.  _Journal of Computational Physics_ , 232(1):288–317, 2013.
  * Huber (1974) G. P. Huber.  Multi-attribute utility models: A review of field and field-like studies.  _Management science_ , 20(10):1393–1402, 1974\.
  * IBM Blog (2019) I. IBM Blog.  How to create a twin to improve your own performance, Sep 2019.  URL https://www.ibm.com/blogs/internet-of-things/iot-digital-twin-rotterdam/.
  * IBM Blog (2020) I. IBM Blog.  Profile: Ucsf health and maximo lead the way on smart medical buildings, Jul 2020.  URL https://www.ibm.com/blogs/internet-of-things/iot-ucsf-health-and-maximo-smart-medical-buildings/.
  * IBM Newsletter (2020) I. IBM Newsletter.  Siemens and ibm deliver service lifecycle management solution, Jul 2020\.  URL https://newsroom.ibm.com/2020-06-17-Siemens-and-IBM-Deliver-Service-Lifecycle-Management-Solution.
  * IBM White Paper (2020) I. IBM White Paper.  Digital twin technologies for high-performance manufacturing, Jan 2020\.  URL https://www.ibm.com/downloads/cas/KX8A3MWX.
  * ICSHM (2020) ICSHM.  International project competition for structural health monitoring.  http://www.schm.org.cn/#/IPC-SHM,2020, 2020.  Last accessed on 2022-05-24.
  * Ijomah et al. (2004) W. L. Ijomah, S. Childe, and C. McMahon.  Remanufacturing: a key strategy for sustainable development.  2004\.
  * ISA (2010) ISA.  Ansi/isa-95.00.01-2010 (iec 62264-1 mod) enterprise-control system integration - part 1: Models and terminology.  https://www.isa.org/products/ansi-isa-95-00-01-2010-iec-62264-1-mod-enterprise, 2010\.  Last Accessed: 2022-05-29.
  * Jiang et al. (2020) C. Jiang, Z. Hu, Y. Liu, Z. P. Mourelatos, D. Gorsich, and P. Jayakumar.  A sequential calibration and validation framework for model uncertainty quantification and reduction.  _Computer Methods in Applied Mechanics and Engineering_ , 368:113172, 2020.
  * Jiang et al. (2021) C. Jiang, Z. Hu, Z. P. Mourelatos, D. Gorsich, P. Jayakumar, Y. Fu, and M. Majcher.  R2-RRT*: reliability-based robust mission planning of off-road autonomous ground vehicle under uncertain terrain environment.  _IEEE Transactions on Automation Science and Engineering_ , 2021.
  * Jiang et al. (2022a) C. Jiang, Y. Liu, Z. P. Mourelatos, D. Gorsich, Y. Fu, and Z. Hu.  Efficient reliability-based mission planning of off-road autonomous ground vehicles using an outcrossing approach.  _Journal of Mechanical Design_ , 144(4), 2022a.
  * Jiang et al. (2022b) C. Jiang, M. A. Vega, M. K. Ramancha, M. D. Todd, J. P. Conte, M. Parno, and Z. Hu.  Bayesian calibration of multi-level model with unobservable distributed response and application to miter gates.  _Mechanical Systems and Signal Processing_ , 170:108852, 2022b.
  * Jiang et al. (2022c) C. Jiang, M. A. Vega, M. D. Todd, and Z. Hu.  Model correction and updating of a stochastic degradation model for failure prognostics of miter gates.  _Reliability Engineering & System Safety_, 218:108203, 2022c.
  * Jiang and Mahadevan (2009) X. Jiang and S. Mahadevan.  Bayesian inference method for model validation and confidence extrapolation.  _Journal of Applied Statistics_ , 36(6):659–677, 2009.
  * Johnson et al. (2015) J. B. Johnson, A. V. Kulchitsky, P. Duvoy, K. Iagnemma, C. Senatore, R. E. Arvidson, and J. Moore.  Discrete element method simulations of mars exploration rover wheel performance.  _Journal of Terramechanics_ , 62:31–40, 2015.
  * Kammer (1991) D. C. Kammer.  Sensor placement for on-orbit modal identification and correlation of large space structures.  _Journal of Guidance, Control, and Dynamics_ , 14(2):251–259, 1991.
  * Kapteyn et al. (2021) M. G. Kapteyn, J. V. Pretorius, and K. E. Willcox.  A probabilistic graphical model foundation for enabling predictive digital twins at scale.  _Nature Computational Science_ , 1(5):337–347, 2021.
  * Kaveh et al. (2022) A. Kaveh, A. Dadras Eslamlou, P. Rahmani, and P. Amirsoleimani.  Optimal sensor placement in large-scale dome trusses via q-learning-based water strider algorithm.  _Structural Control and Health Monitoring_ , page e2949, 2022.
  * Kendall and Gal (2017) A. Kendall and Y. Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  _Advances in neural information processing systems_ , 30, 2017.
  * Kennedy and O’Hagan (2001) M. C. Kennedy and A. O’Hagan.  Bayesian calibration of computer models.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 63(3):425–464, 2001.
  * Kim et al. (2018) T. Kim, B. D. Youn, and H. Oh.  Development of a stochastic effective independence (sefi) method for optimal sensor placement under uncertainty.  _Mechanical Systems and Signal Processing_ , 111:615–627, 2018.
  * Kim et al. (2020) W. Kim, H. Yoon, G. Lee, T. Kim, and B. D. Youn.  A new calibration metric that considers statistical correlation: Marginal probability and correlation residuals.  _Reliability Engineering & System Safety_, 195:106677, 2020.
  * Kuffner and LaValle (2000) J. J. Kuffner and S. M. LaValle.  Rrt-connect: An efficient approach to single-query path planning.  In _Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065)_ , volume 2, pages 995–1001. IEEE, 2000.
  * Kuleshov et al. (2018) V. Kuleshov, N. Fenner, and S. Ermon.  Accurate uncertainties for deep learning using calibrated regression.  In _International conference on machine learning_ , pages 2796–2804. PMLR, 2018.
  * Kulkarni and Venayagamoorthy (2010) R. V. Kulkarni and G. K. Venayagamoorthy.  Particle swarm optimization in wireless-sensor networks: A brief survey.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , 41(2):262–267, 2010.
  * Lakshminarayanan et al. (2017) B. Lakshminarayanan, A. Pritzel, and C. Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ , 30, 2017.
  * Lee et al. (2013a) J. Lee, E. Lapira, B. Bagheri, and H.-a. Kao.  Recent advances and trends in predictive manufacturing systems in big data environment.  _Manufacturing letters_ , 1(1):38–41, 2013a.
  * Lee et al. (2013b) J. Lee, E. Lapira, S. Yang, and A. Kao.  Predictive manufacturing system-trends of next-generation production systems.  _Ifac proceedings volumes_ , 46(7):150–156, 2013b.
  * Lehner and Dorffner (2020) H. Lehner and L. Dorffner.  Digital geotwin vienna: towards a digital twin city as geodata hub, 2020\.
  * Lei and Sandborn (2018) X. Lei and P. A. Sandborn.  Maintenance scheduling based on remaining useful life predictions for wind farms managed using power purchase agreements.  _Renewable Energy_ , 116:188–198, 2018.
  * Li et al. (2015) J. Li, X. Zhang, J. Xing, P. Wang, Q. Yang, and C. He.  Optimal sensor placement for long-span cable-stayed bridge using a novel particle swarm optimization algorithm.  _Journal of Civil Structural Health Monitoring_ , 5(5):677–685, 2015.
  * Li et al. (2021a) M. Li, V. P. Nemani, J. Liu, M. A. Lee, N. Ahmed, G. E. Kremer, and C. Hu.  Reliability-informed life cycle warranty cost and life cycle analysis of newly manufactured and remanufactured units.  _Journal of Mechanical Design_ , 143(11), 2021a.
  * Li et al. (2021b) S. Li, H. Fang, and B. Shi.  Remaining useful life estimation of lithium-ion battery based on interacting multiple model particle filter and support vector regression.  _Reliability Engineering & System Safety_, 210:107542, 2021b.
  * Li et al. (2014) W. Li, W. Chen, Z. Jiang, Z. Lu, and Y. Liu.  New validation metrics for models with multiple correlated responses.  _Reliability Engineering & System Safety_, 127:1–11, 2014\.
  * Li et al. (2016) Y. Li, S. Sui, and S. Tong.  Adaptive fuzzy control design for stochastic nonlinear switched systems with arbitrary switchings and unmodeled dynamics.  _IEEE transactions on cybernetics_ , 47(2):403–414, 2016.
  * Ling and Mahadevan (2013) Y. Ling and S. Mahadevan.  Quantitative model validation techniques: New insights.  _Reliability Engineering & System Safety_, 111:217–231, 2013.
  * Liu et al. (2020) J. Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax Weiss, and B. Lakshminarayanan.  Simple and principled uncertainty estimation with deterministic deep learning via distance awareness.  _Advances in Neural Information Processing Systems_ , 33:7498–7512, 2020.
  * Liu et al. (2008) W. Liu, W.-c. Gao, Y. Sun, and M.-j. Xu.  Optimal sensor placement for spatial lattice structure based on genetic algorithms.  _Journal of Sound and Vibration_ , 317(1-2):175–189, 2008.
  * Liu et al. (2021a) X. Liu, M. Gao, J. Zhao, X. Sun, Z. Li, Q. Li, L. Wang, J. Wang, and W. Zhuang.  Effects of charging protocols on the cycling performance for high-energy lithium-ion batteries using a graphite-siox composite anode and li-rich layered oxide cathode.  _Journal of Power Sources_ , 495:229793, 2021a.
  * Liu and Li (2002) Y. Liu and X.-Y. Li.  Decentralized robust adaptive control of nonlinear systems with unmodeled dynamics.  _IEEE Transactions on Automatic control_ , 47(5):848–856, 2002.
  * Liu et al. (2011) Y. Liu, W. Chen, P. Arendt, and H.-Z. Huang.  Toward a better understanding of model validation metrics.  _Journal of Mechanical Design_ , 133(7), 2011.
  * Liu et al. (2019) Y. Liu, L. Zhang, Y. Yang, L. Zhou, L. Ren, F. Wang, R. Liu, Z. Pang, and M. J. Deen.  A novel cloud-based framework for the elderly healthcare services using digital twin.  _IEEE access_ , 7:49088–49101, 2019.
  * Liu et al. (2021b) Y. Liu, C. Jiang, Z. P. Mourelatos, D. Gorsich, P. Jayakumar, Y. Fu, M. Majcher, and Z. Hu.  Simulation-based mission mobility reliability analysis of off-road ground vehicles.  _Journal of Mechanical Design_ , 143(3), 2021b.
  * Liu et al. (2021c) Y. Liu, C. Jiang, X. Zhang, Z. P. Mourelatos, D. Barthlow, D. Gorsich, A. Singh, and Z. Hu.  Reliability-based multi-vehicle path planning under uncertainty using a bio-inspired approach.  _Journal of Mechanical Design_ , pages 1–44, 2021c.
  * Long et al. (2013) Q. Long, M. Scavino, R. Tempone, and S. Wang.  Fast estimation of expected information gains for bayesian experimental designs based on laplace approximations.  _Computer Methods in Applied Mechanics and Engineering_ , 259:24–39, 2013.
  * Lu et al. (2020) Q. Lu, A. K. Parlikad, P. Woodall, G. Don Ranasinghe, X. Xie, Z. Liang, E. Konstantinou, J. Heaton, and J. Schooling.  Developing a digital twin at building and city levels: Case study of west cambridge campus.  _Journal of Management in Engineering_ , 36(3):05020004, 2020.
  * LU LG et al. (2013) H. LU LG, J. Li, et al.  A review on the key issues for lithium-ion battery management in electric vehicles.  _Journal of power sources_ , 226:272–288, 2013.
  * Lui et al. (2021) Y. H. Lui, M. Li, A. Downey, S. Shen, V. P. Nemani, H. Ye, C. VanElzen, G. Jain, S. Hu, S. Laflamme, et al.  Physics-based prognostics of implantable-grade lithium-ion battery for remaining useful life prediction.  _Journal of Power Sources_ , 485:229327, 2021.
  * Mahadevan et al. (2022) S. Mahadevan, P. Nath, and Z. Hu.  Uncertainty quantification for additive manufacturing process improvement: Recent advances.  _ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part B: Mechanical Engineering_ , 8(1):010801, 2022\.
  * Malik (2021) A. A. Malik.  Framework to model virtual factories: A digital twin view.  _arXiv preprint arXiv:2104.03034_ , 2021.
  * Malik and Brem (2021) A. A. Malik and A. Brem.  Digital twins for collaborative robots: A case study in human-robot interaction.  _Robotics and Computer-Integrated Manufacturing_ , 68:102092, 2021.
  * Malings and Pozzi (2016) C. Malings and M. Pozzi.  Value of information for spatially distributed systems: Application to sensor placement.  _Reliability Engineering & System Safety_, 154:219–233, 2016.
  * Malings et al. (2015) C. Malings, M. Pozzi, and I. Velibeyoglu.  Sensor placement optimization for structural health monitoring.  In _Proceedings of the 10th International Workshop on Structural Health Monitoring_ , 2015.
  * Mandelbaum and Weinshall (2017) A. Mandelbaum and D. Weinshall.  Distance-based confidence score for neural network classifiers.  _arXiv preprint arXiv:1709.09844_ , 2017.
  * Markets and Markets (2020) Markets and M. Markets.  Digital twin market, Sep 2020.  URL https://www.marketsandmarkets.com/Market-Reports/digital-twin-market-225269522.html.
  * Meo and Zumpano (2005) M. Meo and G. Zumpano.  On the optimal sensor placement techniques for a bridge structure.  _Engineering structures_ , 27(10):1488–1497, 2005\.
  * Miao et al. (2013) Q. Miao, L. Xie, H. Cui, W. Liang, and M. Pecht.  Remaining useful life prediction of lithium-ion battery with unscented particle filter technique.  _Microelectronics Reliability_ , 53(6):805–810, 2013.
  * Moghaddass and Zuo (2014) R. Moghaddass and M. J. Zuo.  An integrated framework for online diagnostic and prognostic health monitoring using a multistate deterioration process.  _Reliability Engineering & System Safety_, 124:92–104, 2014.
  * Mousazadeh (2013) H. Mousazadeh.  A technical review on navigation systems of agricultural autonomous off-road vehicles.  _Journal of Terramechanics_ , 50(3):211–232, 2013\.
  * Mukhoti et al. (2021) J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. Torr, and Y. Gal.  Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty.  _arXiv e-prints_ , pages arXiv–2102, 2021.
  * Nado et al. (2021) Z. Nado, N. Band, M. Collier, J. Djolonga, M. W. Dusenberry, S. Farquhar, Q. Feng, A. Filos, M. Havasi, R. Jenatton, et al.  Uncertainty baselines: Benchmarks for uncertainty & robustness in deep learning.  _arXiv preprint arXiv:2106.04015_ , 2021.
  * NASA (2008) NASA.  Standard for models and simulation-nasa technical standard.  _National Aeronautics and Space Administration_ , Washington(DC):Standard No.NASA–STD–7009, 2008.
  * Nath et al. (2017) P. Nath, Z. Hu, and S. Mahadevan.  Sensor placement for calibration of spatially varying model parameters.  _Journal of Computational Physics_ , 343:150–169, 2017\.
  * Nemani et al. (2021) V. P. Nemani, H. Lu, A. Thelen, C. Hu, and A. T. Zimmerman.  Ensembles of probabilistic lstm predictors and correctors for bearing prognostics using industrial standards.  _Neurocomputing_ , 2021.
  * Niculescu-Mizil and Caruana (2005) A. Niculescu-Mizil and R. Caruana.  Predicting good probabilities with supervised learning.  In _Proceedings of the 22nd International Conference on Machine Learning_ , pages 625–632, 2005.
  * Ostachowicz et al. (2019) W. Ostachowicz, R. Soman, and P. Malinowski.  Optimization of sensor placement for structural health monitoring: A review.  _Structural Health Monitoring_ , 18(3):963–988, 2019.
  * Papamarkou et al. (2021) T. Papamarkou, J. Hinkle, M. T. Young, and D. Womble.  Challenges in markov chain monte carlo for bayesian neural networks.  _Statistical Science_ , 2021.
  * (172) N. A. PCoE.  Prognostics center of excellence - data repository.  https://ti.arc.nasa.gov/tech/dash/groups/pcoe/prognostic-data-repository/.  Last accessed on 2022-04-23.
  * PHM (2022) PHM.  Phm society data challenge.  https://data.phmsociety.org/, 2022.  Last accessed on 2022-05-24.
  * PHMAP (2021) PHMAP.  Data challenge at phmap 2021.  http://phmap.org/data-challenge/, 2021.  Last accessed on 2022-05-24.
  * PHME (2022) PHME.  The annual phme data challenge.  https://phm-europe.org/data-challenge, 2022.  Last accessed on 2022-05-24.
  * Plett (2004) G. L. Plett.  Extended kalman filtering for battery management systems of lipb-based hev battery packs: Part 3. state and parameter estimation.  _Journal of Power sources_ , 134(2):277–292, 2004\.
  * Plett (2006) G. L. Plett.  Sigma-point kalman filtering for battery management systems of lipb-based hev battery packs: Part 2: Simultaneous state and parameter estimation.  _Journal of power sources_ , 161(2):1369–1384, 2006.
  * Prudencio and Schulz (2012) E. E. Prudencio and K. W. Schulz.  The parallel C++ statistical library ‘QUESO’: Quantification of Uncertainty for Estimation, Simulation and Optimization.  In _Euro-Par 2011: Parallel Processing Workshops_ , pages 398–407. Springer, 2012.  URL http://dx.doi.org/10.1007/978-3-642-29737-3_44.
  * Ramancha et al. (2022) M. K. Ramancha, J. P. Conte, and M. D. Parno.  Accounting for model form uncertainty in bayesian calibration of linear dynamic systems.  _Mechanical Systems and Signal Processing_ , 171:108871, 2022.
  * Rebba and Mahadevan (2008) R. Rebba and S. Mahadevan.  Computational methods for model reliability assessment.  _Reliability Engineering & System Safety_, 93(8):1197–1207, 2008.
  * (181) N. L. Ricker.  Te code.  http://depts.washington.edu/control/LARRY/TE/download.html.  Last Accessed: 2022-04-17.
  * Ricker and Lee (1995) N. L. Ricker and J. Lee.  Nonlinear model predictive control of the tennessee eastman challenge process.  _Computers & Chemical Engineering_, 19(9):961–981, 1995.
  * Rohrs et al. (1985) C. Rohrs, L. Valavani, M. Athans, and G. Stein.  Robustness of continuous-time adaptive control algorithms in the presence of unmodeled dynamics.  _IEEE Transactions on Automatic Control_ , 30(9):881–889, 1985.
  * Rohrs et al. (1982) C. E. Rohrs, L. Valavani, M. Athans, and G. Stein.  Robustness of adaptive control algorithms in the presence of unmodeled dynamics.  In _1982 21st IEEE Conference on Decision and Control_ , pages 3–11. IEEE, 1982.
  * Sabatino et al. (2015) S. Sabatino, D. M. Frangopol, and Y. Dong.  Sustainability-informed maintenance optimization of highway bridges considering multi-attribute utility and risk attitude.  _Engineering Structures_ , 102:310–321, 2015.
  * Sachan et al. (2012) V. K. Sachan, S. A. Imam, and M. Beg.  Energy-efficient communication methods in wireless sensor networks: A critical review.  _International Journal of Computer Applications_ , 39(17):35–48, 2012.
  * Saha and Goebel (2007) B. Saha and K. Goebel.  Battery data set.  _NASA AMES prognostics data repository_ , 2007.
  * Saha et al. (2008) B. Saha, K. Goebel, S. Poll, and J. Christophersen.  Prognostics methods for battery health monitoring using a bayesian framework.  _IEEE Transactions on instrumentation and measurement_ , 58(2):291–296, 2008.
  * Saha et al. (2009) B. Saha, K. Goebel, and J. Christophersen.  Comparison of prognostic algorithms for estimating remaining useful life of batteries.  _Transactions of the Institute of Measurement and Control_ , 31(3-4):293–308, 2009.
  * Salvatier et al. (2016) J. Salvatier, T. V. Wiecki, and C. Fonnesbeck.  Probabilistic programming in python using pymc3.  _PeerJ Computer Science_ , 2:e55, 2016.
  * Saxena and Goebel (2008a) A. Saxena and K. Goebel.  Phm08 challenge data set.  _NASA Ames Prognostics Data Repository_ , 2008a.
  * Saxena and Goebel (2008b) A. Saxena and K. Goebel.  Turbofan engine degradation simulation data set.  _NASA Ames Prognostics Data Repository_ , pages 1551–3203, 2008b.
  * Scott et al. (2005) E. Scott, J. Brown, C. Schmidt, and W. Howard.  A practical longevity model for lithium-ion batteries: De-coupling the time and cycle-dependence of capacity fade.  In _208th ECS Meeting_ , 2005.
  * Sela and Amin (2018) L. Sela and S. Amin.  Robust sensor placement for pipeline monitoring: Mixed integer and greedy optimization.  _Advanced engineering informatics_ , 36:55–63, 2018.
  * Severson et al. (2019) K. A. Severson, P. M. Attia, N. Jin, N. Perkins, B. Jiang, Z. Yang, M. H. Chen, M. Aykol, P. K. Herring, D. Fraggedakis, et al.  Data-driven prediction of battery cycle life before capacity degradation.  _Nature Energy_ , 4(5):383–391, 2019.
  * Sharma and George (2018) M. Sharma and J. George.  Digital twin in the automotive industry: Driving physical-digital convergence.  _Tata Consultancy Services White Paper_ , 2018.
  * Shen and Huan (2021) W. Shen and X. Huan.  Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning.  _arXiv preprint arXiv:2110.15335_ , 2021.
  * Siemens Newsletter (2020) S. Siemens Newsletter.  Digitalization in industry: Twins with potential, Jun 2020.  URL https://new.siemens.com/global/en/company/stories/industry/the-digital-twin.html.
  * Sisson et al. (2022) W. Sisson, P. Karve, and S. Mahadevan.  Digital twin approach for component health-informed rotorcraft flight parameter optimization.  _AIAA Journal_ , 60(3):1923–1936, 2022.
  * Sjarov et al. (2020) M. Sjarov, T. Lechler, J. Fuchs, M. Brossog, A. Selmaier, F. Faltus, T. Donhauser, and J. Franke.  The digital twin concept in industry–a review and systematization.  In _2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)_ , volume 1, pages 1789–1796. IEEE, 2020.
  * Skardal et al. (2016) A. Skardal, T. Shupe, and A. Atala.  Organoid-on-a-chip and body-on-a-chip systems for drug screening and disease modeling.  _Drug discovery today_ , 21(9):1399–1411, 2016\.
  * Smith (2013) R. C. Smith.  _Uncertainty quantification: theory, implementation, and applications_ , volume 12.  Siam, 2013.
  * (203) D. SNL.  Battery archive.  https://www.batteryarchive.org/.  Last accessed on 2022-04-23.
  * Snoek et al. (2012) J. Snoek, H. Larochelle, and R. P. Adams.  Practical bayesian optimization of machine learning algorithms.  _Advances in neural information processing systems_ , 25, 2012.
  * Song et al. (2019) M. Song, B. Moaveni, C. Papadimitriou, and A. Stavridis.  Accounting for amplitude of excitation in model updating through a hierarchical bayesian approach: Application to a two-story reinforced concrete building.  _Mechanical Systems and Signal Processing_ , 123:68–83, 2019.
  * Soundappan et al. (2004) P. Soundappan, E. Nikolaidis, R. T. Haftka, R. Grandhi, and R. Canfield.  Comparison of evidence theory and bayesian theory for uncertainty modeling.  _Reliability engineering & System safety_, 85(1-3):295–311, 2004.
  * Subramanian and Mahadevan (2019) A. Subramanian and S. Mahadevan.  Error estimation in coupled multi-physics models.  _Journal of Computational Physics_ , 395:19–37, 2019.
  * Suresh and Kumarappan (2013) K. Suresh and N. Kumarappan.  Hybrid improved binary particle swarm optimization approach for generation maintenance scheduling problem.  _Swarm and Evolutionary Computation_ , 9:69–89, 2013.
  * Tan and Zhang (2020) Y. Tan and L. Zhang.  Computational methodologies for optimal sensor placement in structural health monitoring: A review.  _Structural Health Monitoring_ , 19(4):1287–1308, 2020.
  * Tasora et al. (2015) A. Tasora, R. Serban, H. Mazhar, A. Pazouki, D. Melanz, J. Fleischmann, M. Taylor, H. Sugiyama, and D. Negrut.  Chrono: An open source multi-physics dynamics engine.  In _International Conference on High Performance Computing in Science and Engineering_ , pages 19–49. Springer, 2015.
  * Team (a) T. P. D. Team.  Pymc3.  https://docs.pymc.io/en/v3/, a.  Last accessed on 2022-04-17.
  * Team (b) T. Q. D. Team.  Queso.  https://github.com/libqueso/queso/, b.  Last accessed on 2022-04-17.
  * Team (2022) T. S. D. Team.  Shm data sets and software.  https://www.lanl.gov/projects/national-security-education-center/engineering/software/shm-data-sets-and-software.php, 2022\.  Last accessed on 2022-05-24.
  * Thelen et al. (2022) A. Thelen, X. Zhang, O. Fink, Y. Lu, S. Ghosh, B. D. Youn, M. D. Todd, S. Mahadevan, C. Hu, and Z. Hu.  A comprehensive review of digital twin - part 1: Modeling and twinning enabling technologies.  _Structural and Multidisciplinary Optimization_ , 2022.
  * Tong et al. (2014) K. Tong, N. Bakhary, A. Kueh, and A. Yassin.  Optimal sensor placement for mode shapes using improved simulated annealing.  _Smart Struct. Syst_ , 13(3):389–406, 2014.
  * Van Amersfoort et al. (2020) J. Van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal.  Uncertainty estimation using a single deep deterministic neural network.  In _International conference on machine learning_ , pages 9690–9700. PMLR, 2020.
  * Van Dongen et al. (2005) B. F. Van Dongen, A. K. A. de Medeiros, H. Verbeek, A. Weijters, and W. M. van Der Aalst.  The prom framework: A new era in process mining tool support.  In _International conference on application and theory of petri nets_ , pages 444–454. Springer, 2005.
  * VanDerHorn and Mahadevan (2021) E. VanDerHorn and S. Mahadevan.  Digital twin: Generalization, characterization and implementation.  _Decision Support Systems_ , 145:113524, 2021.
  * Vega et al. (2021) M. A. Vega, Z. Hu, T. B. Fillmore, M. D. Smith, and M. D. Todd.  A novel framework for integration of abstracted inspection data and structural health monitoring for damage prognosis of miter gates.  _Reliability Engineering & System Safety_, 211:107561, 2021.
  * Verbeek et al. (2010) H. Verbeek, J. Buijs, B. Van Dongen, and W. M. van der Aalst.  Prom 6: The process mining toolkit.  _Proc. of BPM Demonstration Track_ , 615:34–39, 2010.
  * Viana et al. (2021) F. A. Viana, R. G. Nascimento, A. Dourado, and Y. A. Yucesan.  Estimating model inadequacy in ordinary differential equations with physics-informed neural networks.  _Computers & Structures_, 245:106458, 2021.
  * Walker et al. (2015) E. Walker, S. Rayman, and R. E. White.  Comparison of a particle filter and other state estimation methods for prognostics of lithium-ion batteries.  _Journal of Power Sources_ , 287:1–12, 2015.
  * Wang et al. (2013) D. Wang, Q. Miao, and M. Pecht.  Prognostics of lithium-ion batteries based on relevance vectors and a conditional three-parameter capacity degradation model.  _Journal of Power Sources_ , 239:253–264, 2013.
  * Wang and Wang (2006) P. Wang and T. Wang.  Adaptive routing for sensor networks using reinforcement learning.  In _The Sixth IEEE International Conference on Computer and Information Technology (CIT’06)_ , pages 219–219. IEEE, 2006.
  * Wang et al. (2015) P. Wang, B. D. Youn, C. Hu, J. M. Ha, and B. Jeon.  A probabilistic detectability-based sensor network design method for system health monitoring and prognostics.  _Journal of Intelligent Material Systems and Structures_ , 26(9):1079–1090, 2015.
  * Wang et al. (2009) S. Wang, W. Chen, and K.-L. Tsui.  Bayesian validation of computer models.  _Technometrics_ , 51(4):439–451, 2009.
  * Wang et al. (2008) T. Wang, J. Yu, D. Siegel, and J. Lee.  A similarity-based prognostics approach for remaining useful life estimation of engineered systems.  In _2008 international conference on prognostics and health management_ , pages 1–6. IEEE, 2008.
  * Wang et al. (2019) Z. Wang, H.-X. Li, and C. Chen.  Reinforcement learning-based optimal sensor placement for spatiotemporal modeling.  _IEEE Transactions on Cybernetics_ , 50(6):2861–2871, 2019.
  * Ward et al. (2021) R. Ward, R. Choudhary, A. Gregory, M. Jans-Singh, and M. Girolami.  Continuous calibration of a digital twin: Comparison of particle filter and bayesian calibration approaches.  _Data-Centric Engineering_ , 2, 2021.
  * Weigert et al. (2018) M. Weigert, U. Schmidt, T. Boothe, A. Müller, A. Dibrov, A. Jain, B. Wilhelm, D. Schmidt, C. Broaddus, S. Culley, et al.  Content-aware image restoration: pushing the limits of fluorescence microscopy.  _Nature methods_ , 15(12):1090–1097, 2018.
  * White et al. (2021) G. White, A. Zink, L. Codecá, and S. Clarke.  A digital twin smart city for citizen feedback.  _Cities_ , 110:103064, 2021.
  * Wilkinson et al. (2011) R. D. Wilkinson, M. Vrettas, D. Cornford, and J. E. Oakley.  Quantifying simulator discrepancy in discrete-time dynamical simulators.  _Journal of agricultural, biological, and environmental statistics_ , 16(4):554–570, 2011.
  * Williams and Rasmussen (1995) C. Williams and C. Rasmussen.  Gaussian processes for regression.  _Advances in neural information processing systems_ , 8, 1995.
  * Winterfeldt and Fischer (1975) D. V. Winterfeldt and G. W. Fischer.  Multi-attribute utility theory: models and assessment procedures.  _Utility, probability, and human decision making_ , pages 47–85, 1975\.
  * Xi et al. (2019) Z. Xi, M. Dahmardeh, B. Xia, Y. Fu, and C. Mi.  Learning of battery model bias for effective state of charge estimation of lithium-ion batteries.  _IEEE Transactions on Vehicular Technology_ , 68(9):8613–8628, 2019.
  * Xiong et al. (2009) Y. Xiong, W. Chen, K.-L. Tsui, and D. W. Apley.  A better understanding of model updating strategies in validating engineering models.  _Computer Methods in Applied Mechanics and Engineering_ , 198(15-16):1327–1337, 2009.
  * Yan et al. (2021) J. Yan, S. Laflamme, J. Hong, and J. Dodson.  Online parameter estimation under non-persistent excitations for high-rate dynamic systems.  _Mechanical Systems and Signal Processing_ , 161:107960, 2021.
  * Yang et al. (2020a) C. Yang, K. Liang, and X. Zhang.  Strategy for sensor number determination and placement optimization with incomplete information based on interval possibility model and clustering avoidance distribution index.  _Computer Methods in Applied Mechanics and Engineering_ , 366:113042, 2020a.
  * Yang et al. (2021) Y. Yang, M. Chadha, Z. Hu, M. A. Vega, M. D. Parno, and M. D. Todd.  A probabilistic optimal sensor design approach for structural health monitoring using risk-weighted f-divergence.  _Mechanical Systems and Signal Processing_ , 161:107920, 2021.
  * Yang et al. (2020b) Z. Yang, Y. Lu, H. Yeung, and S. Kirishnamurty.  3d build melt pool predictive modeling for powder bed fusion additive manufacturing.  22662:V009T09A046, 2020b.
  * Yao et al. (1993) L. Yao, W. A. Sethares, and D. C. Kammer.  Sensor placement for on-orbit modal identification via a genetic algorithm.  _AIAA journal_ , 31(10):1922–1928, 1993.
  * Ye et al. (2018) M. Ye, H. Guo, R. Xiong, and Q. Yu.  A double-scale and adaptive particle filter-based online parameter and state of charge estimation method for lithium-ion batteries.  _Energy_ , 144:789–799, 2018.
  * Yeung et al. (2020) H. Yeung, Z. Yang, and Y. Lu.  A meltpool prediction based scan strategy for powder bed fusion additive manufacturing.  _Journal of Additive Manufacturing_ , 35:101383, 2020.
  * Yi et al. (2011) T.-H. Yi, H.-N. Li, and M. Gu.  Optimal sensor placement for structural health monitoring based on multiple optimization strategies.  _The Structural Design of Tall and Special Buildings_ , 20(7):881–900, 2011.
  * Yucesan and Viana (2020) Y. A. Yucesan and F. A. Viana.  A physics-informed neural network for wind turbine main bearing fatigue.  _International Journal of Prognostics and Health Management_ , 11(1), 2020.
  * Zacharaki et al. (2021) A. Zacharaki, T. Vafeiadis, N. Kolokas, A. Vaxevani, Y. Xu, M. Peschl, D. Ioannidis, and D. Tzovaras.  Reclaim: Toward a new era of refurbishment and remanufacturing of industrial equipment.  _Frontiers in Artificial Intelligence_ , page 101, 2021.
  * Zhang et al. (2016) C. Zhang, P. Lim, A. K. Qin, and K. C. Tan.  Multiobjective deep belief networks ensemble for remaining useful life estimation in prognostics.  _IEEE Transactions on Neural Networks and Learning Systems_ , 28(10):2306–2318, 2016.
  * Zhang and Lee (2011) J. Zhang and J. Lee.  A review on prognostics and health monitoring of li-ion battery.  _Journal of power sources_ , 196(15):6007–6014, 2011.
  * Zhang et al. (2019) Q. Zhang, L. Shi, M. Holzman, M. Ye, Y. Wang, F. Carmona, and Y. Zha.  A dynamic data-driven method for dealing with model structural error in soil moisture data assimilation.  _Advances in Water Resources_ , 132:103407, 2019.
  * Zhang et al. (2014) X. Zhang, J. Li, J. Xing, P. Wang, Q. Yang, R. Wang, and C. He.  Optimal sensor placement for latticed shell structure based on an improved particle swarm optimization algorithm.  _Mathematical problems in Engineering_ , 2014, 2014.
  * Zhang et al. (2017) X. Zhang, S. Mahadevan, and X. Deng.  Reliability analysis with linguistic data: An evidential network approach.  _Reliability Engineering & System Safety_, 162:111–121, 2017.
  * Zhao et al. (2010) Y. Zhao, V. Pandey, H. Kim, and D. Thurston.  Varying lifecycle lengths within a product take-back portfolio.  2010\.
  * Zhao et al. (2017) Z. Zhao, B. Liang, X. Wang, and W. Lu.  Remaining useful life prediction of aircraft engine based on degradation pattern learning.  _Reliability Engineering & System Safety_, 164:74–83, 2017.
  * Zhu et al. (2021) J. Zhu, I. Mathews, D. Ren, W. Li, D. Cogswell, B. Xing, T. Sedlatschek, S. N. R. Kantareddy, M. Yi, T. Gao, et al.  End-of-life or second-life options for retired electric vehicle batteries.  _Cell Reports Physical Science_ , 2(8):100537, 2021.