# AI-coupled HPC Workflow Applications, Middleware and Performance

Wes Brewer  Oak Ridge National Laboratory, Oak Ridge, TN, USA  Ana Gainaru
Oak Ridge National Laboratory, Oak Ridge, TN, USA  Frédéric Suter  Oak Ridge
National Laboratory, Oak Ridge, TN, USA  Feiyi Wang  Oak Ridge National
Laboratory, Oak Ridge, TN, USA  Murali Emani  Argonne National Laboratory,
Lemont, IL, USA  Shantenu Jha  Rutgers University, New Brunswick, NJ, USA

###### Abstract

AI integration is revolutionizing the landscape of HPC simulations, enhancing
the importance, use, and performance of AI-driven HPC workflows. This paper
surveys the diverse and rapidly evolving field of AI-driven HPC and provides a
common conceptual basis for understanding AI-driven HPC workflows.
Specifically, we use insights from different modes of coupling AI into HPC
workflows to propose six execution motifs most commonly found in scientific
applications. The proposed set of execution motifs is by definition incomplete
and evolving. However, they allow us to analyze the primary performance
challenges underpinning AI-driven HPC workflows. We close with a listing of
open challenges, research issues, and suggested areas of investigation
including the the need for specific benchmarks that will help evaluate and
improve the execution of AI-driven HPC workflows.

##  1 Introduction

Multiple recent publications have demonstrated how AI coupled with traditional
HPC simulations can provide practical performance enhancements of 103 or more
[1, 2, 3]. AI-coupled-HPC workflows – as opposed to decoupled AI and HPC –
involve the concurrent, real-time coupled execution of AI and HPC tasks in
ways that allow the AI systems to steer or inform the HPC tasks and vice
versa. The real-time coupling and concurrency of HPC and AI tasks are
fundamental, as it allows allows bidirectional influence. The online coupling
of an AI system to HPC workflows can be used for a variety of scenarios, e.g.,
an AI-surrogate model to substitute part of expensive simulation, an AI system
to learn the function or determine the parameters or the practical fields, and
an AI system to guide a campaign using an objective function or performance
Pareto-optimal or resource-optimal experiments (using, for example,
Active/Reinforcement Learning).

Driving the need to couple AI systems to traditional HPC simulations is the
promise of enhancing the “effective performance” of HPC workflows, where
enhancement is measured by “science for a given amount of computing”. AI-
coupled HPC workflows continue to overcome the limitations of traditional
forward simulations in increasingly sophisticated and pervasive ways. Further,
AI-coupled HPC workflows will overcome traditional bottlenecks that prevent
greater scale – physical and temporal or higher resolutions. The integration
of AI into a computational workflow is also a sustainable and scalable way to
obtain significant performance gains and present an opportunity to avoid
simulation enhancements that are overly sensitive to processor architecture
[4, 5]. Put together, AI-coupled HPC workflows present a promising paradigm
[6, 7, 8] which leverages the ubiquitous interest and wide capabilities being
developed for AI but employs them to overcome performance bottlenecks due to
unsustainable approaches.

Integrating AI systems into traditional high-performance computing workflows
has demonstrably enabled highly accurate modeling and holds significant
promise for accelerating scientific discovery. There exist multiple promising
examples of workflows that couple AI methods with traditional HPC workflows,
typically by extending HPC workflows with additional capabilities to support
the concurrent execution of AI modules [9, 10, 11, 12]. Multiple solutions
have successfully coupled AI systems to be either “about” the HPC workflows or
“outside” (typically in the outer loop or even possibly remote from) the main
HPC workflow. However, the ability to substitute an AI system “inside” of an
HPC simulation remains challenging, with most solutions being customized to
specific simulation software or platforms. Figure 1 shows a typical science
application where AI has advanced simulations and experiments by coupling AI
to HPC workflows via different mechanisms. Furthermore, in molecular science
and engineering – which covers a diverse range of domains from biophysics to
materials to chemical engineering – AI has been used to make simulation
workflows more efficient and effective [13]. AI has been used to predict ever
more complex properties, such as energetic materials, solid-state materials
properties, and even the structure of complex materials. In addition, several
machine-learned potentials that enable access to quantum-chemical-like
accuracies at a fraction of the cost have emerged.

Figure 1: Example of an AI-coupled HPC workflow using models to replace
expensive high-fidelity simulations for steering instruments

The focus in this paper is on the scientific workflows that involve the online
(real-time) coupling and concurrent execution of AI and HPC, not offline
training and inference. In addition to providing an overview of existing AI-
coupled HPC workflows, the primary objectives of this paper are to identify
classes of AI-coupled-HPC workflows and to provide an overview of the software
systems used to support them. The paper also discusses performance and systems
challenges, identifying both robust solutions and the lack thereof. Finally,
the paper highlights some open research issues to advance AI-HPC workflows.

The paper is organized as follows: Section 2 reviews coupling modes between AI
and HPC tasks and introduces six execution motifs that describe the
interaction and coordination patterns of AI and HPC components. These motifs
are used to understand a range of AI-driven HPC workflow applications and
primary performance challenges underpinning AI-driven HPC workflows. Execution
motifs are important at multiple levels, not the least of which is their
influence on the design of application-level software systems and task runtime
systems. For instance, the frequency of interaction in a given motif informs
the performance requirements and the optimizations available to lower-level
libraries. Further, the diversity of AI approaches and model types means
intrinsic variation in training and execution times. Section 3 presents
several frameworks and libraries developed to ease the coupling of AI and HPC
components in a single workflow and illustrates how each of these frameworks
relates to the proposed high-level modes of execution motifs. Section 4
discusses various performance issues related to AI-HPC workflows – organized
around the motifs – such as load balancing, workload management, and dataflow
performance bottlenecks. We conclude in Section 5 with a discussion of several
open issues and challenges that the field of AI-coupled HPC workflows
presents.

##  2 Execution Motifs

AI systems can be coupled to HPC workflows using different methods at multiple
levels. Such AI-coupled HPC workflows, integrate AI systems with HPC
simulations are AI-in-HPC, AI-out-HPC, and AI-about-HPC simulations. These
three modes of coupling [6, 5] correspond to the placement of AI methods
relative to HPC simulations. AI-in-HPC represents the scenario in which an AI
system is introduced instead of a component of the HPC simulation, or
possibly, instead of the whole simulation itself, i.e., the AI model serves as
a “total surrogate” [14]. AI-out-HPC captures situations where an AI system
resides “outside” of the traditional HPC simulation loop but dynamically
controls the progression of the HPC workflow [15, 16] (e.g., control of
computational campaigns via reinforcement learning). Finally, AI-about-HPC
represents the situation where AI systems are concurrent and coupled to the
main HPC tasks: analysis and training codes use the output of the HPC
simulation to provide further insights. These three modes are not mutually
exclusive; in fact, we contend [4, 17, 5] that increasingly, scientific
applications will achieve performance gains and methodological advances by
combining all three modes. Thus, if not unified, software solutions for
managing the execution of each component and their data transfers for all
three modes must be compatible.

Building on the concept of coupling modes (AI-x-HPC, where x could be in, out,
or about), we introduce execution motifs. Whereas modes provide insight into
the static coupling of AI systems with HPC simulations, execution motifs
capture and provide information on recurring dynamic interaction patterns
between the AI systems and the HPC simulations, driven by a high-level
functional goal.

We define an execution motif using the characteristics the following
characteristics:

  * •

The first class of characteristics defines the interaction patterns between
the AI and HPC components: (1) data-flow type (e.g., one-to-one or one-to-
many) and direction (i.e., HPC-to-AI or AI-to-HPC or both); (2) control flow
type and direction (e.g., one/many AI components steering one or many HPC
simulations, HPC simulation guiding the AI training process, etc.); and (3)
user-involvement in the process.

  * •

The second class defines the coupling patterns between AI and HPC components,
namely: (4) concurrency requirements between the AI and HPC components (e.g.,
real-time interaction or post-mortem analysis); (5) dynamism requirements in
the composition of the workflow (e.g., spawning/terminating new processes or
training sequences) and in the interaction patterns (e.g., data/control flow
keeps changing, new connections are spawned/terminated); and (6) federation
requirements (e.g, network requirements between AI and HPC components or
heterogeneity in resources).

  * •

Each motif is associated with a scope which represents its primary purpose
(e.g., is AI used to improve or optimize HPC components, is HPC used to build
improved AI models, or both)

These execution motifs serve as identifiers for (groups of) AI-coupled HPC
workflows that exhibit similar interaction and coupling patterns between AI
and HPC components, and thus encounter similar performance bottlenecks and
require similar optimizations when deployed on HPC systems.

Figure 2: Defining characteristics of AI-coupled HPC workflows: (left)
Breakdown of the main characteristics used to define the different behavioral
motifs; (right) Main components (boxes) participating in the workflow
execution with two or more components interacting given different control and
data flow patterns (arrows).

Figure 2 illustrates (left hand-side) one column for each class of
characteristics. Figure 2 (right hand-side) shows the main components
participating in the workflow execution (namely: traditional HPC simulations
and codes; AI analysis and training; instruments; and humans in the loop)
together with the interactions between them (dashed lines show the control
flow and uninterrupted lines, the data flow). For each motif, different
components will be present, and the control and data flow will connect them in
different ways. In this section, we will use this diagram to further
illustrate the specific interaction and coupling patterns exhibited by each
motif.

Based on a literature survey of large-scale workflows, we identify and
characterize six primary execution motifs that couple AI and HPC.
Specifically, we identify five common execution motifs wherein AI enhances HPC
workflows (#1-5), two of them improving both HPC and AI (#3-4) and one where
HPC workflows are used to develop or improve AI systems or models (#6). We
briefly describe these execution motifs below and summarize their
characteristics in Table 1:

1. AI-based steering ensembles of simulations:In this motif, a HPC workflow, such as an ensemble of simulations are “steered” by an AI system. Sometimes, the AI can terminate or spawn simulations, making the coupling between components dynamic. In subsequent tables and figures, we refer to this motif as Steering.  2. Multistage pipeline:This motif represents a campaign composed of multiple stages with a typical AI-based function that reduces/filters simulations to spawn in the next stage based on the output of the current stage.  3. Inverse design:This motif captures the process of calculating, from a set of observations, the causal factors that produced them (e.g., calculating an image in X-ray computed tomography). AI-driven optimizations are used to iteratively identify causal factors from observational data that can be fed back into the experiment or simulation. The AI system can also be improved in the process thanks to the same feedback loop mechanism.  4. Digital replica:This motif includes the concurrent use of simulations and AI digital replicas (e.g., digital twins or multi-fidelity models) for scientific predictions and health monitoring of experiments or high-fidelity simulations. This motif can be executed in real-time, completely autonomously, or involve a scientist in deciding to guide future experiments.  5. Distributed models and dynamic data:This motif corresponds to workflows deployed over wide area networks. These workflows couple HPC simulations and experiments distributed across multiple remote sites with AI-based reduction/analysis and visualization. The AI can be used simply for compression, refactoring, or remote visualization of dynamic data or for steering the simulation or experiments. Workflows in this motif use highly heterogeneous resources, including edge devices, large leadership facilities, local clusters, or individual laptops for running and connecting different components. We refer to this motif as Distributed Models in subsequent tables and figures.  6. Adaptive execution for training large AI models:This motif couples the distributed training of large AI systems with HPC simulations and experiments. Such workflows can include training components using datasets previously generated by HPC simulations or real-time model optimization by running the training sequence concurrent with HPC simulations that are steered to generate the data required to cover the search space more efficiently. In subsequent tables and figures, we refer to this motif as Adaptive Training.

Table 1 summarizes the distinct patterns of each motif according to the
characteristics introduced in Figure 2.

Motif / Scope |  Interaction Patterns |  Coupling Patterns |  Example Use Case
---|---|---|---
Steering
AI improving HPC |  \- Control and data flow in one direction: data from HPC to AI, control from AI to HPC
\- One AI to one or many HPC
\- Optionally human in the loop |  \- Real-time requirements
\- Dynamic composition with HPC simulations spawned or terminated on the fly
\- Usually running in one facility |  AI-out-HPC
\- Command-and-control of physical experiments and simulations (e.g. between
shots feedback for plasma physics)
Multistage Pipeline
AI improving HPC |  \- Data flows in one direction from HPC to one or many AI or HPC components
\- AI filters control many HPC simulations
\- Typically interaction done without human in the loop |  \- Real-time requirements
\- Dynamic composition with branching in the workflow based on filters
\- Running in one facility |  AI-in-HPC and AI-out-HPC
\- Large-scale MD simulations using AI sampling of a system with many degrees
of freedom
InverseDesign
AI improving HPC
HPC improving AI |  \- Control flow from AI to HPC
\- Multiple HPC simulations and/or instruments sending data to AI (one or
many)
\- Typically interaction done without human in the loop |  \- Real-time is optional (AI can use existing datasets)
\- Execution can be concurrent or asynchronous
\- Running in one facility |  AI-in-HPC
\- Materials discovery to address the problem of data sparsity and reduce the
need for domain-specific knowledge
Digital Replica
AI improving HPC
HPC improving AI |  \- Data/control flow in both directions combining experiments and/or HPC with AI
\- Multiple HPC to one AI
\- Potentially human-in-the-loop to adjust future runs |  \- Real-time requirements with monitoring and visualization
\- Static composition with only visualization started on the fly
\- Components can potentially be executed distributed |  AI-about-HPC
\- Digital twin of a fusion reactor running concurrent with ITER [18] and
monitoring
\- Digital twin for new insights into disease mechanisms.
Distributed Models
AI improving HPC |  \- Data and control flow in both direction (AI to HPC and other AI and HPC to AI)
\- One or many HPC and many AI (analysis, replicas)
\- Optional human in the loop |  \- Near real-time requirements
\- Typically static composition, dynamic for adapting to the WAN performance
(e.g. compression accuracy)
\- Geographically distributed |  AI-out-HPC and AI-about-HPC
\- Edge-to-HPC execution of
large-scale simulations
\- Interactive notebook-
based analyses of instrument data and AI output
AdaptiveTraining
HPC improving AI |  \- Data flows from one HPC simulation to multiple AI
\- Control flows from analysis to HPC and AI training
\- No human-in-the-loop |  \- Near real-time requirements for AI, optional with HPC
\- Dynamic composition
\- Running in one facility, possible federated in the future |  AI-out-HPC
\- Hyperparameter optimization workflows
\- Distributed training of large language models
Table 1: Summary of motif characteristics according to their scope,
interaction patterns, and coupling patterns and example use cases.

Hereafter, we describe each motif in greater detail and provide examples of
existing and future implementations of these execution motifs.

### AI-based steering ensembles of simulations

AI-based steering is used in many fields to improve the performance of large
multi-physics, multi-resolution simulations, experiments, or an ensemble of
simulations. Figure 3(a) presents a typical workflow in this motif where the
steering decisions are informed by an AI system, external to the simulations,
that analyzes the generated data in near-real time or adaptively selects
scientifically meaningful simulations or experiments to run next.

  * •

Interaction patterns: Typically, one or many HPC simulations and/or
experiments generate data being analyzed by one AI analysis code. This
analysis frequently includes visualization for human-in-the-loop decision-
making. Data thus flows from the HPC to the AI, while control goes from AI to
HPC.

  * •

Coupling patterns: Near real-time analysis is typically required but not
always necessary, with some analysis codes steering the configuration of the
next ensemble of simulations. Simulations can be terminated, or new instances
can be spawned based on the analysis. If experiments are involved, the AI and
HPC components might be executed in a distributed fashion. However, such
workflows are typically executed on a single system.

  * •

Scope: Workflows in this motif usually create models of the simulations
offline, based on previous results or synthetic data, and use the AI inference
online to improve the HPC components.

(a) AI system steering (ensembles of) high-fidelity HPC simulations or
experiments to avoid undesired results or guide the workflow towards better
solutions.

(b) Sequential coupling of data-dependent multistage AI-coupled HPC workflow
as a pipeline with filtering/analysis codes between stages.

Figure 3: Workflow computational patterns for the Steering (a) and Multistage
Pipeline (b) motifs

The following literature survey focuses on workflows for the Steering
execution motif. These workflows use AI systems to analyze how “well”
(ensembles of) simulations are progressing toward their initial scientific
objective. Based on that analysis, they can change the “trajectory” when
exploring the parameter space. This allows them to avoid undesired results
either or increase performance or accuracy. The training of the AI system is
done offline before the execution of the workflow. Workflows in which the
steering process is also used to guide the online training of the AI system
will be investigated as part of the Adaptive Training motif.

The AI-based steering execution motifs can be found in workflows designed to
solve scientific problems in molecular dynamics [2, 19], fusion [20], or
energy systems simulations [21]. These workflows show orders of magnitude
speed-ups in pursuing their scientific objective. For example, this speed-up
can be measured as the gain in terms of simulated time performed while
covering the same parameter space, quantified by the states sampled by the HPC
simulation. The corresponding studies use either custom-designed solutions to
couple the codes [19] or workflow management systems, such as DeepDriveMD [22]
for protein folding simulations, or Colmena for electrolyte design [23].

Generally, the steering is done in near real-time, with the AI inference
running concurrently and interacting with the simulations. A human-in-the-loop
component can be added to the workflow, as in [24], to enable the interactive
deployment and steering of ensembles of fluid dynamics simulations. This
framework provides distributed steering and visualization through a user
interface. It allows for the performant execution of interactive simulations
for many scenarios while increasing their accuracy via hierarchical
refinements thanks to inputs from both the analysis codes and scientists.

DeepDriveMD [2] is a general-purpose framework to run an analysis code (as a
separate application) that pre-processes the simulation data (e.g. selecting
only a subset of atoms of interest or calculating physical parameters for a
protein’s native state), and then uses machine learning (ML) models on the
pre-processed data to decide which simulations to run next and terminates the
less productive MD simulations. By focusing on promising regions of the
parameter space, AI-based steering can efficiently navigate complex, high-
dimensional systems that were previously intractable. AI systems can also
significantly reduce the computational resources required to obtain accurate
results by steering simulations toward more relevant configurations.

The aforementioned example is a typical implementation of the Steering motif,
but other examples using different coupling solutions while sharing the
characteristics presented in Table 1 exist. The AI system analyzes the output
of the HPC simulation and either steers the existing simulation or triggers
new simulations based on the outcome.

### Multistage pipeline

Many HPC applications are executed as a pipeline with distinct stages and data
dependencies between these stages. The transition between subsequent stages
can be controlled via AI-based logic that decides whether the next stage needs
to be triggered based on reaching some set criteria or meeting a specified
objective. A transition can also involve an analysis to filter out data
between subsequent stages. All these patterns are composed into the Multistage
Pipeline motif illustrated by Figure 3(b).

  * •

Interaction patterns: Data moves from HPC stages generating data to one or
many AI analyses consuming it, but can also be passed between HPC applications
executed in sequential stages. AI controls which HPC stage will be executed.
Thus, data and control flow in one direction in a many-to-many or many-to-one
fashion. Most of the time, the process is automated without requiring humans
to analyze the output of the AI codes.

  * •

Coupling patterns: While near real-time interaction is a requirement,
processes are not typically spawned or terminated as in the Steering motif.
However, the workflow flow is dynamic by definition, with branching based on
filters. Typically, the entire workflow is executed on a single system.

  * •

Scope: AI is primarily used to branch the execution of HPC to guide it toward
optimized results. The scope of the Multistage Pipeline motif is thus to
improve HPC.

The following examples are typical implementations of the Multistage Pipeline
motif that we identified in the literature. The AI/HPC stages are either
separate applications or integrated into multiple hybrid applications
typically deployed on HPC systems sequentially or pipelined.

High-Throughput Virtual Screening Pipelines are emblematic of this motif. For
instance, a virtual drug discovery pipeline can use AI to improve the
effective sampling of individual stages [25]. Integrating multiple workflows,
which can each be an AI-coupled HPC workflow, into a single multistage
pipeline is another example of AI-driven discovery [26]. AI can also be used
in workflows composing multiple multi-fidelity simulations to dynamically
adjust how many and what fidelity levels to run based on the desired outcome
[27].

Less complex multistage pipelines can be found in neuroscience [28, 29] and
cancer research [30]. Both fields employ workflows composed of a sequence of
medical image processing applications that process the same Magnetic
Resolution Image (MRI) or Whole Slide Image (WSI) with the output of one
application informing the following decision. Each image-processing
application is typically a mix of HPC and AI codes composed of multiple
stages, including cleaning, standardizing, analysis, and post-processing
within one large code or as separate applications. One such workflow is, for
instance, used to identify lesions in the brain [29]. It consists of a 3-stage
pipeline, starting with a set of MRI images that go through a whole
segmentation code, followed by a surface reconstruction application, and
finally by an application to create a structure of the brain in MRI space.
Based on the output of these three applications, the workflow can further
trigger additional codes depending on the research objectives.

Similar strategies are used in biological and material sciences [31]. For
instance, an AI system can parse the output of a first simulation to extract
the respective starting configurations of an ensemble of simulations, each
performed on patches consisting of multiple lipids and containing one or more
proteins [32]. With further advancements in AI algorithms in these fields, the
coupling patterns might also evolve, allowing a closer interaction between the
AI and HPC. This can be done either by having the AI produce new intermediate
data on demand or by allowing the AI to integrate data from various resources
to optimize existing ones in the HPC simulations that are executed
concurrently.

### Inverse design

Deep learning has been used to solve inverse problems, i.e., determining
causal factors from a set of observations, in many scientific fields,
including material science [33, 34], nanophotonics [35, 36, 37], or heat
transfer [38]. However, the lack of sufficiently large training data sets is
often a crucial issue for these learning algorithms since their accuracy
heavily relies on the amount and the quality of the available data. This is,
for instance, the case with material characteristics identification, which
requires a huge amount of user time to acquire observational data. HPC
simulations have recently started generating sufficient noise-free data to
train AI models. The new challenge is getting accurate simulation results,
often requiring extensive testing and expertise to fine-tune input parameters.
This can also be time-consuming and resource-intensive, especially for complex
simulations since it requires validation over many tests. AI-coupled HPC
workflows following the Inverse Design motif address this data sparsity
problem and reduce the demand for domain-specific knowledge in determining the
input parameters of a simulation.

Figure 4(a) illustrates the typical structure of the Inverse Design execution
motif. Inverse Design algorithms usually rely on gradient-based AI algorithms
searching through a high-dimensional design space’s enormous degrees of
freedom. They are coupled with simulations or experiments to guide the
simulation towards more efficient designs and may re-train or update the
models online as the simulation and experiment progress.

  * •

Interaction patterns: Workflows in this motif execute multiple HPC simulations
(potentially together with instruments), sending data to AI (possibly multiple
if online training is used). Control flows from AI to HPC using iterative
refinement of design parameters. Typically, there is no human steering in the
loop.

  * •

Coupling patterns: The AI can use data that has been previously generated (in
which case real-time interaction is optional). The execution of the HPC and AI
components is static, concurrent, or asynchronous and usually occurs in the
same system.

  * •

Scope: AI models are improved to reflect physical phenomena better, while the
model can improve HPC simulations. Thus, the Inverse Design motif aims at
improving both AI and HPC components.

A powerful illustration of this motif is its application to material design
[39]. The method determines material behaviors and consists of three phases:
data generation, training, and the inverse phase. The data generation phase
uses a set of HPC simulations, each testing different material properties
using various material parameters as inputs. In the training phase, a Deep
Neural Network (DNN) model is trained on the simulation outputs and the
material parameters. The inverse phase compares the prediction and real input
values to improve the DNN model and simulation performance. Based on the
model’s output, new simulations are triggered, improving both the HPC
simulation’s performance and the AI system’s accuracy.

Examples of inverse design workflows can be found in several other fields:
bioinformatics, biochemistry, material science, and nanophotonics. Recent
breakthroughs in AI-based protein structure prediction (i.e., sequence from
the structure) have set the stage for the use of AI models for solving the
inverse problem in molecular sciences (e.g., protein design) [40, 41, 42].
Similarly, in materials science [43, 44] workflows for the inverse design
problem have been implemented where the physics simulation is coupled with
different optimization methods (e.g., directional Gaussian smoothing for
nanophotonics [43]) to improve the model used.

(a) Iterative refinement of design parameters combining experiments with HPC
simulations and AI surrogates

(b) Physics-based models combining HPC high-fidelity simulations and AI
reduced order models and/or digital twins

Figure 4: Workflow computational patterns for the Inverse Design (a) and
Digital Replica (b) motifs

### Digital replica

The Digital Replica motif incorporates workflows that couple the training or
usage of AI systems that simulate the behavior of physical phenomena (e.g.,
for digital twins [45]) with the HPC simulation and/or a scientific instrument
for a variety of scenarios and situations in a range of disciplines and
fields. These workflows typically utilize low-fidelity models or advanced
data-driven AI models to generate insights not possible using traditional
high-fidelity simulations or observational models (given time and computing
resource restrictions). Figure 4(b) presents an overview of the computational
pattern for this motif. The digital replica receives data from the HPC
simulation/instrument. It triggers analyses to steer the simulation (in near
real-time automatically or post-mortem using a human-in-the-loop intervention
to tune the parameters of the next simulation run). Initial training is
typically done offline before the execution of the workflow and re-triggered
to update the model based on discrepancies between what is predicted by the
simulation and feedback from the analysis.

  * •

Interaction patterns: Data and control flow in both directions, with the HPC
simulation/instrument data being used to steer the training of AI models and
the data generated by the digital replica being used to predict and control
the HPC components. Humans may be involved in monitoring and adjusting current
and future runs.

  * •

Coupling patterns: Real-time interaction is required. Typically, the
composition of the workflow is static, and the flow patterns change only when
visualization is turned on. The AI and HPC components can run in the same
facility or are geographically distributed (e.g. when instruments are
involved).

  * •

Scope: Both AI and HPC components are improved, the AI predictions improve the
run of the HPC and the data generated by HPC improve the digital replica.

Relevant studies that fit this motif come from diverse domains (e.g.,
bioinformatics / medical domain [46, 47]) where machine learning and
multiscale modeling are combined to create a digital twin, steer expensive
analysis codes towards new insights into disease mechanisms, and help to
identify new targets and treatment strategies.

The coupling of HPC simulations with surrogate AI models can be implemented
and deployed in HPC systems differently with different performance
implications [48]. This study explores three coupling schemes: tightly
coupled, which forms a single executable with one-to-one node mapping; semi-
tightly coupled, with separate executables mapped in various ways; and loosely
coupled, where separate executables run on different machines using many-to-
many mapping. It also recommends running digital twins on different
architectures and with different models. It shows the potential benefits of
these workflows with a use case in materials science.

Fusion energy researchers are turning to digital twins as a promising avenue
for efficient whole-device optimization and virtual component qualification,
crucial for successfully designing fusion reactor components under conditions
that defy real-world testing. Digital twin workflows are primed to play a
crucial role in ensuring the success of ITER [18], and pave the way for the
future of commercial fusion energy. Several studies [49, 50, 51] presents the
vision of the fusion science community to develop future workflows. They all
discuss the design of a workflow combining digital twins of different parts of
the fusion reactor (e.g., the Central Solenoid Converter Power Supply grid
[49]) coupled with real-time simulation strategies to monitor and study the
performance of the ITER grid while the reactor is functional.

### Distributed models and dynamic data

The data analytics modules (e.g., the AI system) in previous motifs are
typically executed using the same computing resources as the simulation. In
some cases (e.g., when visualization is used or instrument data needs to be
managed), parts of the workflow can be executed on a remote site, including
the scientist’s laptop. However, resource management strategies are usually
static, and workflows do not adapt their structure to performance changes. The
rapid development of edge infrastructures and localizing large amounts of
potentially unnecessary data have motivated the development of edge analytics.
The corresponding execution motif is illustrated by Figure 5(a). Workflows in
this motif couple HPC simulations and instruments with geographically
distributed AI digital twins, AI surrogate models, and AI analytics (including
multiple clusters, compute resources on the edge, user laptops, etc.). The
workflows dynamically adapt to the network and compute capabilities from the
point of view of task placement and resource management (e.g., triggering new
analysis or compression, adapting the accuracy of the data transfer,
allocating new resources, or spawning redundant computation for better
resilience).

  * •

Interaction patterns: Data and control flow in both directions from multiple
AI to multiple HPC (e.g., edge AI monitoring or AI filtering steering, AI
digital replicas generating data, HPC simulations generating data, and
updating the AI edge analysis components). For near real-time experiment
steering, no human is involved in the process; however, configuring the
execution of future experiments or large-scale simulations typically requires
humans in the loop to monitor the output of the previous analysis.

  * •

Coupling patterns: Near real-time is required to analyze the state of the
simulations/digital replicas and the state of the networks for the workflow to
adapt. The workflows typically use static composition in the executed
components but employ dynamic flow paths and configurations to provide good
performance regardless of the state of the systems and applications. The
execution is always performed in a federated manner.

  * •

Scope: The main scope of workflows in the Distributed Models motif is to
improve the execution and/or productivity of the HPC simulation or
instruments.

Typical examples of this motif include diverse models on the edge-to-exascale
continuum [52] and coupled simulations across multiple HPC centers [53, 54].
The workflows presented in [52] combine resources and services at the center
of the network (in Cloud and HPC data centers) with computing and storage
capabilities at the edge and in transit along the data path.

For coupling simulations across multiple HPC centers, it is possible to rely
on a single workspace that provides a global view of information shared from
multiple geo-distributed HPC data centers [53]. The feasibility of distributed
workflows has been shown by coupling the Rosetta-based scientific tools with
interactive notebook-based analyses executed outside the HPC center [54].
Another application of this approach is for neutron scattering science [55],
for which the performance challenges related to the transfer of large files
between the HPC center and the experimental facility have been identified as
one of the main performance bottlenecks for the deployment of workflow motif.
A complementary work to these studies investigates the overhead and
performance of coupling non-HPC systems (equivalent to edge devices) to handle
HPC workflows [56].

Finally, a survey of the field of federated learning [57] highlights potential
performance bottlenecks. The study of challenges associated with orchestrating
complex workflows in geographically dispersed environments [58] also
highlights the lack of efficient mechanisms for managing multistage workflows
that dynamically traverse distributed computing resources, edge devices, and
instrument networks. Addressing this gap requires novel paradigms for
orchestrating workflow execution across diverse platforms and geographical
boundaries.

(a) HPC simulations coupled with experiments and AI surrogate models and AI
analytics geographically distributed using dynamic interaction and resource
management

(b) Adaptive methods for training ensemble of models with large datasets on
HPC potentially coupled with simulations that generate the data needed for
training

Figure 5: Workflow computational patterns for the Distributed Models (a) and
Adaptive Training (b) motifs

### Adaptive execution for training large AI models

The previous motifs include training as part of their execution workflow
either offline before running the workflow or online by updating the models
based on the discrepancies between the simulation output and the predictions.
This last motif (Adaptive Training) focuses on the training phase as a
standalone workflow. It encapsulates all the adaptive methods to train one or
an ensemble of models using an existing dataset or running coupled with the
HPC simulation that generates the required training data (Figure 5(b)). The
training algorithm can interact with the simulation and guide its execution to
guarantee better data coverage for building robust models. This guidance can
be done within the training code or through a separate analysis code (e.g.,
the figure’s “Data quality” box). The analysis code that verifies the accuracy
of the created models may trigger a new training event (e.g., using different
model parameters) or spawn new simulation instances. The workflow can have a
human-in-the-loop component to guide the process towards specialized models.

  * •

Interaction patterns: Data flows from HPC to multiple AI; if surrogates are
used to generate data, the data flow is from AI to AI training. Control flows
from the analysis code to the HPC and AI training. Typically, the process is
automatic, and the human controls only the initial configuration of the
workflow.

  * •

Coupling patterns: Real-time coupling between AI and HPC is optional since
previously generated datasets can be used. The AI and analysis components
require real-time coupling. Dynamic composition is required, with the analysis
code optimizing the training process by terminating low-accuracy models and
spawning processes with configurations likely to increase the accuracy.
Typically, the training is done on the same system. However, federating
learning might start using similar workflows in the future.

  * •

Scope: The scope of the Adaptive Training workflows is to train accurate
models and to improve AI components.

There have been a variety of HyperParameter Optimization (HPO) approaches in
the recent years for designing hyperparameters for deep learning on HPC,
including using Bayesian optimization [59], evolutionary algorithms [60], or
genetic algorithms [61]. These methods have been used in HPC for the
development of AI surrogate models and traditional semi-analytic reduced
models (e.g., to build an AI model predictive control for optimal real-time
control landing of spacecraft [62]) or for building AI-based analysis routines
(e.g., the MENNDL HPO is used to build models for cancer pathology to classify
tumor-infiltrating lymphocytes [60]). Other examples of this motif include
Neural Architecture Search (NAS) [63, 64], Large Language Model (LLM) searches
[65, 66], Adaptive LLM training [67, 68]. Enabling dynamic interactions
between learning agents like HPO, NAS or LLMs, and HPC simulations (with or
without human intervention) is becoming increasingly used in several fields:
computational fluids dynamics [69], fusion [20], energy physics [21], and
climate science [70]. These studies combine HPC with AI to optimize the
training process by steering the simulation to generate real-time data. The
FASTRAN-IPS [20] workflow is an example of the training being done online in a
strongly coupled fashion. The simulation generates data iteratively that is
then used to train a surrogate model. At every step, the AI algorithm
identifies gaps in the parameter space and steers the simulation to generate
the missing data. Coupling AI and simulation with visualization is also used
to allow users to interactively explore complex parameter spaces given by a
large number of simulation runs and interactively steer or span new
simulations to cover the missing parts in the parameter space. This is when
reduced-form simulations developed through machine learning techniques are
coupled with visualization to allow interactive feedback and steer the
simulation to generate new data [21]. Estimates of the results of the full
simulations at a fraction of the computational cost are used to steer the
learning process, as the workflow requires a human to inspect the outcome of
the simulation.

### Summary: Scientific Workflow Applications Using Multiple Motifs

The motifs presented in this section use different coupling and interaction
patterns when executing HPC and AI for different scopes. These motifs are not
mutually exclusive; we expect future large-scale workflows to combine several
motifs. One such example deploys an AI-driven workflow, combining three motifs
[71]: (1) it adaptively steer MD simulation ensembles; (2) it further couples
this workflow with AI-based methods in a multistage pipeline fashion; (3) it
uses adaptive execution to learn which parts of the parameter space have been
sampled sufficiently and to trigger simulations from under-sampled regions.
Each component depends on a distinct solution: the steering uses DeepDriveMD,
the learning employs a 3D PointNet-based adversarial autoencoder [72], and the
overall integration relies on a custom solution.

Scientific computing infrastructure trends toward an integrated research
infrastructure (IRI) [73] that will unite data resources, experimental user
facilities, and advanced computing resources to accelerate the deployment of
large-scale simulations and workflows. Such integrated infrastructures will
accelerate the trend of coupling multiple HPC simulations, surrogate models,
and digital twins with AI analysis at multiple levels.

However, while initiatives like the IRI will drastically simplify workflow
building and deployment, enabling the efficient execution of complex workflows
that span diverse scientific landscapes and combine multiple AI-HPC coupling
motifs is non trivial. There is currently no solution that fits all
requirements across multiple applications and fields and often specialized
solutions are preferred at all levels of the software stack. In the next
sections, we present solutions for workflow management systems targeted at
individual motifs and/or specific to certain domains, discuss the performance
of current workflows, and highlight potential directions for this fast-paced
changing field.

##  3 AI-coupled HPC Workflow Management Frameworks

For two decades, automation, simplification, and workflow management
optimization have been active research topics in the HPC and distributed
systems communities. This has led to developing many libraries, frameworks,
and tools for domain scientists to advance their scientific agenda. More
recently, diverse research communities in both industry and academia have been
working on allowing AI frameworks to go beyond being a standalone black-box
operator to facilitate their inclusion into more complex workflows. While the
previous section highlighted that several studies involving AI-coupled HPC
workflows relied on custom solutions with low portability and high maintenance
cost, we observe a convergence of the efforts related to workflow management
in both the HPC and AI communities with the recent development of workflow
frameworks specifically targeting AI-coupled HPC workflows. We review some of
these frameworks and identify their connections with the execution motifs
identified in the previous section. Then, we identify several challenges
related to the development of those frameworks.

###  3.1 Overview of Workflow Management Frameworks

Table 2 lists these frameworks and summarizes their main characteristics. When
applicable, we also indicate the underlying workflow technology on which the
framework is built. We distinguish frameworks developed with a specific
application domain from more general-purpose tools that address issues shared
by several application domains, e.g., optimizing data access patterns and
related I/O operations.

Framework |  Application Domain |  Execution Motif(s) |  Underlying System
---|---|---|---
CANDLE [74, 75] |  Cancer research |  Steering — Multistage Pipeline — Adaptive Training |  SWIFT/T

DeepDriveMD [76] |  Biophysics/Molecular Simulations |  Steering — Multistage Pipeline — Adaptive Training |  Radical

MuMMI [77] |  Biophysics/Molecular Simulations |  Steering — Multistage Pipeline — Adaptive Training |  Maestro [78]

IMPECCABLE [79] |  Virtual drug discovery pipeline |  Multistage Pipeline — Inverse Design — Adaptive Training |  Radical

Colmena [23] |  General-purpose steering of ensembles |  Steering — Adaptive Training |  Parsl

SmartSim [80, 81] |  A scalable, open-source, multi-physics simulation API. |  Steering — Multistage Pipeline — Digital Replica — Adaptive Training |  Redis

Stimulus [82] |  General-purpose data management library |  Inverse Design — Digital Replica — Adaptive Training |  -

NoPFS [83] |  General-purpose I/O middleware for model training |  Inverse Design — Digital Replica — Adaptive Training |  -

Table 2: Characteristics of domain-specific and general-purpose AI-coupled HPC
Workflow frameworks.

The Cancer-Distributed Learning Environment (CANDLE) [74, 75] application
workflow includes data acquisition, analysis, model formulation, and molecular
dynamics simulations. AI is used to develop and test protein binding
hypotheses and better understand proteins’ behavior on cell membranes. This
requires performing Hyperparameter optimization to identify the most effective
model implementations efficiently. CANDLE directly incorporates parameter
exploration methods for the efficient exploration of parameter spaces of
orders greater than $10^{9}$. These methods manage large amounts of data and
rely on scalable data parallelism to speed up the AI model.

The DeepDriveMD [76] framework has been designed to effectively reduce the
dimensionality of MD simulations to get insights into how proteins fold. The
objectives are to build latent representations that automatically correspond
to biophysically relevant reaction coordinates (RCs) and drive MD simulations
to automatically sample potentially novel conformational states based on these
RCs. The AI-coupled HPC workflow underlying DeepDriveMD is composed of four
main steps: (i) A large ensemble of MD simulations generates the initial data
needed to (ii) train a Deep Learning algorithm, (iii) whose inference step
identifies new starting points in the ensemble of MD simulations, and (iv)
enable the actual steering of the ensemble. This AI-driven steering may
consist of either expanding the pool of initial MD simulations or pruning
simulations that seem to be stuck in meta-stable states.

The objective of the Multiscale Machine-Learned Modeling Infrastructure
(MuMMI) [77] is to couple a large-scale parallel simulation of a macro-scale
model based on dynamic density functional theory that spans micrometer length-
and millisecond time-scales with a micro-scale model based on high-fidelity MD
simulations. MuMMI completes this workflow with an AI-driven importance-
sampling framework to bridge the resolution gap between the models. It
identifies sub-regions of scientific importance in the results of the macro-
scale model, which then serves as candidate inputs to many thousands of
coarse-grained MD simulations. The results of these fast-responding
simulations are analyzed in situ to avoid the cost of storing them on disk and
feed a self-healing feedback loop to improve the parameters of the macro
model. To manage such an AI-coupled HPC workflow and steer the framework
towards the highest scientific interest simulations, MuMMI relies on the
Maestro workflow conductor [78]. Regarding data management, the coarse-grained
MD simulations launched by MuMMI could generate PB of data in over 400 million
files, a number to multiply by 10 when AI is added to the workflow. MuMMI
implements several local in situ and data aggregation strategies to reduce the
demand on the filesystem.

The IMPECCABLE [79] project aims at defining an integrated modeling pipeline
to identify effective chemical compounds to fight COVID-19, by screening tens
of billions of molecules. IMPECCABLE uses AI methods to improve both the
initial sampling of candidate molecules and the design of surrogate models of
multi-physics simulations. It also integrates these AI components and HPC
simulations into a single unified three-stage pipeline. IMPECCABLE relies on
the Radical-Cybertools [84] to manage the execution of this heterogeneous AI-
coupled HPC workflow on leadership computing facilities.

In addition to these frameworks proposed to address the challenges related to
the coupling of AI and HPC in a specific scientific domain, we detail
hereafter, the intents and characteristics of more general-purpose frameworks
that propose solutions applicable to several domains.

Colmena [23] is a framework allowing users to steer simulation ensembles using
an AI model. It handles the dispatch of highly computational simulation runs
and model retraining tasks, collating the simulation results and invocating
the AI model. It relies on the Parsl workflow management system to handle the
heavy computations and their results. Colmena also offers a framework called
the Thinker to design a decision-making multi-agent process adapted to the
users’ needs. Each agent can post task requests, handle results, or react to
resource-related events using Redis queues. The AI component thus serves as
the actual pilot of the simulation campaigns, deciding which inputs to select,
in what order, and when to execute each simulation run.

The SmartSim [80, 81] framework, developed by Hewlett Packard Enterprise
enables the composition of HPC simulations and AI models into an in situ
workflow. The HPC simulations act as data producers, that generate training
and inference data while the AI component consumes this data either for
training or inference. SmartSim relies on an in-memory database to store any
information shared by the HPC simulations and the AI components for the
duration of the run, which allows for interactions without having to pay for
expensive file-based accesses on a shared parallel file system. It also
exposes a multi-language communication API based on the publish/subscribe
paradigm, which facilitates the connection of scientific codes, usually
written in C, C++, or Fortran, and the Python-based ML frameworks.

Stimulus [82] is a data management library designed to ingest scientific data
into popular AI frameworks efficiently and safely. It aims to address the
problem of data format and access pattern incompatibility when HPC simulations
and AI components are combined in one of the motifs introduced in the previous
section. Indeed, following a strongly coupled approach in which HPC
simulations and AI components would share the same format is intractable and
will duplicate efforts. The approach followed by Stimulus is to unify several
popular scientific data formats (e.g., NetCDF, HDF5, ADIOS) under a single
interface that hides the complexity of each format. Additionally, Stimulus
proposes a generic data ingestion pipeline that any tensor-operator-based AI
framework can execute.

NoPFS (Near-Optimal PreFetching System) [83] is Machine Learning I/O
middleware aims to reduce shared filesystem contention when ingesting data to
train a model. To this end, NoPFS leverages the clairvoyance an I/O system can
have by determining how access will be made from the initial seed of a pseudo-
random number generator. This clairvoyance allows NoPFS to efficiently
prefetch data, exploiting the whole storage hierarchy available on a compute
node and minimizing access to the shared filesystem. It is not an AI-coupled
HPC workflow system per se, but using NoPFS to manage the I/O model training
would free some throughput on the parallel file system, which could then be
used by HPC simulations.

Note that the integration of AI components into more complex analysis
workflows is not limited to the HPC community. For instance, the Kafka-ML
project [85] aims to connect the Apache Kakfa distributed messaging system to
AI models to get more insight from the continuous data streams from the
Internet of Things sensors. Similarly, the Redis NoSQL database server
includes a module, RedisAI, allowing to develop inference servers fed with
Redis data streams [81] In the context of initiatives such as the Integrated
Research Infrastructure from DOE that aims to connect experimental facilities
producing such data streams to leadership-class HPC systems, we envision that
stream processing frameworks will become part of complex AI-coupled HPC
workflows soon.

###  3.2 Challenges

Studying these different frameworks highlights two main challenges to
developing AI-coupled HPC workflow management frameworks. The first challenge
is the coexistence and communication between HPC and AI tasks in the same
workflow. This communication is mainly impaired by the difference in
programming languages used in HPC (i.e., C, C++, and Fortran) and AI (i.e.,
Python). One solution to circumvent this language barrier would be to consider
that all communications between HPC simulations and AI models are done through
data. However, this raises other issues related to differences in data format
and access patterns [86]. The successful and efficient management of AI-
coupled HPC workflows thus call for a more unified data plane management in
which high-level data abstractions could be exposed, and the complexities of
the format conversion and data storage and transport could be hidden from both
the HPC simulations and AI models.

The second challenge concerns using the insight provided by AI models in the
HPC simulations. For the motifs outlined, this insight is used to guide,
steer, or modify the shape of the workflow by triggering or stopping new
simulations. This implies that the workflow management system must be able to
ingest and react dynamically to inputs coming from the AI models. However,
most existing workflow management systems enforce a static description and
execution of the workflows. New methods must be designed to enable more
flexible and adaptive workflow executions. Moreover, the more complex and
evolving data transfer patterns associated with such more dynamic workflows
may directly impact the performance of AI-coupled HPC workflows if not managed
properly. As feedback loops have additional dependencies between workflow
components, any delay in data production and storage directly impacts the
components that consume data, which have to wait to start their execution and
remain idle.

In this section, we considered frameworks developed to automate the execution
of AI-coupled HPC workflows in different scientific domains or designed to
address some common challenges from the perspective of their matching with the
proposed execution motifs. However, the sheer amount of data handled by these
frameworks (e.g., ”scaling to full memory ($>$ PB) on the largest
supercomputers” [74]), the high level of task concurrency they manage (e.g.,
16k concurrent tasks for DeepDriveMD [76]) or both (e.g., 36k concurrent tasks
and several PB of raw data for MuMMI [77]) can have some important performance
implications. In the next section, we focus on these performance
considerations of AI-coupled HPC workflows.

##  4 Performance

To analyze the performance of AI-coupled HPC workflow executions, we consider
its fundamental components and various strategies. Furthermore, we discuss
each workflow component’s challenges and performance implications and explore
how different coupling strategies affect performance. Then, we review the six
execution motifs in the context of these components and their coupling,
addressing the various challenges and bottlenecks. Summarizing the section in
Table 3, we provide a comprehensive overview of the main characteristics of
each motif, the associated typical performance bottlenecks, and coupling
strategies. Lastly, we investigate the implications of software implementation
and the significance of benchmarking such workflows.

As we delve deeper into the intricacies of AI-coupled HPC workflows, it
becomes evident that several open issues emerge, particularly regarding
interoperability, data handling, and resource allocation. These challenges
underscore the importance of understanding the performance and highlight
broader opportunities to enhance the efficiency and effectiveness of such
workflows.

###  4.1 Workflow Components

Each workflow component has performance implications individually and in their
coupling. In general, for AI, especially for AI raining, there is a trade-off
between computation vs. communication [87]. Offloading to AI accelerators
requires data movement and may be largely affected by how the accelerators are
integrated into the main supercomputer. GPUs typically achieve lower latencies
for large batch queries, although they incur significant loading overhead;
conversely, CPUs usually exhibit lower loading latencies and excel with
smaller batch sizes [88]. This topic has been studied comprehensively in Ref.
[89] for an AI-driven simulation across cross-facility infrastructure
performing simulations on a traditional HPC supercomputer while offloading the
AI model to a Cerebras accelerator, a wafer-scale application-specific
integrated circuit (ASIC) device.

Data IngestionData TransformationModel TrainingModel Deployment (a) Machine
learning training and deployment pipeline

Data IngestionSimulationData Production (b) Pipeline for typical HPC
applications

Figure 6: The HPC and AI pipelines concurrently running within a workflow
typically overlap different stages depending on the type of workflow they are
deployed in

Understanding these performance implications is crucial when considering the
entire machine learning training and deployment pipeline, as shown in Figure
6(a) [90]. The computation and communication trade-offs inherent in using
accelerators like GPUs or specialized hardware such as Cerebras influence this
pipeline at each step. For example, while GPUs might accelerate the Model
Training stage for large batch sizes, the initial Data Ingestion and Data
Transformation stages might still depend heavily on traditional CPU
processing. This interplay between different hardware capabilities emphasizes
the need for a nuanced approach in optimizing each pipeline stage for
efficient AI model development and deployment in HPC environments. Note that
we do not consider the evaluation of the model in performance analysis, as it
is primarily concerned with the accuracy of the model, while its computational
cost will be discussed in the section on the Model Deployment.

The HPC pipeline illustrated in Figure 6(b) has typical I/O and communication
network performance considerations. As HPC simulations evolved to include GPU
kernels and/or AI in their codes, the trade-offs described in the previous
paragraph are also becoming relevant for the HPC components. When deployed
together, the AI and HPC components will overlap different stages of their
respective pipelines and face new performance bottlenecks that are unnecessary
during individual runs. For example, the Data Production phase of the HPC
application will typically overlap with Data Ingestion for the Steering motif,
which might cause concurrent read/write access to the same datasets, resulting
in performance penalties due to I/O contention.

####  4.1.1 Data Ingestion

Traditional HPC simulations rely on I/O libraries highly tuned towards large
sequential file access (like ADIOS2 or HDF5), Deep learning frameworks
typically train on mini-batches of data, which tends to perform poorly on
parallel file systems such as Lustre and GPFS [91]. Efforts such as tf.data
seek to improve parallel I/O performance by encapsulating the batches as files
[92]. On the HPC side, studies are trying to integrate AI loaders with I/O
libraries having different strategies and performance implications. For
instance, Stimulus [82], discussed in Section 3, is a data management library
for ingesting scientific data into AI frameworks such as TensorFlow, PyTorch,
and Caffe. Other solutions (like HVAC [93]) use caching mechanisms to exploit
the node-local storage or near node-local storage to accelerate read I/O and
metadata lookups for deep learning applications running on HPC systems.

####  4.1.2 Data Transformation

Before data is used, it typically needs to be pre-processed, which may involve
feature scaling, resizing, etc. This is true both for training and inference.
Brewer et al. [94] studied the life-cycle performance of distributed inference
requests and showed that pre-processing on the CPU can be the most expensive
component in the inference life-cycle. Recently, there have been efforts to
speed up pre-processing using GPU. For example, Schifferer et al.[95] were
able to speed up their feature engineering pipeline by using a combination of
RAPIDS CuDF, Dask, UCX, and XGBoost on a single machine with multiple V100
GPUs.

####  4.1.3 Training

Model training is performed either on a single device or multiple devices in a
distributed way, which is generally implemented as data-based parallelism,
model-based parallelism, or both. Data-based parallelism methods shard the
training data, such that synchronous parallel processes each use a portion of
the training data, and either collective communication libraries – such as
NCCL, MPI, or Gloo – are used to perform an all-reduce to average the
gradients during training [96], or RPC-based methods such as gRPC are used to
communicate the gradients. Model-based parallelism may be required when the
memory footprint of the model exceeds the capacity afforded by the GPU.
Additional techniques such as Tensor-parallelism and Pipeline-parallelism
extend this approach by partitioning across or within the model’s layers.

The performance of AI models is sensitive to floating point number precision,
and there exists support from software stacks on diverse systems to support
various formats such as half-precision to double-precision. Mixed-precision
enabled implementations vary the precision per layer and can be enabled on the
fly , such as the Automatic Mixed Precision (AMP) on Nvidia GPUs. Benchmarks
play a pivotal role in evaluating the performance of training such models,
which we will further discuss in Section 5.3.

####  4.1.4 Model Deployment

Model deployment involves optimizing the model for inference performance,
which may involve freezing weights, fusing network layers, quantization, and
calibration [97]. Tools such as TensorRT for NVIDIA accelerators and OpenVINO
for Intel accelerators may perform such optimizations for specific targets.
Brewer et al. [98] show a 10X speedup using TensorRT to deploy a MobileNetV2
model on an NVIDIA T4 GPU. Inference is performed asynchronously parallelly
using in-memory approaches such as TensorFlow C API or distributed inference
serving approaches such as TensorFlow Serving or RedisAI [81].

Large-scale inference, such as required for LLMs as Transformers, poses
additional challenges. Pope et al. [99] studied the inference performance of
Transformer models with 500B+ parameters and elucidated the challenges
involved in efficiently scaling inference for LLMs. First, because the models
are generative, they proceed sequentially in time, so subsequent inferences
depend on the former ones. Second, models with a large memory footprint,
exceeding the capacity of a single accelerator, need to be partitioned across
multiple accelerators, resulting in additional communication overhead. They
study optimal ways of partitioning the models to maximize inference efficiency
and demonstrate scaling the Transformer model across 64 TPU v4 chips.

The speed and efficiency of these inference processes are heavily influenced
by how multiple components are coupled (e.g., how the HPC simulation is
coupled to an AI model) and whether the inference is executed in memory or
sent to a remote inference server on a CPU or GPU. The inference latency, a
critical factor in these scenarios, is affected by the network and
interconnect performance and the capability of the AI accelerator or GPU. Fast
and efficient inference is thus a culmination of optimized model deployment
strategies, appropriate use of hardware resources, and effective management of
network and interconnect systems. Using benchmarks is crucial to assessing
these complex challenges in model deployment and large-scale inference. This
aspect, including how benchmarks inform strategies for scaling and optimizing
inference performance, will be elaborated in Section 5.3.

###  4.2 Coupling Approaches and Protocols

Having discussed the performance implications of individual workflow
components, we now focus on the broader context of AI-coupled HPC workflows.
In particular, we will explore various coupling approaches used in AI-coupled
HPC workflows, each utilizing specific protocols and middleware patterns
tailored to their use cases. To clarify, an approach refers to the overall
method or strategy used for coupling components, including both the
interaction patterns (e.g., data-flow and control flow types and directions)
and coupling patterns (e.g., concurrency and dynamism requirements), whereas
the protocol describes the implementation details of how the data and control
messages are exchanged. For instance, inference deployments often employ
request-response type interaction patterns, such as HTTP or gRPC, which
facilitate communication between client applications and server-side inference
models. For example, SmartSim deployments of digital replicas use the the
Redis Serialization Protocol (RESP) to perform request-response inference
calls. In contrast, distributed deep learning architectures using in adaptive
execution typically rely on ring-all reduce type coupling patterns, using
protocols like MPI, NCCL/RCCL, or Gloo, to efficiently handle gradient
averaging across multiple nodes. More traditional machine learning
applications, like those found in data analytics, tend to use map-reduce style
interaction patterns, exemplified by frameworks such as Spark or Hadoop, for
processing and analyzing large datasets. Additionally, distributed models in
edge computing or IoT scenarios may adopt publish-subscribe interaction
patterns, with protocols like MQTT (used in systems like KubeEdge), to manage
data flow between various distributed devices.

To exemplify performance differences between types of distributed training
implementations, consider that early versions of TensorFlow, which used the
parameter server (request-response) architecture for distributed training,
achieved only 50% parallel scaling efficiency, whereas early versions of
Horovod using ring-all reduce achieved 90% parallel scaling efficiency [96].
In addition, implementations such as MPI can take advantage of InfiniBand,
whereas protocols such as gRPC only support communication over Ethernet.

The critical insight to remember is that the performance of a specific
workflow is substantially influenced not only by the efficiency of its
elements but also by the way those elements are interconnected and integrated.
Several studies have attempted to investigate how the coupling of workflow
components affects overall performance [100, 101, 102, 103, 104, 105, 106].
Several studies have attempted to investigate how the coupling of workflow
components affects overall performance [100, 101, 102, 103, 104, 105, 106].

Regarding protocol performance, Brewer et al. [100] benchmarked inference
performance using three different inference serving techniques: HTTP over
InfiniBand via a Python-based HTTP server, gRPC over Ethernet via TensorFlow
Serving, and the Redis serialization protocol (RESP) via RedisAI. They
experimented with several different approaches for load balancing, including
using a thread pool, an HAProxy round-robin load balancer, and using MPI.
Achieving good scalability required maximizing the batch sizes, which were on
the order of 262k samples, and running a single load balancer per node, which
round-robinned the inference requests to the multiple GPUs on that particular
node (i.e., strong coupling). They demonstrated one million inferences per
second for their model on 192 GPUs on the Summit supercomputer. Boyer et al.
[102] extended this approach to benchmark digital replicas in online
inferencing mode. They noted that naïve methods of integrating the simulation
with the AI model caused substantial slow-downs, which were significantly
mitigated by employing the previously mentioned inference server and load
balancing techniques.

Regarding coupling strategies, Yin et al. [101] investigated coupling a Monte
Carlo simulation to a machine-learned surrogate model using both TensorFlow
C++ API and RedisAI inference server. They studied the effect of coupling on
inference performance by defining three different coupling types: (1) tightly
coupled—in-memory inference with one-to-one mapping between process and
accelerator, (2) loosely coupled—distributed inference service on different
nodes with many-to-many mapping between the simulation and the model inference
server, and (3) semi-tightly coupled—running on same node using inference
serving approach, which can have one-to-one, many-to-one, or many-to-many
mapping. This is depicted in Fig. 7. They show that tightly coupled
deployments are preferred for latency-bound applications, whereas loosely
coupled deployments provide better usability via the easy-to-use SmartRedis
API.

Figure 7: Various strategies for AI-HPC coupling [101]. Tightly coupled
systems embed the AI inference in-memory with the simulation. The resource
here represents a type of AI accelerator such as GPU or ASIC. The simulation
can be decoupled from the AI model via an inference server. Semi-tightly
coupled systems may integrate the components within a single compute node or
across multiple nodes on a single supercomputer. Loosely-coupled integrations
distribute the components across disparate resources, e.g., in the cloud.

###  4.3 Motif Performance

We now analyze performance issues for each of the motifs introduced in Section
2 regarding their components and coupling methods. Table 3 summarizes the
typical performance bottlenecks for each motif and associated implementation.

Execution Motif |  Coupling between HPC simulation and AI component |  Typical Bottlenecks
---|---|---
Steering |  In-situ coupling, loose or tight coupling strategies |  Data transfer between simulation and AI, latency in decision-making
Multistage Pipeline |  Chaining of components, use of intermediate data storage |  Data transfer between stages, time spent in each stage
Inverse Design |  Iterative feedback loops, surrogate-assisted optimization |  Convergence of optimization algorithm, computationally expensive simulations
Digital Replica |  Co-training of models, hybrid modeling approaches |  Model integration and communication, training physics-informed AI models
Distributed Models |  MQTT, IoT protocols (e.g., ZigBee), streaming data |  Edge-to-HPC network latency, limited memory and compute capacity of edge devices
Adaptive Training |  MPI, NCCL, use of data parallelism or model parallelism |  Interconnect performance, network size variability leading to long-job impediment (NAS)
Table 3: Summary of execution motif characteristics, performance bottlenecks,
and coupling strategies [107].

####  4.3.1 AI-based steering ensembles of simulations

The Steering motif uses machine-learned models to steer ensembles of
simulations. Colmena is an example of such a framework for implementing such a
task on HPC and has been demonstrated for molecular design [23]. This type of
architecture is similar to distributed reinforcement learning applications,
designed to maximize a reward function by running many simulations
(environments) in parallel [108]. Dataflow performance is the main challenge
for these workflows, especially when a centralized server for task planning
that can become a performance bottleneck.

In the Steering motif, data is ingested from parallel simulations, processed,
and used to train machine learning models. These models are then evaluated for
effectiveness in guiding simulations and deployed in an HPC environment to
steer ensembles. For example, Colmena has three processes: a thinker, a task
server, and a worker. The thinker communicates with the task server using a
request-response architecture, specifically Redis queues, which use the Redis
Serialization Protocol (RESP). Workers are scaled up on the system using Parsl
[109].

####  4.3.2 Multistage pipeline

Campaigns involved in the Multistage Pipeline motif are common when
integrating several methods at different scales and resolutions. For example,
the space of possible molecular configurations might be very large, and any
single method cannot explore all possibilities with uniform probability. By
using different methods with varying levels of cost and accuracy, the space of
possible configurations can be traversed intelligently.

In addition to dataflow across different “executables”, this gives rise to
numerous performance issues. For example, the concurrent execution of multiple
different stages with different computational costs and resource requirements
results in a significant load balancing and workload management challenge.
Furthermore, suppose AI models are used in conjunction with or as a substitute
for each stage. In that case, many of the problems intrinsic to other motifs
arise (e.g., dataflow performance bottlenecks in the Steering motif).

Computational campaigns such as IMPECCABLE [25] use a master/worker-type
architecture to manage tasks, whereas MPI is used to scale horizontally across
multiple nodes. Inference workloads tend to be I/O bound due to tens of
thousands of compressed files that must be distributed across all the worker
nodes.

####  4.3.3 Inverse design

Inverse design is a technique of calculating the causal factors that produced
them from a set of observations. Such approaches in various domains such as
the search for catalysts [110], nanophotonics [111], and material design
[112]. Performance in this motif is primarily influenced by how representative
the model can capture the causal factors that produced it. This is prone to
vary based on the model architecture, computational cost, and ability to work
with high data rates. High-energy X-ray Diffraction Microscopy (HEDM) relies
on the high-precision (sub-pixel) position of Bragg diffraction peaks. The
computational cost of conventional methods becomes a bottleneck, especially
for in-situ experiments with complex detector technologies. BraggNN [113] is a
neural network designed to localize Bragg peak positions more efficiently than
a traditional conventional pseudo-Voigt peak fitting method in extracting the
peak information. A key challenge is enabling trained models to accommodate
high data rates from emerging synchrotron facilities [114].

####  4.3.4 Digital replica

The Digital Replica motif describes the integration of simulations with
machine learning models, either in cognitive simulations (CogSim) or digital
twins. The performance depends primarily on how the model is integrated into
the simulation (coupling type) and compute heterogeneity. As mentioned
earlier, this falls into two approaches: (1) in-memory or (2) distributed (via
inference server). Jia et al. [115] used an in-memory inference approach via
the TensorFlow C API to integrate a molecular dynamics (MD) simulation with a
machine-learned model. Their model can efficiently scale up to the entire
Summit supercomputer, achieving 91 PFLOPS in double precision and 162/275
PFLOPS in mixed-single/half-precision. For this reason, they won the Gordon
Bell Prize in 2020.

Partee et al. [81] investigated coupling a climate-scale, numerical ocean
simulation to a machine-learned oceanic eddy kinetic energy (EKE) surrogate
model on a heterogeneous cluster, where the simulation was running on many CPU
compute nodes. In contrast, the AI model was being served on one or more GPU
nodes using a RedisAI-based inference server on a single GPU node with
multiple CPU-based compute nodes sending inference requests to the GPU node
(i.e., weak coupling). Their model achieved 1.86 million online inferences per
second.

Boyer et al. [102] tested both in-memory inference and distributed inference
using TensorFlow Serving and RedisAI for three types of physics simulations.
They show that using an inference serving approach can outperform in-memory
approaches by enabling multi-threading on the inference server and sending
concurrent requests to the same GPU, exploiting advantages akin to hyper-
threading.

####  4.3.5 Distributed models and dynamic data

The Distributed Models motif describes workflows where components of the
workflow are typically distributed across disparate resources, called the
edge-to-cloud continuum, and may be realized as reinforcement learning,
distributed deep learning, online learning, transfer or multi-task learning,
federated learning, and lifelong or continual learning workflows [52].

There are a couple of performance issues to consider for this motif: (1)
performing inference at the edge (whether this will be performed via single-
stream, multi-stream, or offline mode), and (2) continually retraining the
model to keep up with data skew and drift. When inferencing at the edge,
possible considerations must be taken because edge devices usually have
reduced memory and computing capacity compared with HPC. Such implementations
are complicated by the additional policy considerations for implementation.
Tools such as FuncX/Globus Compute[116] lower the barrier to deployment of
such models by exposing endpoints publicly via Globus, where authentication is
done in the cloud.

Suhas et al. [117] discuss the challenges of building a federated ecosystem
for interfacing experimental facilities with simulations on HPC. They mention
that one of the main challenges in this workflow motif is to “close the
latency gap between computation and instruments”. Yin et al. [55] study an
example of this approach applied to neutron scattering and also note the
performance challenges of transferring large files between the HPC and the
experimental facilities.

In addition to the considerations outlined, Dube et al. [118] emphasize the
evolving landscape of HPC in the context of distributed models, underscoring
the convergence of HPC with AI and data analytics across diverse environments.
This ’Internet of Workflows’ highlights the growing need for computational
models that efficiently manage data movement and optimize workflows across
cloud, edge, and traditional supercomputers. Understanding and adapting to
these trends is crucial for enhancing performance and scalability in
distributed AI-HPC systems.

####  4.3.6 Adaptive execution for training large AI models

We discuss this motif for three types of implementations: Hyperparameter
Optimization (HPO), Neural Architecture Search (NAS), and training LLMs. HPO
is typically used to optimize a set of training hyperparameters, such as batch
size, learning rate, etc.; this is typically performed by training several
neural networks in an asynchronous parallel fashion. Optimization methods
generally fall into either random search or Bayesian camps, with novel
augmentations such as Hyperband designed to speed up the evaluations by
adaptive resource allocation and early stopping [119]. Tools such as
KerasTuner may be run distributed using a chief-worker strategy to scale such
methods across many nodes on HPC [120].

NAS, on the other hand, is primarily concerned with designing optimal neural
network architectures. Two popular methods for NAS include evolutionary
algorithms (primarily in the form of genetic algorithms) [121] and
reinforcement learning [122]. Regarding implementing such approaches on HPC,
Young et al. [123] present the Multi-node Evolutionary Neural Networks for
Deep Learning (MENNDL), which was used to design a Convolutional Neural
Network (CNN) on the Titan supercomputer at Oak Ridge National Laboratory.
Martinez et al. [124] developed a similar approach called Deep Learning for
Evolutionary Optimization (DLEO) and used it to solve a complex compressed
sensing problem for reconstructing vibration spectra. Such implementations are
generally map-reduce style architecture implementations, which map many
training events to multiple accelerators on HPC, reduce the model performance,
generate a new set of candidate networks to evaluate, and so on until
convergence. Because of such a pattern, researchers have exploited map-reduce
frameworks such as Apache Spark to perform such tunings on HPC [125]. Such
implementations become more complex when the models are so large each
candidate must be distributed across multiple GPUs, as in the case of training
LLMs. Such examples of adaptive training utilize asynchronous map-reduce style
communications organized by the job scheduler. Because different architectural
candidates may take longer to evaluate, the typical bottleneck is waiting on a
straggling candidate architecture job before proceeding to the next generation
in the optimization process. Another performance implication is the cost of
the reduction call for each generation.

LLMs are particularly challenging to train primarily because the models are
generally much larger than what can fit within the memory of a single GPU, and
the models need to be trained on colossal amounts of data. For example, GPT-3
has 175 billion trainable parameters and is trained on 499 billion tokens
[126]; it has been estimated that it would take 288 years to train GPT-3 on a
single V100 GPU [127]. Considerable developments in data and model parallelism
have enabled significant speedup by using data-based parallelism methods, such
as Horovod, and a combination of pipeline and tensor model parallelism [128].
Using such a triad strategy of data, tensor, and pipeline model parallelism
along with DeepSpeed by Microsoft, Narayan et al. [127] have demonstrated the
ability to train an LLM with one trillion parameters at 502 PF/s on 3,072
GPUs. Additionally, with the increasing number of non-traditional AI
accelerators, training LLMs require approaches to leverage the hardware. Emani
et al. [129] provided a comparative evaluation of training LLM models on
diverse AI accelerators. Recent studies have also further explored efficient
training strategies and performance implications of up to a trillion-parameter
LLMs on leadership-class supercomputers [130, 131]. Achieving such a feat
required overcoming GPU memory limitations and minimizing inter-node
communication latencies using a combination of pipeline and tensor model
parallelism.

##  5 Open challenges and opportunities

In this section, we identify two main challenges in deploying AI-coupled HPC
workflows at scale: (i) the need for co-designing facilities, middleware, and
benchmarks that address the performance of both traditional HPC simulations
and emerging AI algorithms and (ii) the need to design efficient interfaces
that would allow users to describe their workflow needs and their AI-HPC
interaction. In light of these challenges, this section aims to reveal open
opportunities in each area for the future advancement of AI-HPC workflows.

###  5.1 Hardware-Software Nexus

One of the main challenges for traditional HPC systems in accommodating AI-HPC
workflows is engineering a system adept at performing both traditional
simulations and modern AI tasks. This balance hinges heavily on the
performance of the internal interconnect, which is highlighted by the vast
contrast in performance results between the HPL-MxP [132] and the HPCG [133]
benchmarks. The former primarily measures FLOP performance, while the latter
measures GPU-CPU interconnect and GPU memory bandwidth performance.

Recognizing this challenge, innovators like SambaNova, Groq, GraphCore,
Untether AI, Tachyum, NextSilicon, Cerebras, and Fungible are pushing the
envelope, with a central focus on optimizing data movement–a crucial step to
boosting efficiency in modern AI training and inference. One of the main ways
this is achieved is by “moving the compute element directly adjacent to memory
cells” (untether.ai). Secondly, special-purpose ASIC chips are designed for
inference, such as tensor processing units (TPU), intelligent processing units
(IPU), data processing units (DPU), language processing units (LPU), tensor
cores, etc. Another factor to consider is the exploitation of reduced or
mixed-precision training and inference. Using tensor cores at reduced
precision offers order-of-magnitude improvements in inference performance
[100, 98], while GPU CUDA cores support training performance at double
precision. There does not always seem to be a good general understanding of
what level of precision is required for such scientific applications; such
largely depends on the type of application.

Understanding diverse AI accelerators is critical to leveraging them for AI-
HPC workflows. These accelerators exhibit great diversity in their hardware
and software stacks, which enables them to run a subset of AI models
efficiently. For example, the Cerebras wafer scale engine has massive compute
cores. In contrast, the SambaNova SN30 DataScale system has one terabyte of
memory per device, which supports models with large memory footprints, and
Groq aims to accelerate inference-based workloads.

The performance of AI models is impacted by choice of implementation, such as
C++ or Scikit-learn for classical ML algorithms or frameworks such as Horovod
[134], PyTorch DDP [135], DeepSpeed [136], Colossal AI [137], MagmaDNN [138],
TensorRT[139], and ONNX [140]. Each framework provides unique capabilities to
run the AI models efficiently across varying deployment scales such as edge,
cloud, and HPC scales. When DL models are run at a large scale, their
performance is impacted by the collective communication overheads, and there
exist various algorithms imbibed into these frameworks, such as AllReduce for
DDP and Scatter_Gather for DeepSpeed. Specialized libraries such as NCCL
provide tunable parameters to further optimize collective communications.

With the increasing model complexity and memory footprint, distributed
implementation of AI models [141] is an effective way to deploy models at
scale. These approaches include techniques such as data-parallel, model-
parallel, tensor-parallel, and pipeline-parallel implementations in currently
used frameworks. Ben-nun and Hoefler [142] present a survey of approaches for
parallelizing various DL networks through parallelism in network training and
inference. The authors discuss techniques critical to efficiently running
distributed models, such as neural architecture search, asynchronous
stochastic optimization, and collective communication schemes. Mayer and
Jacobsen [143] discuss the challenges and techniques for scalable deep
learning on distributed infrastructures, including hardware resource
scheduling, methods, and training and model data management.

###  5.2 Data and I/O Challenge

HPC platforms have settled on POSIX-compliant parallel file systems as the
primary storage engine and interface. Even for an object-based solution such
as DAOS, a POSIX shim layer is still needed to meet the requirement of more
traditional modes in scientific applications. However, large, sequential read-
and-write is no longer the dominant workload pattern for many data-intensive
AI-driven workflows. Instead, small, random read/write is much more prevalent
– which traditional POSIX-based parallel file systems have struggled to keep
pace with. The nominal approach is to provision a node-local storage layer,
where a small amount of SSD-backed devices are positioned close to the compute
blades and generally outside the storage network used by the parallel file
system. The challenge is that decoupling from primary storage necessitates
data staging, which can be achieved through policy-driven data tiering or
workflow support to streamline the process. The current solution space is not
mature enough and thus renders this a particular bottleneck to overcome for
data-intensive workflow integration. Projects such as NoPFS and SmartRedis
discussed in Section 3 are attempts in the right direction. More recent OLCF-6
RFP [144] calls for an AI-optimized storage system and highlights the
challenges and needs in this domain.

###  5.3 Benchmarking

To effectively evaluate the performance of AI-HPC workflows, benchmarking them
on diverse use case scenarios/motifs is essential. Such benchmarks could
reveal potential performance bottlenecks and, hence, lead to optimized
solutions. Several efforts in this direction include MLPerf [145], HPCAI500
[146], SAIBench from BenchCouncil [147], and HPL-AI [132]. MLPerf [145],
developed by the MLCommons consortium of industry, academia, and research
laboratories, is a state-of-the-art benchmark suite that includes various
machine learning tasks. The MLPerf HPC benchmark [148] has been developed to
represent scientific machine learning workloads typical of traditional HPC
environments. There is also an ongoing effort to include AI-HPC workflow
benchmarks in this suite in collaboration with the MLCommons Science group
[149].

In the context of training benchmarks, MLPerf [145] and HPCAI500 [146] are
notable for their comprehensive approach. These benchmarks often utilize
metrics such as “training time to convergence” and “time to train to a
specified accuracy”, providing a more nuanced view of performance than
traditional metrics like “time per batch” or “number of epochs”. Recent trends
in benchmarking have shifted towards measuring the time required to train
models to a certain level of accuracy, reflecting a more realistic assessment
of AI model efficiency in practical scenarios. However, these benchmarks’
major limitations are the smaller dataset sizes and the complexity of the deep
learning models in scientific applications, which do not always represent
realistic scenarios in HPC centers. The HPL-MxP (formerly HPL-AI) benchmark
[132], for instance, implements the LINPACK benchmark at a lower precision.
Currently, there is a lack of a comprehensive suite of AI-coupled HPC
workflow-based benchmarks covering most of the motifs presented in Section 2.
To leverage these benchmarks effectively, it is important to define
performance metrics such as end-to-end execution time, AI model throughput,
and accuracy.

In addition to training benchmarks, evaluating inference performance is
equally crucial in AI-coupled HPC workflows. Methods such as MLPerf Inference
[150] offer a comprehensive framework for this, measuring sequential inference
performance across various modes, including single-stream, multi-stream,
server, and offline (batch) modes. Inference benchmarks like smiBench [100]
are particularly useful for studying offline distributed inference performance
on HPC systems. Inference latency is a key metric representing the time taken
for a batch of data to be processed, with results typically reported in
seconds or milliseconds. This metric, along with throughput measured in Hz or
samples per second, provides a detailed view of the efficiency of AI models
during the inference phase. The MLPerf Inference benchmark, for instance,
adapts its metrics based on the scenario, using 90th-percentile latency for
single-stream inferencing and throughput for offline batch inferencing [150].

What do the motifs teach us about benchmarking? Most of the current
benchmarking efforts consider an application or model in isolation. There is a
gap in incorporating workflows that help measure end-to-end workflows’
performance and the crucial components to accelerate the workflow. Since it
may be expensive to use the workflows with massive datasets, expensive
simulation runs, and complex models, it may be worthwhile to investigate
benchmarks that are representative of the actual production runs besides using
synthetic data instead of experimental or simulated datasets.

Understanding the performance intricacies is extremely challenging as we see
an increased adoption of workflow tools in scientific machine learning
applications. This challenge could be addressed by leveraging existing
benchmarking efforts in ML and HPC and working to design and develop new
benchmarks accordingly. As this benchmarking process is expensive regarding
resources and human effort, it is worthwhile to invest in building benchmark-
friendly versions of end-to-end workflow use cases, which mimic realistic
workflow execution with minimal overheads in datasets and the scale of runs.

###  5.4 Cross-cutting: Interoperability, Energy-Efficiency and Resource
Allocation

AI-coupled HPC workflows often require a high-level of interoperability,
typically entailing connecting/interfacing with multiple endpoints for data,
AI models, and workflow integration. There are two emerging patterns of
choice: (1) REST API [151] and (2) message queue-based approach [152]. Each
has pros and cons; currently, no widely accepted interfacing mechanisms exist.
Further, time-sensitive patterns and long-term campaign patterns, as
highlighted by the IRI blueprint report [73], demonstrate the need for
interfacing beyond a single institution’s boundary, which brings a host of
related challenges such as federated IDs, authentication, authorization,
access control, etc., to implement a true end-to-end workflow integration.
These challenges are more pronounced in motifs connecting experiments with HPC
simulations (e.g., Distributed Models and Inverse Design) and in supporting
Steering and Multistage Pipeline motifs, which is only partially addressed by
specialized frameworks such as MuMMI or DeepDriveMD, and more general
frameworks such as SmartSim (ref. Section 3).

The importance of energy efficiency in high-performance computing (HPC) has
intensified due to the hard limits on maximum power capacity. A critical
observation in most AI-HPC workflows is their relatively low utilization of
peak performance, which may be due to data movement or other I/O activities
[94]. This sub-optimal utilization prompts a vital question: what should be
the ultimate goal in this context? The answer might lie in optimizing how one
could optimize productivity (Science/Watt) rather than raw performance
(FLOP/Watt). This optimization raises a further dilemma: should the focus be
on time-to-solution or throughput? One approach could be accelerating
processes through high concurrency levels and GPU utilization. Alternatively,
engaging GPUs less frequently might be more power-efficient, leading to a
trade-off of lower throughput for energy conservation, albeit with longer
processing times [153]. Other research areas for improving the energy
efficiency of AI-coupled HPC workflows involve dynamic scheduling policies,
data locality, ensemble pruning of unnecessary runs, and optimizations of
convolution implementations [154]. Benchmarking AI-coupled HPC workflows will
be key for understanding how to optimize energy efficiency [155].

The traditional batch-based job scheduling approach to resource allocation in
HPC is at odds with scientific workflows’ iterative and interactive nature.
These systems, designed for resource efficiency, struggle with the flexibility
and on-demand resource allocation required for modern scientific discovery.
Specifically, to meet the requirements of AI-coupled HPC workflows outlined in
this paper, HPC resource management must evolve to accommodate the changing
demands of scientific research. This could involve more adaptive scheduling
algorithms, leveraging cloud resources for elasticity, or creating virtualized
environments for better resource control. A shift towards an application-
centric approach in HPC resource allocation is necessary to support scientific
workflows’ diverse and evolving patterns. This ensures that computational
resources are used efficiently and infrastructural limitations do not hinder
discoveries. Hybrid techniques such as LSF-Kubernetes [156] and serverless
heterogeneous computing [157] seek to address this issue.

##  6 Summary and Discussion

This paper surveyed the extensive landscape of AI-driven HPC workflows,
categorizing them into six execution motifs commonly used in scientific
workflows: Steering, Multistage Pipeline, Inverse Design, Digital Replica,
Distributed Models, and Adaptive Training. We highlighted prominent middleware
exemplars of each motif and discussed their respective performance
implications, revealing the performance challenges due to the complexity of
integrating workflows across disparate resources. Moreover, we identify the
main challenges of deploying AI-coupled HPC workflows at scale, such as the
need for co-designed facilities and further development of middleware and
benchmarks. We also identify opportunities for advancement in each area.
Specifically, our survey identified a significant gap in the available
workflow benchmarks to assess the performance of the six execution motifs. As
such, we acknowledge one of the weaknesses in our survey is related to this
dearth of benchmarks, which limited our performance discussions to simpler
workflows. To address this issue, we propose a new suite of benchmarks
designed to target each execution motif.

As the landscape of AI-enabled HPC workflows continues to evolve, this survey
provides a snapshot of the current state of the art. It serves as a roadmap
for developing next-generation workflows. For example, there are three general
categories of how advanced AI systems such as Foundation Models (FM) [158] or
generative AI may be used with or as part of AI-coupled HPC workflows: (1)
Workflows to generate the data and train AI systems; (2) Once the advanced AI
system is trained, use those as intelligent ways of steering the simulation;
(3) Using advanced AI systems to create the coupling framework between the AI
and the simulation.

AI-coupled HPC workflows also offer benefits for pre-training and fine-tuning
of advanced AI systems. For example, in the context of the Adaptive Training
motif, training LLMs at scale on HPC offers potentially more efficient
alternatives at larger scales and lower costs than training in the cloud due
to specific performance optimizations at scale. In general, the deployment of
advanced AI systems within AI-coupled HPC workflows will benefit from the
robust computational power of HPC systems.

A significant application of generative AI in AI-coupled HPC workflows is
their potential to generate new workflows and associated job scripts based on
specific application needs, user requirements, and the computing
infrastructure. By understanding the objectives and constraints expressed in
natural language, LLMs can propose optimized workflows, tailor job scripts,
and suggest resource allocations that align with the desired outcomes. This
type of optimal workflow generation enhances efficiency and ensures that
computational resources are utilized most effectively.

Workflows are becoming first-class applications for procuring future DOE HPC
systems and implementing the Integrated Research Infrastructure (IRI) [73].
For example, the NERSC-10 RFP technical specifications [159] highlight three
classes of workflows to support scientific research campaigns on the future
HPC system: high-performance simulations and modeling workflows, high-
performance AI workflows, and cross-facility workflows for real-time steering
of experiments. These workflows will likely exhibit one or more or a
combination of the AI-coupled HPC workflow motifs introduced in Section 2.
Similarly, the technical specifications of OLCF-6 at the Oak Ridge National
Laboratory [144] also emphasize the fact that scientific campaigns are
inherently formulated as workflows that include “phases of HPC simulation,
high-bandwidth communication, interleaved AI training with simulation and
inferencing, and data processing”. In both cases, one of the main objectives
in explicitly including workflows in the specifications is to move (part of)
the management of such workflows from user space to the software ecosystem
operated by the facilities. A closer and hopefully better integration with the
platform should address the performance challenges related to AI-coupled HPC
workflows that we listed in Section 4.

Moreover, ESnet conducted a meta-analysis of workflow patterns across DOE
Office of Science programs in the context of the forthcoming IRI [160]. This
report highlights three main science patterns: time-sensitive workflows
requiring near-real-time response across facilities for timely decision-making
and steering; data integration-intensive workflows that combine and analyze
data from multiple sources; and long-term campaign workflows that require
sustained access to resources over years or decades to accomplish a well-
defined objective. In that context, the integration of AI and HPC in workflows
will likely occur in time-sensitive and data integration-intensive workflows
and be an important driver in designing the IRI framework.

The next step in scientific computing performance, scale, and sophistication
will require AI-coupled HPC workflows to be a central component of future
scientific campaigns. A thorough understanding of AI-coupled HPC workflows and
the development of new benchmarks will create a strong foundation for
addressing future advancements in the post-exascale era [161].

## Acknowledgments

This manuscript has been authored by UT-Battelle, LLC, under contract DE-
AC05-00OR22725 with the US Department of Energy (DOE). The US government
retains and the publisher, by accepting the article for publication,
acknowledges that the US government retains a nonexclusive, paid-up,
irrevocable, worldwide license to publish or reproduce the published form of
this manuscript, or allow others to do so, for US government purposes. DOE
will provide public access to these results of federally sponsored research in
accordance with the DOE Public Access Plan (https://www.energy.gov/doe-public-
access-plan).

This research used resources of the Oak Ridge Leadership Computing Facility at
the Oak Ridge National Laboratory, supported by the Office of Science of the
U.S. Department of Energy under Contract No. DE-AC05-00OR22725.

## Declaration of generative AI and AI-assisted technologies in the writing
process

During the preparation of this work the authors used GPT-4 in order to improve
grammar structures in certain places. After using this tool/service, the
authors reviewed and edited the content as needed and take full responsibility
for the content of the publication.

## References

  * [1] Andreas S. Panayides, Amir Amini, Nenad D. Filipovic, Ashish Sharma, Sotirios A. Tsaftaris, Alistair Young, David Foran, Nhan Do, Spyretta Golemati, Tahsin Kurc, Kun Huang, Konstantina S. Nikita, Ben P. Veasey, Michalis Zervakis, Joel H. Saltz, and Constantinos S. Pattichis.  Ai in medical imaging informatics: Current challenges and future directions.  IEEE Journal of Biomedical and Health Informatics, 24(7):1837–1857, 2020.
  * [2] Alexander Brace, Igor Yakushin, Heng Ma, Anda Trifan, Todd Munson, Ian Foster, Arvind Ramanathan, Hyungro Lee, Matteo Turilli, and Shantenu Jha.  Coupling streaming AI and HPC ensembles to achieve 100–1000$\times$ faster biomolecular simulations.  In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 806–816. IEEE, 2022.
  * [3] Austin Clyde, Xuefeng Liu, Thomas Brettin, Hyunseung Yoo, Alexander Partin, Yadu Babuji, Ben Blaiszik, Jamaludin Mohd-Yusof, Andre Merzky, Matteo Turilli, Shantenu Jha, Arvind Ramanathan, and Rick Stevens.  Ai-accelerated protein-ligand docking for sars-cov-2 is 100-fold faster with no significant change in detection.  Scientific Reports, 13(1):2105, Feb 2023.
  * [4] Geoffrey Fox, James Glazier, JCS Kadupitiya, Vikram Jadhao, Minje Kim, Judy Qiu, James P Sluka, Endre Somogy, Madhav Marathe, and Abhijin Adiga.  Learning everywhere: Pervasive machine learning for effective high-performance computation.  In IEEE International Parallel and Distributed Processing Symposium Workshops, pages 422–429. IEEE, 2019.  https://arxiv.org/abs/1902.10810.
  * [5] Shantenu Jha and Geoffrey Fox.  Understanding ML Driven HPC: Applications and Infrastructure.  In 2019 15th International Conference on eScience (eScience), pages 421–427. IEEE, 2019.  https://arxiv.org/abs/1909.02363.
  * [6] Shantenu Jha, Vincent R Pascuzzi, and Matteo Turilli.  Ai-coupled hpc workflows.  Chapter 28, Artificial Intelligence for Science, pp. 515-534 (2023), page arXiv:2208.11745, August 2022.
  * [7] Merchant, A., Batzner, S., Schoenholz, S.S. Scaling deep learning for materials discovery. Nature (2023). https://doi.org/10.1038/s41586-023-06735-9.
  * [8] Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, and Weihua Hu.  Learning skillful medium-range global weather forecasting.  Science, page eadi2336, 2023.
  * [9] Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, and Andreea Deac.  Scientific discovery in the age of artificial intelligence.  Nature, 620(7972):47–60, 2023.
  * [10] Alexander Lavin, David Krakauer, Hector Zenil, Justin Gottschlich, Tim Mattson, Johann Brehmer, Anima Anandkumar, Sanjay Choudry, Kamil Rocki, and Atılım Güneş Baydin.  Simulation intelligence: Towards a new generation of scientific methods.  arXiv preprint arXiv:2112.03235, 2021.
  * [11] Abigail Dommer, Lorenzo Casalino, Fiona Kearns, Mia Rosenfeld, Nicholas Wauer, Surl-Hee Ahn, John Russo, Sofia Oliveira, Clare Morris, Anthony Bogetti, Anda Trifan, Alexander Brace, Terra Sztain, Austin Clyde, Heng Ma, Chakra Chennubhotla, Hyungro Lee, Matteo Turilli, Syma Khalid, Teresa Tamayo-Mendoza, Matthew Welborn, Anders Christensen, Daniel G. A. Smith, Zhuoran Qiao, Sai Krishna Sirumalla, Michael O’ Connor, Frederick Manby, Anima Anandkumar, David Hardy, James Phillips, Abraham Stern, Josh Romero, David Clark, Mitchell Dorrell, Tom Maiden, Lei Huang, John McCalpin, Christopher Woods, Alan Gray, Matt Williams, Bryan Barker, Harinda Rajapaksha, Richard Pitts, Tom Gibbs, John Stone, Daniel Zuckerman, Adrian Mulholland, Thomas Miller, Shantenu Jha, Arvind Ramanathan, Lillian Chong, and Rommie Amaro.  $\\#$covidisairborne: Ai-enabled multiscale computational microscopy of delta sars-cov-2 in a respiratory aerosol.  bioRxiv, 2021.
  * [12] Yihang Wang, Joao Marcelo Lamim Ribeiro, and Pratyush Tiwary.  Machine learning approaches for analyzing and enhancing molecular dynamics simulations.  Current Opinion in Structural Biology, 61:139–145, 2020.
  * [13] David E Graff, Eugene I Shakhnovich, and Connor W Coley.  Accelerating high-throughput virtual screening through molecular pool-based active learning.  Chemical science, 12(22):7866–7881, 2021.
  * [14] M. F. Kasim, D. Watson-Parris, L. Deaconu, S. Oliver, H. Hatfield, D. H. Froula, G. Gregori, M. Jarvis, S. Khatiwala, J. Korenaga, J. Topp-Mugglestone, E. Viezzer, and S. M. Vinko.  Building high accuracy emulators for scientific simulations with deep neural architecture search.  Machine Learning: Science and Technology, page in press, 2021.
  * [15] Huachen Tao, Tianyi Wu, Matteo Aldeghi, Tony C Wu, Alan Aspuru-Guzik, and Eugenia Kumacheva.  Nanoparticle synthesis assisted by machine learning.  Nature reviews materials, 6(8):701–716, 2021.
  * [16] Zhenpeng Yao, Benjamín Sánchez-Lengeling, N Scott Bobbitt, Benjamin J Bucior, Sai Govind Hari Kumar, Sean P Collins, Thomas Burns, Tom K Woo, Omar K Farha, Randall Q Snurr, et al.  Inverse design of nanoporous crystalline reticular materials with deep generative models.  Nature Machine Intelligence, 3(1):76–86, 2021.
  * [17] Geoffrey Fox and Shantenu Jha.  Learning Everywhere: A Taxonomy for the Integration of Machine Learning and Simulations.  In 2019 15th International Conference on eScience (eScience), pages 439–448. IEEE, 2019.  https://arxiv.org/abs/1909.13340.
  * [18] N. Holtkamp.  An overview of the iter project.  Fusion Engineering and Design, 82(5):427–434, 2007.  Proceedings of the 24th Symposium on Fusion Technology.
  * [19] Silvina Caino-Lores, Michel Cuendet, Jack Marquez, Ekaterina Kots, Trilce Estrada, Ewa Deelman, Harel Weinstein, and Michela Taufer.  Runtime steering of molecular dynamics simulations through in situ analysis and annotation of collective variables.  In Proceedings of the Platform for Advanced Scientific Computing Conference, PASC ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  * [20] Wael Elwasif, Ross Whitfield, Jin Myung Park, and Mark Cianciosa.  Containers for massive ensemble of i/o bound hierarchical coupled simulations.  In 2020 2nd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC), pages 26–34, 2020.
  * [21] Brian Bush, Nicholas Brunhart-Lupo, Bruce Bugbee, Venkat Krishnan, Kristin Potter, and Kenny Gruchalla.  Coupling visualization, simulation, and deep learning for ensemble steering of complex energy models.  In 2017 IEEE Workshop on Data Systems for Interactive Analysis (DSIA), pages 1–5, 2017.
  * [22] Hyungro Lee, Heng Ma, Matteo Turilli, Debsindhu Bhowmik, Shantenu Jha, and Arvind Ramanathan.  Deepdrivemd: Deep-learning driven adaptive molecular simulations for protein folding.  CoRR, abs/1909.07817, 2019.
  * [23] Logan Ward, Ganesh Sivaraman, J Gregory Pauloski, Yadu Babuji, Ryan Chard, Naveen Dandu, Paul Redfern, Rajeev Assary, Kyle Chard, Larry Curtiss, Rajeev Thakur, and Ian Foster.  Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computing.  In Proceedings of the 2021 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC), pages 9–20. IEEE, 2021.
  * [24] Atanas Atanasov, Hans-Joachim Bungartz, Jérôme Frisch, Miriam Mehl, Ralf-Peter Mundani, Ernst Rank, and Christoph van Treeck.  Computational steering of complex flow simulations.  CoRR, abs/1807.00567, 2018.
  * [25] Aymen Al Saadi, Dario Alfe, Yadu Babuji, Agastya Bhati, Ben Blaiszik, Thomas Brettin, Kyle Chard, Ryan Chard, Peter Coveney, Anda Trifan, Alex Brace, Austin Clyde, Ian Foster, Tom Gibbs, Shantenu Jha, Kristopher Keipert, Thorsten Kurth, Dieter Kranzlmüller, Hyungro Lee, Zhuozhao Li, Heng Ma, Andre Merzky, Gerald Mathias, Alexander Partin, Junqi Yin, Arvind Ramanathan, Ashka Shah, Abraham Stern, Rick Stevens, Li Tan, Mikhail Titov, Aristeidis Tsaris, Matteo Turilli, Huub Van Dam, Shunzhou Wan, and David Wifling.  Impeccable: Integrated modeling pipeline for covid cure by assessing better leads.  50th International Conference on Parallel Processing (ICPP ’21), August 9–12, 2021, Lemont, IL, USA. ACM, New York, NY, USA, 12 pages, 2021.
  * [26] Abigail Dommer, Lorenzo Casalino, Fiona Kearns, Mia Rosenfeld, Nicholas Wauer, Surl-Hee Ahn, John Russo, Sofia Oliveira, Clare Morris, Anthony Bogetti, Anda Trifan, Alexander Brace, Terra Sztain, Austin Clyde, Heng Ma, Chakra Chennubhotla, Hyungro Lee, Matteo Turilli, Syma Khalid, Teresa Tamayo-Mendoza, Matthew Welborn, Anders Christensen, Daniel GA Smith, Zhuoran Qiao, Sai K Sirumalla, Michael O’Connor, Frederick Manby, Anima Anandkumar, David Hardy, James Phillips, Abraham Stern, Josh Romero, David Clark, Mitchell Dorrell, Tom Maiden, Lei Huang, John McCalpin, Christopher Woods, Alan Gray, Matt Williams, Bryan Barker, Harinda Rajapaksha, Richard Pitts, Tom Gibbs, John Stone, Daniel M. Zuckerman, Adrian J. Mulholland, III Thomas Miller, Shantenu Jha, Arvind Ramanathan, Lillian Chong, and Rommie E Amaro.  #covidisairborne: Ai-enabled multiscale computational microscopy of delta sars-cov-2 in a respiratory aerosol.  The International Journal of High Performance Computing Applications, 37(1):28–44, 2023.
  * [27] Hyun-Myung Woo, Xiaoning Qian, Li Tan, Shantenu Jha, Francis J. Alexander, Edward R. Dougherty, and Byung-Jun Yoon.  Optimal decision making in high-throughput virtual screening pipelines, 2022.
  * [28] Oscar Esteban, Christopher J. Markiewicz, Ross W. Blair, Craig A. Moodie, A. Ilkay Isik, Asier Erramuzpe, James D. Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, Hiroyuki Oya, Satrajit S. Ghosh, Jessey Wright, Joke Durnez, Russell A. Poldrack, and Krzysztof J. Gorgolewski.  fmriprep: a robust preprocessing pipeline for functional mri.  Nature Methods, 16(1):111–116, Jan 2019.
  * [29] Yuankai Huo, Andrew J Plassard, Aaron Carass, Susan M Resnick, Dzung L Pham, Jerry L Prince, and Bennett A Landman.  Consistent cortical reconstruction and multi-atlas brain segmentation.  Neuroimage, 138:197–210, September 2016.
  * [30] David J Foran, Eric B Durbin, Wenjin Chen, Evita Sadimin, Ashish Sharma, Imon Banerjee, Tahsin Kurc, Nan Li, Antoinette M Stroup, Gerald Harris, Annie Gu, Maria Schymura, Rajarsi Gupta, Erich Bremer, Joseph Balsamo, Tammy DiPrima, Feiqiao Wang, Shahira Abousamra, Dimitris Samaras, Isaac Hands, Kevin Ward, and Joel H Saltz.  An expandable informatics framework for enhancing central cancer registries with digital pathology specimens, computational imaging tools, and advanced mining capabilities.  J. Pathol. Inform., 13(100167):5, January 2022.
  * [31] Robert Pollice, Gabriel dos Passos Gomes, Matteo Aldeghi, Riley J Hickman, Mario Krenn, Cyrille Lavigne, Michael Lind̃ner D’Addario, AkshatKumar Nigam, Cher Tian Ser, Zhenpeng Yao, et al.  Data-driven strategies for accelerated materials design.  Accounts of Chemical Research, 54(4):849–860, 2021.
  * [32] Siewert J. Marrink, Luca Monticelli, Manuel N. Melo, Riccardo Alessandri, D. Peter Tieleman, and Paulo C. T. Souza.  Two decades of martini: Better beads, broader scope.  WIREs Computational Molecular Science, 13(1):e1620, 2023.
  * [33] Jia Wang, Yingxue Wang, and Yanan Chen.  Inverse design of materials by machine learning.  Materials, 15(5), 2022.
  * [34] Avni Jain, Jonathan A. Bollinger, and Thomas M. Truskett.  Inverse methods for material design.  AIChE Journal, 60(8):2732–2740, May 2014.
  * [35] Sunae So, Trevon Badloe, Jaebum Noh, Jorge Bravo-Abad, and Junsuk Rho.  Deep learning enabled inverse design in nanophotonics.  Nanophotonics, 9(5):1041–1057, 2020.
  * [36] Raphaël Pestourie, Carlos Pérez-Arancibia, Zin Lin, Wonseok Shin, Federico Capasso, and Steven G. Johnson.  Inverse design of large-area metasurfaces.  Opt. Express, 26(26):33732–33747, Dec 2018.
  * [37] Xiao Han, Ziyang Fan, Zeyang Liu, Chao Li, and L. Jay Guo.  Inverse design of metasurface optical filters using deep neural network with high degrees of freedom.  InfoMat, 3(4):432–442, 2021.
  * [38] J.J. García-Esteban, J. Bravo-Abad, and J.C. Cuevas.  Deep learning for the modeling and inverse design of radiative heat transfer.  Phys. Rev. Appl., 16:064006, Dec 2021.
  * [39] Li Zhong, Dennis Hoppe, Naweiluo Zhou, and Oleksandr Shcherbakov.  Hybrid workflow of simulation and deep learning on hpc: A case study for material behavior determination.  In 2021 IEEE International Conference on Cluster Computing (CLUSTER), pages 698–704, 2021.
  * [40] Eric Stach, Brian DeCost, A Gilad Kusne, Jason Hattrick-Simpers, Keith Brown, Kristofer Reyes, Joshua Schrier, Simon Billinge, Tonio Buonassisi, Ian Foster, et al.  Autonomous experimentation systems for materials development: A community perspective.  Matter, 4(9):2702–2726, 2021.
  * [41] AkshatKumar Nigam, Robert Pollice, Gary Tom, Kjell Jorner, Luca A Thiede, Anshul Kundaje, and Alan Aspuru-Guzik.  Tartarus: A benchmarking platform for realistic and practical inverse molecular design.  arXiv preprint arXiv:2209.12487, 2022.
  * [42] Tianle Wang, Sudip K. Seal, Ramakrishnan Kannan, Cristina Garcia-Cardona, Thomas Proffen, and Shantenu Jha.  A parallel machine learning workflow for neutron scattering data analysis.  In Proceedings of the 2023 IEEE International Parallel and Distributed Processing Symposium Workshops, pages 795–798, 2023.
  * [43] Jiaxin Zhang, Sirui Bi, and Guannan Zhang.  A directional gaussian smoothing optimization method for computational inverse design in nanophotonics.  Materials & Design, 197:109213, 2021.
  * [44] Christopher Yeung, Benjamin Pham, Ryan Tsai, Katherine T. Fountaine, and Aaswath P. Raman.  Deepadjoint: An all-in-one photonic inverse design framework integrating data-driven machine learning with optimization algorithms.  ACS Photonics, 0(0):null, 0.
  * [45] National Academies of Sciences, Engineering, and Medicine et al.  Foundational Research Gaps and Future Directions for Digital Twins.  The National Academies Press, 2023.
  * [46] Mark Alber, Adrian Buganza Tepole, William R Cannon, Suvranu De, Salvador Dura-Bernal, Krishna Garikipati, George Karniadakis, William W Lytton, Paris Perdikaris, Linda Petzold, et al.  Integrating machine learning and multiscale modeling—perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences.  NPJ digital medicine, 2(1):115, 2019.
  * [47] Karli Gillette, Matthias A.F. Gsell, Anton J. Prassl, Elias Karabelas, Ursula Reiter, Gert Reiter, Thomas Grandits, Christian Payer, Darko Štern, Martin Urschler, Jason D. Bayer, Christoph M. Augustin, Aurel Neic, Thomas Pock, Edward J. Vigmond, and Gernot Plank.  A framework for the generation of digital twins of cardiac electrophysiology from clinical 12-leads ecgs.  Medical Image Analysis, 71:102080, 2021.
  * [48] Junqi Yin, Feiyi Wang, and Mallikarjun Shankar.  Strategies for integrating deep learning surrogate models with hpc simulation applications.  In 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 01–10, 2022.
  * [49] Manuela Minetti, Andrea Bonfiglio, Ivone Benfatto, and Ye Yulong.  Strategies for real-time simulation of central solenoid iter power supply digital twin.  Energies, 16(13), 2023.
  * [50] David J. Kropaczek, Vittorio Badalassi, Prashant K. Jain, Pradeep Ramuhalli, and W. David Pointer.  Digital Twins for Nuclear Power Plants and Facilities, pages 971–1022.  Springer International Publishing, Cham, 2023.
  * [51] Michelle Tindall, Sebastian Rosini, Natasha Bowden, Thomas Deighan, Matthew Mavis, Stephen Biggs-Fox, and Thomas Barrett.  Towards a fusion component digital twin – virtual test and monitoring of components in chimera by systems simulation.  Fusion Engineering and Design, 191:113773, 2023.
  * [52] Daniel Rosendo, Alexandru Costan, Patrick Valduriez, and Gabriel Antoniu.  Distributed intelligence on the edge-to-cloud continuum: A systematic literature review.  Journal of Parallel and Distributed Computing, 2022.
  * [53] Awais Khan, Taeuk Kim, Hyunki Byun, and Youngjae Kim.  Scispace: A scientific collaboration workspace for geo-distributed hpc data centers.  Future Generation Computer Systems, 101:398–409, 2019.
  * [54] Alexander S. Ford, Brian D. Weitzner, and Christopher D. Bahl.  Integration of the rosetta suite with the python software stack via reproducible packaging and core programming interfaces for distributed simulation.  Protein Science, 29(1):43–51, 2020.
  * [55] Junqi Yin, Guannan Zhang, Huibo Cao, Sajal Dash, Bryan C. Chakoumakos, and Feiyi Wang.  Toward an autonomous workflow for single crystal neutron diffraction.  In Kothe Doug, Geist Al, Swaroop Pophale, Hong Liu, and Suzanne Parete-Koon, editors, Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation, pages 244–256, Cham, 2022. Springer Nature Switzerland.
  * [56] Aurelio Vivas and Harold Castro.  Estimating the overhead and coupling of scientific computing clusters.  SIMULATION, 99(3):245–261, 2023.
  * [57] Cédric Prigent, Gabriel Antoniu, Alexandru Costan, and Loïc Cudennec.  Supporting efficient workflow deployment of federated learning systems across the computing continuum.  In SC 2022-SuperComputing, 2022.
  * [58] M Asch, T Moore, R Badia, M Beck, P Beckman, T Bidot, F Bodin, F Cappello, A Choudhary, B de Supinski, E Deelman, J Dongarra, A Dubey, G Fox, H Fu, S Girona, W Gropp, M Heroux, Y Ishikawa, K Keahey, D Keyes, W Kramer, J-F Lavignon, Y Lu, S Matsuoka, B Mohr, D Reed, S Requena, J Saltz, T Schulthess, R Stevens, M Swany, A Szalay, W Tang, G Varoquaux, J-P Vilotte, R Wisniewski, Z Xu, and I Zacharov.  Big data and extreme-scale computing: Pathways to convergence-toward a shaping strategy for a future software and data ecosystem for scientific inquiry.  The International Journal of High Performance Computing Applications, 32(4):435–479, 2018.
  * [59] Prasanna Balaprakash, Michael Salim, Thomas D Uram, Venkat Vishwanath, and Stefan M Wild.  Deephyper: Asynchronous hyperparameter search for deep neural networks.  In 2018 IEEE 25th international conference on high performance computing (HiPC), pages 42–51. IEEE, 2018.
  * [60] Robert M. Patton, J. Travis Johnston, Steven R. Young, Catherine D. Schuman, Thomas E. Potok, Derek C. Rose, Seung-Hwan Lim, Junghoon Chae, Le Hou, Shahira Abousamra, Dimitris Samaras, and Joel H. Saltz.  Exascale deep learning to accelerate cancer research.  CoRR, abs/1909.12291, 2019.
  * [61] Daniel Martinez, Wesley Brewer, Gregory Behm, Andrew Strelzoff, Andrew Wilson, and Daniel Wade.  Deep learning evolutionary optimization for regression of rotorcraft vibrational spectra.  In 2018 IEEE/ACM Machine Learning in HPC Environments (MLHPC), pages 57–66, 2018.
  * [62] Reza Pirayeshshirazinezhad, Sandra G. Biedroń, Jorge Alberto Díaz Cruz, Salvador Sosa Güitrón, and Manel Martínez-Ramón.  Designing monte carlo simulation and an optimal machine learning to optimize and model space missions.  IEEE Access, 10:45643–45662, 2022.
  * [63] Yonatan Geifman and Ran El-Yaniv.  Deep active learning with a neural architecture search.  Advances in Neural Information Processing Systems, 32, 2019.
  * [64] Alberto Marchisio, Vojtech Mrazek, Andrea Massa, Beatrice Bussolino, Maurizio Martina, and Muhammad Shafique.  Rohnas: A neural architecture search framework with conjoint optimization for adversarial robustness and hardware efficiency of convolutional and capsule networks, 2022.
  * [65] Paul Barham, Aakanksha Chowdhery, Jeff Dean, Sanjay Ghemawat, Steven Hand, Daniel Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang, Sudip Roy, et al.  Pathways: Asynchronous distributed dataflow for ml.  Proceedings of Machine Learning and Systems, 4:430–449, 2022.
  * [66] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [67] Andrew Blanchard, Debsindhu Bhowmik, John Gounley, Jens Glaser, Belinda S Akpa, and Stephan Irle.  Adaptive Language Model Training for Molecular Design.  Journal of Cheminformatics, 15, 2022.
  * [68] Andrew E Blanchard, John Gounley, Debsindhu Bhowmik, Mayanka Chandra Shekar, Isaac Lyngaas, Shang Gao, Junqi Yin, Aristeidis Tsaris, Feiyi Wang, and Jens Glaser.  Language models for the prediction of sars-cov-2 inhibitors.  The International Journal of High Performance Computing Applications, 36(5-6):587–602, 2022.
  * [69] Yiqian Mao, Shan Zhong, and Hujun Yin.  Drlfluent: A distributed co-simulation framework coupling deep reinforcement learning with ansys-fluent on high-performance computing systems.  Journal of Computational Science, 74:102171, 2023.
  * [70] Donatello Elia, Sonia Scardigno, Jorge Ejarque, Alessandro D’Anca, Gabriele Accarino, Enrico Scoccimarro, Davide Donno, Daniele Peano, Francesco Immorlano, and Giovanni Aloisio.  End-to-end workflows for climate science: Integrating hpc simulations, big data processing, and machine learning.  In Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W ’23, page 2042–2052, New York, NY, USA, 2023. Association for Computing Machinery.
  * [71] Lorenzo Casalino, Abigail Dommer, Zied Gaieb, Emilia P. Barros, Terra Sztain, Surl-Hee Ahn, Anda Trifan, Alexander Brace, Anthony Bogetti, Heng Ma, Hyungro Lee, Matteo Turilli, Syma Khalid, Lillian Chong, Carlos Simmerling, David J. Hardy, Julio D. C. Maia, James C. Phillips, Thorsten Kurth, Abraham Stern, Lei Huang, John McCalpin, Mahidhar Tatineni, Tom Gibbs, John E. Stone, Shantenu Jha, Arvind Ramanathan, and Rommie E. Amaro.  Ai-driven multiscale simulations illuminate mechanisms of sars-cov-2 spike dynamics.  International Journal of High-Performance Computing Applications (IJHPCA), 2020.  Gordon Bell Prize Award (SC20)