BPExxx

    Braid Policy Engine
ML

    machine learning
FAIR

    findable, accessible, interoperable, and reusable
APS

    Advanced Photon Source
ALCF

    Argonne Leadership Computing Facility
SSRL

    Stanford Synchrotron Radiation Lightsource
SSX

    Serial Synchrotron Crystallography
XPCS

    X-Ray Photon Correlation Spectroscopy
HEDM

    High Energy Diffraction Microscopy
AP

    Action Provider
DAG

    Directed Acyclic Graph
ORM

    Object Relational Model
AWS

    Amazon Web Services
ECS

    Elastic Container Service
vCPU

    Virtual CPU
ACU

    Aurora Capacity Unit
CLI

    Command Line Interface
SDK

    Software Development Kit
CRUD

    Create, Read, Update, Delete
SEM

    Scanning Electron Microscope

# Steering a Fleet:
Adaptation for Large-Scale, Workflow-Based Experiments ††thanks:

Jim Pruyne  University of Chicago
Chicago, IL
<EMAIL_ADDRESS>Valerie Hayot-Sasson  University of Chicago
Chicago, IL
<EMAIL_ADDRESS>Weijian Zheng  Argonne National Laboratory
Lemont, IL
<EMAIL_ADDRESS>Ryan Chard  Argonne National Laboratory
Lemont, IL
<EMAIL_ADDRESS>Justin M. Wozniak  Argonne National Laboratory
Lemont, IL
<EMAIL_ADDRESS>Tekin Bicer  Argonne National Laboratory
Lemont, IL
<EMAIL_ADDRESS>Kyle Chard  University of Chicago
Chicago, IL
<EMAIL_ADDRESS>Ian T. Foster  University of Chicago
Chicago, IL
<EMAIL_ADDRESS>
###### Abstract

Experimental science is increasingly driven by instruments that produce vast
volumes of data and thus a need to manage, compute, describe, and index this
data. High performance and distributed computing provide the means of
addressing the computing needs; however, in practice, the variety of actions
required and the distributed set of resources involved, requires sophisticated
“flows” defining the steps to be performed on data. As each scan or
measurement is performed by an instrument, a new instance of the flow is
initiated resulting in a collection of concurrently running flows, a “fleet”,
with the overall goal to process all the data collected during a potentially
long-running experiment. During the course of the experiment, each flow may
need to adapt its execution due to changes in the environment, such as
computational or storage resource availability, or based on the progress of
the fleet as a whole such as completion or discovery of an intermediate result
leading to a change in subsequent flow’s behavior. We introduce a cloud-based
decision engine, _Braid_ , which flows consult during execution to query their
run-time environment and coordinate with other flows within their fleet. Braid
accepts streams of measurements taken from the run-time environment or from
within flow runs which can then be statistically aggregated and compared to
other streams to determine a strategy to guide flow execution. For example,
queue lengths in execution environments can be used to direct a flow to run
computations in one environment or another, or experiment progress as measured
by individual flows can be aggregated to determine the progress and subsequent
direction of the flows within a fleet. We describe Braid, its interface,
implementation and performance characteristics. We further show through
examples and experience modifying an existing scientific flow how Braid is
used to make adaptable flows.

## Keywords

Research process automation; Globus; High-performance computing; Distributed
Computing; Scientific Computing; Cloud computing

##  I Introduction

The scope of experimental science continues to increase rapidly particularly
in the rate and granularity at which measurements are taken with a
corresponding increase in generated data and the need to analyze, store, and
archive these data. When data processing can occur concurrently with data
generation, valuable insights can be obtained during an experiment which can,
in turn, allow researchers to tune or direct experiments. Processing
experimental data as they are generated typically involves a multi-step
process including data movement, computation, ML model training and inference,
data indexing and visualization, and human-based validation. To run
automatically and reliably, these steps are encoded in a “flow” which
precisely defines these steps and which can be reliably executed and monitored
by a workflow management system.

Commonly, a flow is started for each measurement, sample or image generated by
a scientific instrument. This simple, one-to-one correspondence between flow
executions and experimental events may be necessary as it reduces complexity
at scientific instruments where computing capacity may be low, event rates may
be high, and the stakes, such as lost data, for making erroneous changes in
experimental result collection processes may be high. This pattern results in
many flow executions taking place during a single experiment, often running
concurrently as their run-time is greater than the interval between the events
which generate them. We think of this collection of flows as a “fleet”
reflecting their independent progress with a collective goal to reach a single
destination or outcome – the result of the experiment.

When viewed as a collective fleet, we see limitations to the flow execution
per experimental event approach. In particular, when running independently, it
is difficult for results from early flow executions to be used by other
members of the fleet to steer their progress. It is also difficult for the
fleet as a whole to adapt to changes in its execution environment such as
availability of compute, storage, or other resources due to the rate and
automated manner in which flow executions comprising the fleet are launched.
We address these inefficiencies by introducing a new service, _Braid_ , which
can be used by flows within a fleet, as well as outside processes which may
observe fleet progress or environmental conditions, to enact adaptations which
individually steer flows within the fleet toward their collective goal.

Braid securely accepts and stores measurements from environmental monitors or
intermediate results from flows within a fleet to provide a basis upon which
adaptation decisions can be performed by running flows. The measurements may
be statistically aggregated over time intervals or measurement quantity to
determine trends or to eliminate outliers within the measurements. Flows may
then compare multiple such aggregations to determine their strategy. For
example, average waiting time for computation resources over a suitable time
window may be used to determine which of many computing sites a flow would
choose to run in. We present, from experience, multiple classes of flows which
can take advantage of the ability to adapt to improve scientific results, the
Braid service interface and examples of building flows using Braid and results
from running a fleet of flows which use Braid to perform a multi-stage
scientific computation.

##  II Motivation

In prior work [1] , we described a set of production flows and their roles in
various scientific experiments. These flows were used to act on data as they
are collected during an experiment. Once started, each flow executes
independently without consideration to performance or outcomes of other flows
or the experiment as a whole. Enabling communication between flows allows for
interactions between the various processes and components crucial to research
outcomes. Effective communication can ensure that data and results can be
easily shared, interpreted, and validated, reducing redundancy and errors,
enhancing the reproducibility and reliability of results. It also enables the
automation of tasks, streamlining the progression from one stage of an
experiment to another, which is critical for handling complex, large-scale
analyses and high-throughput data inherent in contemporary scientific
research. This optimization through communication is pivotal not only for
accelerating scientific discoveries but also for operational efficiency, by
removing redundant execution, improving utilization of instruments, and
reducing time to science.

Automation significantly enhances the efficacy and reliability of both large-
scale instruments, such as synchrotron x-ray sources, and smaller laboratory
instruments, such as SEMs, providing advantages across various facets of
scientific research. Large experimental facilities depend on automated data
management practices to improve throughput, precision, and increase resource
utilization by optimizing the analysis and cataloging of datasets while
facilitating precise control over experimental conditions. In the context of
smaller lab instruments, such as SEMs, data automation brings time efficiency,
cost-effectiveness, and consistency in results. Irrespective of the scale,
automation’s ability to streamline experimental operations, mitigate errors,
and manage high-throughput data is indispensable for the advancement of
contemporary science.

Globus Flows [2] has been applied to a variety of use cases and is routinely
used to perform and automate data management and analysis tasks for scientific
instruments. These flows can vary in complexity and perform a multitude of
tasks, however, a common pattern involves invoking a flow on each dataset as
it is generated; the flow then orchestrates the transfer of data and performs
analysis on a remote resource before publishing results and indexing relevant
metadata in a catalog. A number of challenges can arise when performing such a
flow, especially when data are rapidly produced and the flows leverage shared
networks and compute resources. Several of these challenges are described
below.

###  II-A Resource Constraints

Facilities rarely coordinate their availability (or unavailability) windows.
Thus, for example, an instrument facility and a computation facility may have
differing maintenance schedules. In turn, the instrument may continue to run,
producing data and flows to be processed, with the usual compute facility
unavailable. Thus, the flows may at best be queued: eliminating the benefit of
the anticipated, rapid response time of the automated flows. At worst, the
flows will fail and potential results may be lost.

Resources often have limits that restrict the number of concurrent computation
jobs or data transfers. A basic approach to experiment automation may submit
flows whenever new data is created independent of these limits. Like in the
resource unavailability situation, this may result in queuing or failures.
Failures are particularly troublesome in this case because these limits are
often unexpected or rarely encountered, and thus not robustly handled by
flows.

###  II-B Experiment, Simulation, and Training Convergence

Experiments (and simulations and model training) are often run repeatedly
until results converge. Further, in some cases experiments may fail, e.g., a
sample may be destroyed or machine learning may not converge. Such cases
require the ability to terminate flows either when success metrics are
achieved earlier than expected, or when the incremental increases are low, or
when results are not meeting required goals.

Such cases are normally represented by fleets with many flows running
concurrently and a need then to send signals between flows to change their
behavior. In such cases, we must determine, through policy, that an experiment
is or is not converging or that individual flow runs within the fleet are
erroneous. We want policy to be able to: cut short fleets that converge
quickly so as not to do work that is not needed, steer back on course or abort
a fleet that is headed the wrong way or re-route perhaps by re-processing to
work around elements that are considered erroneous.

###  II-C Dependencies Between Flows

As automated experiments grow more ubiquitous, they also grow more complex. We
have observed processes which come in “waves” in the sense that data generated
as output or results by one fleet becomes input and the trigger for a
subsequent fleet which further processes these initial results. Thus, there is
a need for methods to determine when appropriate or sufficient data is
generated by the initial fleet to indicate that the subsequent fleet should be
started. Commonly, this is the case when a desired state of the initial fleet
is reached such that the next phase may start.

###  II-D Summary

  * •

Flows must be able to observe the state of both resources and the fleet they
are a part of to adapt to changes in either.

  * •

Various conditions, especially errors that occur only under load, can be
difficult to plan for and thus are even more difficult to encode in flows.

  * •

Adaptation modes can include routing to alternative resources, throttling or
otherwise limiting the rate or degree to which resources are being used, or
can involve changing the steps, such as the computation mode.

##  III Braid Design

Braid provides abstractions, delivered via a service-based REST API, to
efficiently steer flow runs singularly and collectively toward a common goal.
Because the scale in terms of number of flows and time over which they run can
be large, and because the environment in which they run can also fluctuate and
have transient states, we require that our adaptations not be overly reliant
on short-term measurements or predictions. Thus, the general approach is to
accumulate measurements over numerous iterations or time and make decisions
based on statistical measures or trends while a fleet is performing its work.
The goal is to provide a minimal set of abstractions enabling flow runs to
coordinate while also making use of Braid within flows as easy as possible
avoiding complex syntax or the need for sophisticated logic to be introduced
into what are often otherwise simple sequences of steps. We intend Braid’s
functionality to be easy to use in new flows or to incorporate into pre-
existing flows at precisely the steps which would require an adaptation may
guide the flow’s execution.

###  III-A Concepts

We define a series of concepts which build closely upon one another to provide
the functionality necessary to coordinate fleets of concurrently running flows
and allow them to adapt to and interact with the environment and resources on
which they run.

####  III-A1 Datastreams

Any decision making capability relies on information or data. Therefore, the
foundation of Braid is the container for information termed the _Datastream_.
Commonly, a datastream will be established to monitor a particular resource,
such as expected wait time for computation, available space at (or bandwidth
to) a storage site, or to monitor the progress of an experimental process as
described in Section II. A datastream is composed of a sequence of _Samples_
where each sample is a numeric value represents a measurement or snapshot of
the state of the resource being monitored. Braid associates a timestamp with
each sample, providing order which is used when performing aggregation on a
datastream as described in Section III-A2.

####  III-A2 Metrics

While datastreams represent the information stored within Braid, _Metrics_ are
the method for generating a single, summarization value over the samples
within a datastream. Metrics are defined by  1) the datastream the metric is
to be computed over;  2) the _operation_ to be used to compute the metric
value;  3) the _interval_ within the datastream to compute the metric on; and
4) a _parameter_ value which is used by some operations to refine their
result.

Braid supports the following set of operations for computing the metric value:
1) Average;  2) Standard Deviation;  3) Count;  4) Sum;  5) Minimum;  6)
Maximum;  7) Mode;  8) Continuous Percentile;  9) Discrete Percentile;  10)
Last;  11) First; and  12) Constant.  The “Constant” operation always returns
its parameter value which is useful in further uses of Braid, as described
below, in which metric values are compared to one another. The other
operations compute a value over the specified interval which may be given as a
time interval or in terms of the number of samples, relative to the first and
last samples in the datastream.

####  III-A3 Policy

Ultimately, the purpose of Braid is to provide decision making support for
flows. A _Policy_ is the decision making abstraction. It consists of multiple
metric specifications with the policy selecting the maximum or minimum of the
computed metrics as the policy decision. Thus, metrics define the criteria for
a decision, and the policy determines the preferred value among the metrics.
The constant metric operation described above effectively allows a policy to
compare a metric to a set value such as a threshold.

The metrics within a policy provide the criteria upon which a policy decision
can be made, but they do not clearly describe how the policy decision can be
used or enacted. For example, a policy may compare metrics representing the
queue length at multiple compute sites averaged over the last ten minutes, but
the metric does not provide information on how to access the preferred compute
site. Thus, each metric in a policy request also has a _Decision_ value
associated with it, and it is this value that Braid returns as the outcome of
the policy request and which can then be used by a flow in subsequent steps.
Thus, the policy’s decision value can configure subsequent steps of a flow
directly, without branching or other logic which often adds significant
complexity to the flow development process. Furthering the example of
selecting among compute sites, the decision values associated with the metrics
would include identifiers, addresses, or other configuration information
specific to using the selected compute site.

Often, a datastream is configured to monitor a particular resource so
properties of that resource can be configured as a “default decision” on a
datastream so that a policy specification can omit the decision value on any
metrics involving datastreams which define a default decision. In this way,
the creator of a datastream who likely has most direct knowledge of the
resource being monitored or measured by the datastream can provide this
information within the datastream definition and then future users, such as
those writing policy statements into their flows, need not be familiar with
these specific details and use the values provided by the datastream creator.

###  III-B Implementation

Braid is deployed as a scaleable, cloud-hosted service making it suitable for
use by production scientific experiments. The datastream, metric and policy
concepts described above are accessed through a REST-style programming
interface. Datastreams are the foundation resource in the REST model. Each
datastream is given a descriptive name at creation and the service generates a
unique identifier which is used to reference the datastream in all subsequent
operations including the datastream’s life-cycle, adding samples and
evaluating metrics and policies.

####  III-B1 Authorization

As an internet accessible service, Braid must also provide protection for data
it stores. As a datastream is referenced in all operations including adding
samples and evaluating metrics and policies, the datastream also provides the
primary abstraction through which authorization is performed via a set of
roles associated with it. The user who creates the datastream is assigned the
_Owner_ role, and they may update the datastream to change the name, and the
identity of users in other roles (including the owner role allowing an owner
to transfer ownership to another user). The _Provider_ role governs which
users are allowed to add samples to the datastream. The _Querier_ role
likewise defines which users are allowed to access the datastream to evaluate
metrics and policies. These roles provide a valuable separation of concerns.
For example, a user may establish a datastream to monitor a resource, grant a
provider role to a user with access to monitor the resource and make the
datastream information available to a collection of other users, those running
flows which need to access the datastream, by granting the querier role to
them. In this scenario, neither the owner user nor any of the querier users
need be concerned with malicious or erroneous samples entering the datastream
as long as the provider is reliable and trusted.

All authentication operations are performed via Globus Auth [3] which provides
an OAuth-2 [4] interface for the generation and validation of tokens
associated with each REST API request. Further, identities associated with
Globus Auth users may also be placed in Globus Groups [5] and thus roles can
be assigned to these groups allowing a changeable set of users to be
associated with any role without making updates directly updating Braid to
enumerate the members.

####  III-B2 Client Tools

We provide client tooling in the form of a CLI and a Python SDK which can each
be used to access the full set of service interfaces. The CLI is typically
used when setting up an experiment environment such as creating and setting
roles on datastreams or providing initial samples to set state needed when
flows begin running. The SDK provides a means of accessing the Braid service
in more complicated or dynamic ways often involving interacting with resources
and providing on-going sampling of their states. We show examples of the use
of the CLI and SDK in Section IV.

####  III-B3 Flow-specific interfaces

In addition to the typical REST interface as supported by the above tools,
Braid’s principal use case for use within flows requires a method for it to be
invoked from a flow. We run our flows on the Globus Flows service, and
therefore Braid implements the “Action Provider” interface [2]. In particular,
the flows are able to invoke the following Braid functions:  1) adding samples
to a datastream;  2) evaluating a policy; and  3) waiting for a policy to
reach a particular decision.  Each of these operations within a flow is
subject to the same authorization requirements: to add a sample, the user
running the flow must be a provider to the datastream and to evaluate a
policy, the flow running user must be a querier of the datastream.

The ability to not just evaluate, but also wait for a policy decision (the
last operation in the previous list) provides a commonly used method for
synchronizing flows without need for loops, retries, back-offs or timeouts to
be encoded in a flow’s logic which is highly error prone given the care needed
to encode these operations correctly and the limited expressive capability of
flow syntax. The use of these operations within flows will be shown in more
detail in the example in Section IV.

##  IV Usage

Here, we present a detailed example of a fleet of flows adapting their
execution and coordinating their execution using Braid. The example presented
is similar to results presented in Section VI though somewhat simplified to
illustrate the use of Braid. The details presented here will also be
applicable when understanding the results presented later. The scenario
illustrated here can be summarized as follows:

> A computational experiment consisting of a number of flows, containing
> independent computations, are set to run concurrently. Each computation
> creates a result as well as an estimate for the progress of the experiment
> as a whole. When a sufficient number of progress estimates indicate that the
> computation is completed, each flow must also execute a “finalization”
> computation correlating its initial result with the overall result computed
> across the flows.
>
> Each flow may run using either of two compute clusters and the availability
> of capacity within the clusters may fluctuate during the lifetime of the
> experiment so the environment to be used should be selected at the start of
> each flow rather than at the start of the experiment as a whole.

In the scenario, we use three Datastreams for evaluating these two Policies:
1) When has the experiment made sufficient progress that finalization can be
computed by comparing a single datastream to a constant, threshold value; and
2) For each computation, which of the two clusters should be used by comparing
recent values within datastreams monitoring each of the clusters.  To run the
complete experiment, Braid must be used in the following manners:  1) in an
“administration” capacity, via the CLI, to setup initial state to be used
during the experiment;  2) in a “monitoring” mode, via the SDK, in which
samples representing the capacity of the clusters are periodically sent to the
appropriate datastreams; and  3) within flow execution when Braid will both be
consulted for policy decisions and updated with samples representing the
progress of the experiment.  Each of these interaction methods is outlined
below.

###  IV-A Administrative Usage

CLI usage to create a datastream takes the following form:

Listing 1: Datastream creation via the CLI Interface

⬇

policy datastream create

\--name cluster1

\--providers <monitoring_user_id>

\--queriers <experiment_runner_user_id>

\--default-decision '{"cluster_id": "<id_val>"}'

The parameters to this command define all of the properties for the datastream
to be created including:  1) the human readable name that will be presented
when datastream information is presented later (for example via a datastream
list sub-command);  2) the “providers” user(s) who can provide samples into
the datastream (e.g. via the programmatic usage described next)  3) the
“queriers” user(s) who can retrieve information from the datastream including
via policy evaluations (e.g. within a flow execution demonstrated below); and
4) the “default decision” value returned from a policy evaluation when a
metric referencing this datastream is selected in a policy evaluation.
Specifying the default decision here allows the administrator, who is most
likely to know the details needed to access this cluster, to provide that
information so various flows which may use the cluster need not each have
these details embedded.

###  IV-B Programmatic Usage

Monitoring the cluster could be performed with a daemon process written using
the SDK such as the following:

Listing 2: Populating a Datastream via the SDK Interface

⬇

client = BraidClient()

while True:

availability = get_cluster_availability()

client.add_sample(datastream_id,

value=availability)

time.sleep(sample_interval)

This code snippet starts by creating an instance of the SDK’s client class,
and then enters its monitoring loop. Each iteration of the loop gets an
estimate for the “availability” of the cluster and then sends it as a sample
to the configured datastream. Finally, the monitor sleeps for some interval
such that updates are sent at a regular interval. In our example scenario, we
have two available clusters, so two instances of this monitoring script may be
run. These scripts should continue to run throughout the experiment so that
updated availability information is always available to the flows.

###  IV-C Flow Execution Usage

The administrative and programmatic interactions set up use by each of the
many flows within the fleet that makes up the experiment. The flow
implementing our example scenario comprises five steps, three of which are
invocations of Braid. The step are described below in the sequence they appear
in the complete flow. We use a syntax based on the Amazon States Language [6]
with the inclusion of a new property “ActionUrl” which makes the step perform
a call out to the service at the corresponding URL which is either the Braid
service or a canonical compute service used in the example.

⬇

{

"ActionUrl": "<braid_location>/policy_eval",

"Parameters": {

"metrics": [{

"datastream_id": "cluster_monitor_id_1",

"op": "avg"}, {

"datastream_id": "cluster_monitor_id_2",

"op": "avg"

}],

"policy_start_time": -600,

"target": "max"

}

"ResultPath": "$.PolicyDecision"

}

The first step of the flow uses Braid to evaluate a policy determining which
cluster will be used throughout the run of this flow instance. The metrics to
compare are from the datastreams populated programmatically above
(cluster_monitor_id_1, cluster_monitor_id_2) computing the average (avg) for
each over the last ten minutes (policy_start_time value -600 seconds) to
select the maximum value. The ResultPath specification causes the policy
decision output to be stored in the state of the flow under the key
PolicyDecision and, as configured in the administrative section, the decision
will include a value for cluster_id indicating which cluster, not just
datastream, is considered to have the best availability to run the flow’s
computations most rapidly.

⬇

{

"ActionUrl": "<compute_service␣location>",

"Parameters": {

"cluster_id.$": "$.PolicyDecision.decision.cluster_id",

"comptation_parameters": {...}

},

"ResultPath": "$.ComputationResult"

}

The second step of the flow runs the primary computation associated with the
flow via a computation service. The computation service invocation is
parameterized with the cluster_id of the location on which the computation
should run. This is set referencing the value returned in the decision of the
previous, policy evaluation step using a JSONPath reference (as defined by to
the Amazon States Language). Per the scenario definition, the computation
result will contain a “quality” value which will be stored under the
ComputationResult key in the flow’s state.

⬇

{

"ActionUrl": "<braid_location>/add_sample",

"Parameters": {

"datastream_id": "result_quality_datastream_id",

"value.$": "$.ComputationResult.result_quality"

},

"ResultPath": "$.SampleResult"

}

The quality of the computation is next populated into the datastream created
for monitoring the progress of the fleet as a whole. This is done using the
add_sample flow interface. The value populated is retrieved from the flow’s
state under the $.ComputationResult.result_quality key created in the previous
step.

⬇

{

"ActionUrl": "<braid_location>/policy_wait",

"Parameters": {

"metrics": [{

"datastream_id": "result_quality_datastream_id",

"op": "discrete_percentile",

"op_param": 0.9,

"decision": "wait"

},{

"datastream_id": "result_quality_datastream_id",

"op": "constant",

"op_param": 0.95,

"decision": "proceed"

}],

"policy_start_limit": -10,

"target": "min",

"wait_for_decision": "proceed"

},

"ResultPath": "$.WaitPolicyDecision"

}

The previous step sampled the result quality to Braid and must wait for other
flows within the fleet to also report quality results and, further, for those
collected results to meet a threshold value before each flow may proceed. The
policy wait flow interface, introduced in Section III-B3, serves this purpose
by holding a flow blocked until the policy evaluation matches a desired
decision value (the underlying flow syntax allows for setting maximum step run
times and handling timeout exceptions should a policy wait hold the flow for
an unexpectedly long time). The condition to be met is “9 out of the last 10
quality samples must have a value of at least 0.95” which is expressed by:  1)
evaluating a metric on the quality datastream to compute the 90th (0.9)
percentile of the samples and setting the decision value to “wait” if this
metric meets the policy criteria;  2) evaluating a second metric using a
constant operation set to the value of 0.95 with a decision value of
“proceed”;  3) specifying that the metrics should be computed over the ten
most recent samples (policy_start_limit value -10);  4) that the target
criteria should be the minimum of the computed metrics and thus if the
constant is less than the computed quality its decision value “proceed” is
selected; and  5) the flow will remain paused at this step until the decision
has the value “proceed.”

⬇

{

"ActionUrl": "<compute_service␣location>",

"Parameters": {

"cluster_id": "$.PolicyDecision.decision.cluster_id",

"comptation_parameters": {...}

},

"ResultPath": "$.FinalizationComputationResult"

}

The last step of the flow is another computation using the same cluster as the
earlier computation. At this step of the flow, both this flow’s quality and
the aggregate quality computed in the policy of the previous step are
available parameters to the computation. Thus, it could calculate a final
result comparing its result to the aggregate or it could simply store its
relative results in a filesystem.

The flow presented here encapsulates some of the motivating concepts described
in Section II and is similar to the practical, science based application
described in Section VI. While concepts such as resource selection, experiment
progress measurement and synchronization of flows within a fleet are each
somewhat complex, the flow itself is fairly simple to read and to code. It
involves no conditionals and no loops: constructs which are more prone to
errors especially when implemented by non-programmers who are often the target
audience for workflow systems often promoted as “no code” solutions.

##  V Evaluation

Braid is deployed as a cloud-hosted service on AWS using a variety of
capabilities for deployment, protected interfacing to the internet, flexible
execution environments, and flexible database back-ends. In particular, we
employ the AWS ECS [7] to host the service and Aurora [8] running Postgres [9]
for the database layer. The service is implemented in Python using the FastAPI
[10] framework and packaged as a Docker container for deployment to ECS. Use
of flexible cloud hosting means that we can easily scale up or down the
capacity of the service based on load and cost constraints.

To obtain a baseline for the performance and scaleability of the service, we
established a deployment in ECS with two service instances, each with 8GB of
memory and 2 vCPUs. The Aurora database cluster is provisioned to scale to a
maximum of 4 ACUs. In production use, we impose rate limits on samples
ingested as well as metric and policy evaluations performed. To constrain
storage consumption, we cap the total number of samples retained in any one
datastream to one million entries with older entries automatically removed.

###  V-A Performance Testing

We employ a series of micro-benchmarks to evaluate both the rate at which
samples can be ingested and the rate at which metric evaluations of various
types can be completed. In each of the tests, the clients which perform the
measurement are hosted in AWS to minimize the variability introduced by
internet wide latency.

Figure 1: Request rates driven by a single client

The first micro-benchmarks measures the rate at which samples can be ingested
into a single datastream by a single client which waits for completion of each
request before initiating the next. We implement this benchmark using Locust
[11], a Python performance testing tool. Figure 1 shows that in this scenario,
Braid can process an average of 37 and a maximum of 41 requests per second
with no significant degradation as the datastsream grows. We do see, periodic,
sharp dips in request rate. These can be attributed to Braid periodically re-
validating the credentials presented by the client making the request. This
validation requires a remote call to the authorization service so adds
signficant time to what the client observes as the rate for that request.

Figure 2: Request and failure rates driven by multiple, concurrent clients

Our second micro-benchmark measures how the service behaves under concurrent
load from multiple clients each adding samples to a separate datastream. Once
again, we use Locust to perform this evaluation. Our results (Figure 2)
demonstrate that the service can deliver a peak of just over 500 requests per
second with a sustained mean rate of approximate 470 requests per second. We
again see the saw-tooth shape due to periodic requests for authenticating
validation credentials. As we approach 250 concurrent clients, we begin to see
errors due to clients timing out waiting for a result and at about 270 clients
the error rate increases dramatically as server-side connection pools begin to
fill up causing client connections to be dropped. We view these results as
inputs to help us set reasonable rate caps for our clients and guidance on
setting the size of the service we provision in AWS. We see that the
moderately sized service cluster we provisioned for this experiment would be
able to handle signficant load.

Figure 3: Metric performance across various datastream sizes

Finally, we want to know the time required to evaluate the various metric
operations (as enumerated in Section III-A2) for various size datastreams. We
need to verify that subject to the limits we impose, metric computations can
be completed in the time needed to perform adaptation for our flows. We
initialized Braid with datastreams of sizes ranging from 10 to 1,000,000
entries. We then selected a datastream and a metric operation combination to
be computed and repeated until each combination had been computed at least 10
times. We perform this random combination to get worst-case results, avoiding
caching or other optimizations in the database that might be caused by
repeating the same operation consecutively. Figure 3 shows the result of these
tests. The most important result is that, even for datastreams of size
1,000,000, any metric can be computed in no more than about 100 milliseconds.
Outliers do show up and we attribute these to network delay as a small change
in network latency has a signficant impact on transaction times measured in
the 10’s of milliseconds.

We attribute the generally good performance to Braid’s implementation the
metrics: each is computed using a single SQL query using a built-in function
or, in the case of First and Last a combination of SQL “ORDER BY” and “LIMIT”
clauses. Operations common to flows such as data transfer and computation
typically take a minimum of a few seconds, and often minutes or longer, so
these times to compute metrics would not have a significant effect on overall
flow execution times.

##  VI Experience with High Energy Diffusion Microscopy

High-energy X-ray diffraction methods allow scientists to non-destructively
analyze bulk metallic polycrystalline engineering materials. Specifically, the
HEDM [12] technique can extract three dimensional microstructure information
and grain-specific properties. HEDM produces digital scans of a material’s
evolving microstructure and its associated attributes, enabling scientists to
observe and understand how materials transform when external stimuli, such as
increased pressure load, are applied [13].

Automated computational methods have been developed that can provide near-
real-time feedback to scientists. In particular, unsupervised learning has
been used to identify anomalies within X-ray scans as they are collected. The
technique works as follows: prior to the experiment starting, scientists
identify a key condition, such as the pressure on the material, which
represents a _baseline_ upon which training will be performed to generate
cluster centers. Subsequent to the training, each scan, including those
collected prior to the baseline condition, are assessed relative to the
cluster centers to calculate an _anomaly score_. The domain scientists also
determine a threshold for the anomaly scores which would imply that the
microscopic indicator has occurred and thus the experiment can be concluded.

We encode the automatic anomaly detection process in two flows: a training
flow that is run only once, when the baseline scan is collected, and a second
which is run for every scan to compute its anomaly score. We start the anomaly
score flow as each scan is collected, but the computation of the score cannot
actually be performed until training on the baseline scan is complete.

We track the experiment through three phases:  1) scans have been collected,
but the baseline scan has not yet been collected, so anomaly score
calculations cannot yet be computed and the flows for these scans must wait
until the flow that performs the training has completed;  2) training has been
completed, and thus waiting flows can progress, using the model created by
training, and all incoming scans can have their anomaly scores computed
immediately; and  3) a sufficient quantity of scans have produced the desired
anomaly score, and thus the experiment can be considered complete and no
subsequent scans need to be taken.

These two flows use two datastreams to guide the experiment through the three
phases. A “coordination” datastream tracks the phase of the experiment. When
this datastream is created during administrative experiment setup, an initial
sample value of 1.0 is ingested establishing that the experiment is in phase
one. The second datastream receives samples for each of the anomaly scores
when they are computed and is then consulted by each flow to determine if the
desired anomaly score threshold has been reached and thus the experiment will
move to phase three indicating completion. The steps of the flows are as
follows:

Listing 3: Steps in the “Training” flow

⬇

Transfer scan data from instrument to computer

Perform Training

Add sample value 2.0 to coordination datastream

Listing 4: Steps in the “Anomaly Score Computation” flow

⬇

Transfer scan data from instrument to computer

Wait for policy on coordination datastream to be at least 2.0

Perform anomaly score computation

Sample computed anomaly score to anomaly score datastream

Evaluate policy to determine updated experiment phase based on aggregate
Anomaly score values

Sample policy output to the coordination Datastream

In our environment, data (scan) collection and computation take place at
different sites, requiring the first step of each flow to transfer data from
the instrument to the computation site. In the training flow, the subsequent
steps are to perform the training computation, which takes on the order of
minutes, and then publish the constant value 2.0 to the Coordination
datastream indicating the move to the second phase. The model generated by the
training is written to the filesystem at the computation site so that it can
be used by the anomaly score computation.

The anomaly score computation is similar to the example shown in Section IV.
It starts with a data transfer step, and then immediately uses a policy wait
step to hold until the coordination datastream has a value of at least 2.0
indicating that the training is complete. This ordering allows the transfer to
be completed even if training has not yet completed such that when training is
complete, the flow can continue immediately. The next steps are similar to the
example: computation, followed by sampling the result of the computation
followed by a policy evaluation over the recent samples. Here, the result of
the policy is published directly to the coordination datastream. The condition
for the policy is written by the domain scientists based on their knowledge of
the anomaly score calculation and the experiment being conducted. In the
experiment shown here, the policy is the same as shown in the example of
Section IV: “9 of the last 10 anomaly scores with a value greater the 0.95”.
In this case, the decision value for meeting this policy is set to 3.0 such
that when it is sampled to the coordination datastream, we indicate that the
experiment as a whole has reached phase 3: completion.

Figure 4: Progress of the HEDM experiment

Figure 4 shows the results of running the flows described above over a dataset
comprising 262 individual scan files. Each scan is assigned an integer _scan
index_. The scan index value increases as the experiment progresses; in this
experiment, from 246 to 751. However, the experimental process does not
increment the index value uniformly over time. That is, there is no set
interval between the assigned scan index values. Thus, while the horizontal
axis of the figure is the scan index, the values plotted are not uniformly
distributed across the horizontal access. We emulate a run of the experiment
by iterating over each of the scan files with a delay of 10 seconds between
each. For each scan file, we start an instance of the anomaly score
computation flow. When the baseline scan, which is determined by the domain
scientists to be index 318 in this dataset, is selected, the training flow is
started.

The shading of the figure indicates the phase of the experiment. The red
background is phase 1, prior to completion of the training flow. Phase 2 is
indicated by the yellow background, and phase 3 is the green background. The
blue line indicates the number of anomaly score computation flows running at
the time the flow for that scan index is started. We see that the number of
concurrent flows increases during phase 1 as no flows can complete until
training has completed. Upon entering phase 2, the waiting flows complete
quickly and the experiment reaches a relatively steady state in which between
five and eight flows are active until the end of the experiment. The red plot
presents the anomaly score calculated for each scan. At scan index 556 the
experiment completion policy is achieved and transition to phase 3 is signaled
by that flow sampling the 3 value to the coordination datastream. Phase 3
continues for an additional 81 scans, indices 557 through 751, which are
unneeded based on the anomaly score criteria being met indicating that the
desired transition of the material under test has been observed. Thus, these
81 scans, representing about 30% of the total taken during the experiment
could have been avoided. This, in turn, can free up the apparatus and the
scientists to move on to further experiments yielding more results in the same
amount time and resource allocation, a large potential benefit when access to
large scale scientific instruments is scarce.

##  VII Related Work

Braid provides capability similar to many other systems in the parallel and
distributed processing space while our focus particularly on coordination of
workflows via advanced policy also leads to differences. Other systems with
rich, high-level policy goals are commonly oriented around authorization
mechanisms, but we include some illustrative examples here. Thus, the related
work for Braid consists of wide area workflow control, policy specification
formats, policy management databases, and policy rule engines.

###  VII-A Workflow control

Wide-area, resilient distributed computing is a challenging control problem.
Adaptive approaches in a GRAM [14] or Condor [15] include FTSH [16], which
enabled scripted replication and retry in the presence of a range of failure
modes, including soft failures due to congestion. Handling exceptions in
workflows explicitly has been studied [17], an approach that requires
programmer attention. Replicated workflows [18] have continued to be studied
on commercial and open cloud infrastructure.

Our approach is based on previously developed technologies, which include
distributing computation via Globus Compute (funcX) [19], transferring files
via Globus Transfer [20, 21], managing data permissions via Globus Share [22],
and managining persistent identifiers via Globus Identifiers [23]. Relevant
alternative approaches include distributing work via SSH [24], but the high-
level logic of the workflow must be encoded somehow to allow high-level
policies to adapt and coordinate the workflow execution. Run-time adaptivity
was explored in Pegasus [25], an approach in which workflow schedules are
adapted over time as performance data on the underlying resources becomes
available.

###  VII-B Policy specification formats and languages

Policies must be specified by a user, human or otherwise. Multiple traditions
exist for policy specification, including highly collaborative, user friendly
specification models from the business operations space, and technical
specifications often based on existing structured file formats from the
database and data management space.

In Web services standards there has been an effort to create rich XML-based
languages defining policies that determine access control. Powerful
ontological languages such as the Web Ontology Language (OWL) [26] have
gathered support as they are more powerful in terms of semantic expressiveness
and provide simple reasoning capabilities.

eXtensible Access Control Markup Language (XACML) [27] is an OASIS standard
that defines an XML based architecture providing functionality to create,
evaluate and enforce policies. In addition to the access control policy
language it also defines a request/response language. XACML enables the use of
a decision point to store and evaluate user defined policies. XACML policies
are customizable and therefore facilitate user defined functions, making it
well suited for DRIVE as policies must be highly customizable and able to be
applied in different domains (provider types). The Policy Decision Point (PDP)
makes the decision whether or not to authorise a request and the Policy
Enforcement Point (PEP) enforces the policy decisions.

Enterprise Privacy Authorization Language (EPAL) [28] is a XML-based
functional language developed by IBM to represent enterprise security
policies. EPAL is very similar to XACML in most respects including
representation, functionality and policy enforcement model. A comparison of
features of the two found that EPAL supports a subset of almost all features
included in XACML [29].

Ponder [30] is a policy language designed specifically for management and
security of distributed systems. Ponder policies are described using an
extensible object oriented model which allows definition and multiple
instantiations of policies. The language however is designed to specify
general policies with little interdependence and the language itself is
somewhat complex.

###  VII-C Policy management databases

Following specification, policies must be encoded into persistent storage and
accessible via useful queries.

KAoS [31] uses OWL to represent policy. Designed for operation inside a Tomcat
web server, accepts policy creation specifications from users using web-based
tools. KAoS encodes the policies into OWL using Jena, hiding complexity and
simplifying the user experience. KAoS then distributes the policy to the
distributed JVMs that are acting as PDPs.

Relational databases have been conceptually extended to contain record-level
privacy constraints, for example by IBM [32]. This effort considered the
translating a P3P [33] policy from XML to an internal restriction
specification. The restrictions are capable of managing intersections and
unions. The approach was limited to managing the database itself, does not
seem to have an implementation, and does not seem to satisfy Codd’s rule 1
(representing all data in tables).

MirAIe [34] uses a shared cache such as Redis. This cache is responsible for
recording the state of various devices triggers and caching trigger-related
from sensors, avoiding recomputing the state of all triggers and devices.

###  VII-D Policy rule engines

Finally, policies must be evaluated at time of use. Generally, rule engines
are commonly used in business processing systems, which process some variation
of the Rete algorithm [35] to process large numbers of rules.

Rule based engines such as Jess [36] are also commonly used to express
policies in Java applications. Jess was designed to support expert systems by
evaluating user-defined rules over a user-defined data. Following the Rete
algorithm, rules may be triggered when their conditions are satisfied,
including the execution of Java code. Drools [37, 38] is a business rules
engine that also implements the Rete algorithm and integrates with an
enterprise application suite and JSR 94 [39]. Distributed Drools evaluation
has been performed in Spark [40] by placing the Drools engine inside an RDD
map function [41]. The IBM Operational Decision Manager [42] offers a visual
rule definition and execution programming model, and also integrates with
enterprise services.

Rei [43] is a Prolog-based policy engine that supports Prolog and RDF [44]
based descriptions, and evaluates them using a Java engine. Rei contains a
rich set of relationships among Subjects and their Rights, Obligations, and
Delegations.

The MirAIe rule engine [34] is capable of responding to sensors in an Internet
of things context. The outputs can include calls to web services or messages
issued via Kafka [45]. MirAIe includes a horizontally scalable trigger
processing service, deployed in Kubernetes [46], that is reliable through the
use of reliable messages and a replicated underlying trigger cache.

Many existing systems contain embedded rule engines that serve the purpose of
that system. iRODS [47, 48] is a storage management system that enables access
to distributed storage resources through a partially centralized service.
iRODS uses a rules-based paradigm for self-management, including the expected
user access rules for access control, and also rules for data placement,
migration, parallel access, and consistency levels. PolicyCop [49] monitors
network traffic and validates it against rules specified by the user in a
database; rules violations may trigger changes in the underlying Software
Defined Network (SDN) being managed by the system.

##  VIII Conclusions

As instrument-based science increasingly relies on computing and data
management services to process the output from these devices, reliable,
scaleable computing paradigms such as flows, and fleets of concurrent flows
become essential to completing these experiments. Further, as the fleets
themselves grow and the flows become more complex, we have discovered that
additional services are needed to steer these experiments toward their desired
results. The Braid service provides the capabilities needed to add steps into
these flows, building on measurements both within the fleet of flows and from
their surrounding environments, such that the flows can make these dynamic
adaptations. Braid has been shown to scale to the degree necessary to support
these many flows and the abstractions and service interfaces have been used to
create flows not previously possible which improve the efficiency of existing
scientific processes. We operate Braid as an always-on, cloud hosted service
continuously available for production usage. We anticipate that as the variety
of experiments running through flows expands we will continue to encounter
more opportunities to support these flows with additional services with the
continued goal of increasing the rate and reliability with which these types
of data-driven experiments can be completed.

## References

  * [1] R. Vescovi, R. Chard, N. D. Saint, B. Blaiszik, J. Pruyne, T. Bicer, A. Lavens, Z. Liu, M. E. Papka, S. Narayanan, et al., “Linking scientific instruments and computation: Patterns, technologies, and experiences,” Patterns, vol. 3, no. 10, p. 100606, 2022.
  * [2] R. Chard, J. Pruyne, K. McKee, J. Bryan, B. Raumann, R. Ananthakrishnan, K. Chard, and I. T. Foster, “Globus automation services: Research process automation across the space–time continuum,” Future Generation Computer Systems, vol. 142, pp. 393–409, 2023.
  * [3] S. Tuecke, R. Ananthakrishnan, K. Chard, M. Lidman, B. McCollam, S. Rosen, and I. Foster, “Globus Auth: A research identity and access management platform,” in 12th IEEE International Conference on e-Science, pp. 203–212, Oct 2016.
  * [4] D. Hardt, “OAuth 2.0 authorization framework specification,” 2012.  http://tools.ietf.org/html/rfc6749.
  * [5] K. Chard, M. Lidman, B. McCollam, J. Bryan, R. Ananthakrishnan, S. Tuecke, and I. Foster, “Globus Nexus: A platform-as-a-service provider of research identity, profile, and group management,” Future Generation Computer Systems, vol. 56, pp. 571–583, 2016.
  * [6] “Amazon States Language.”  https://states-language.net/. Accessed September 2023.
  * [7] “Fully Managed Container Solution – Amazon Elastic Container Service (Amazon ECS) - Amazon Web Services.”  https://aws.amazon.com/ecs. Accessed September 2023.
  * [8] “Relational Database – Amazon Aurora MySQL PostgreSQL – AWS.”  https://aws.amazon.com/rds/aurora/. Accessed September 2023.
  * [9] “PostgreSQL: The world’s most advanced open source database.”  https://www.postgresql.org. Accessed September 2023.
  * [10] “FastAPI.”  https://fastapi.tiangolo.com/. Accessed September 2023.
  * [11] “Locust - A modern load testing framework.”  https://locust.io/. Accessed September 2023.
  * [12] R. Pokharel, “Overview of high-energy x-ray diffraction microscopy (HEDM) for mesoscale material characterization in three-dimensions,” in Materials Discovery and Design, pp. 167–201, Springer International Publishing, 2018.
  * [13] D. Naragani, M. D. Sangid, P. A. Shade, J. C. Schuren, H. Sharma, J.-S. Park, P. Kenesei, J. V. Bernier, T. J. Turner, and I. Parr, “Investigation of fatigue crack initiation from a non-metallic inclusion via high energy x-ray diffraction microscopy,” Acta Materialia, vol. 137, pp. 71–84, 9 2017\.
  * [14] K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin, W. Smith, and S. Tuecke, “A resource management architecture for metacomputing systems,” Lecture Notes in Computer Science, vol. 1459, 1998.
  * [15] D. Thain, T. Tannenbaum, and M. Livny, “Distributed computing in practice: The Condor experience,” Concurrency and Computation: Practice and Experience, vol. 17, no. 2-4, 2005.
  * [16] D. Thain and M. Livny, “The ethernet approach to grid computing,” in Proc. HPDC, 2003.
  * [17] N. Russell, W. van der Aalst, and A. ter Hofstede, “Workflow exception patterns,” in Advanced Information Systems Engineering (E. Dubois and K. Pohl, eds.), (Berlin, Heidelberg), pp. 288–302, Springer Berlin Heidelberg, 2006.
  * [18] D. R. Schäfer, K. Rothermel, and M. A. Tariq, “Replication schemes for highly available workflow engines,” IEEE Transactions on Services Computing, vol. 14, no. 2, pp. 559–573, 2021.
  * [19] R. Chard, Y. Babuji, Z. Li, T. Skluzacek, A. Woodard, B. Blaiszik, I. Foster, and K. Chard, “FuncX: A federated function serving fabric for science,” in 29th International Symposium on High-Performance Parallel and Distributed Computing, (New York, NY, USA), p. 65–76, Association for Computing Machinery, 2020.
  * [20] B. Allen, J. Bresnahan, L. Childers, I. Foster, G. Kandaswamy, R. Kettimuthu, J. Kordas, M. Link, S. Martin, K. Pickett, and S. Tuecke, “Software as a service for data scientists,” Communications of the ACM, vol. 55, p. 81–88, feb 2012.
  * [21] Z. Liu, R. Kettimuthu, J. Chung, R. Ananthakrishnan, M. Link, and I. Foster, “Design and evaluation of a simple data interface for efficient data transfer across diverse storage,” ACM Transactions on Modeling and Performance Evaluation of Computing Systems, vol. 6, no. 1, pp. 1–25, 2021.
  * [22] K. Chard, S. Tuecke, and I. Foster, “Efficient and secure transfer, synchronization, and sharing of big data,” IEEE Cloud Computing, vol. 1, no. 3, pp. 46–55, 2014.
  * [23] R. Ananthakrishnan, K. Chard, M. D’Arcy, I. Foster, C. Kesselman, B. McCollam, J. Pruyne, P. Rocca-Serra, R. Schuler, and R. Wagner, “An open ecosystem for pervasive use of persistent identifiers,” in Practice and Experience in Advanced Research Computing, p. 99–105, ACM, 2020.
  * [24] J. Alt, R. Ananthakrishnan, K. Chard, R. Chard, I. Foster, L. Liming, and S. Tuecke, “OAuth SSH with Globus Auth,” in Practice and Experience in Advanced Research Computing, pp. 34–40, ACM, 2020.
  * [25] K. Lee, N. W. Paton, R. Sakellariou, E. Deelman, A. A. A. Fernandes, and G. Mehta, “Adaptive workflow processing and execution in pegasus,” Concurrency and Computation: Practice and Experience, vol. 21, no. 16, pp. 1965–1981, 2009.
  * [26] W3C, “OWL 2 Web Ontology Language Document Overview (Second Edition),” 2012\.
  * [27] D. Ferraiolo, R. Chandramouli, V. Hu, and R. Kuhn, “A comparison of Attribute Based Access Control (ABAC) standards for data service applications,” NIST Special Publication, vol. 800, no. 2016, p. 178, 2016\.
  * [28] P. Ashley, S. Hada, G. Karjoth, C. Powers, and M. Schunter, “Enterprise Privacy Authorization Language (EPAL),” IBM Research, vol. 30, p. 31, 2003.
  * [29] A. H. Anderson, “A comparison of two privacy policy languages: EPAL and XACML,” in Proceedings of the 3rd ACM workshop on Secure Web Services, pp. 53–60, 2006.
  * [30] N. Damianou, N. Dulay, E. Lupu, and M. Sloman, “The Ponder policy specification language,” in Policies for Distributed Systems and Networks: International Workshop, POLICY 2001 Bristol, UK, January 29–31, 2001 Proceedings, pp. 18–38, Springer, 2001.
  * [31] M. Johnson, P. Chang, R. Jeffers, J. M. Bradshaw, V. W. Soo, M. R. Breedy, L. Bunch, S. Kulkarni, J. Lott, N. Suri, et al., “KAoS semantic policy and domain services: An application of DAML to web services-based grid architectures,” in Proceedings of the AAMAS, vol. 3, 2003.
  * [32] R. Agrawal, P. Bird, T. Grandison, J. Kiernan, S. Logan, and W. Rjaibi, “Extending relational database systems to automatically enforce privacy policies,” in 21st International Conference on Data Engineering (ICDE’05), pp. 1013–1022, 2005.
  * [33] L. Cranor, M. Langheinrich, M. Marchiori, M. Presler-Marshall, and J. Reagle, “The Platform for Privacy Preferences 1.0 (P3P 1.0) specification,” W3C recommendation, vol. 16, 2002.
  * [34] N. Nayan, “The design of a distributed rule engine framework.” DZone, 2023.
  * [35] C. L. Forgy, “Rete: A fast algorithm for the many pattern/many object pattern match problem,” Artificial Intelligence, vol. 19, no. 1, pp. 17–37, 1982\.
  * [36] E. J. Friedman-Hill, “Jess, the Java Expert System Shell,” tech. rep., Sandia Labs., Livermore, CA (United States), 1997.
  * [37] P. Browne, JBoss Drools business rules.  Packt Publishing Ltd, 2009.
  * [38] “Drools.” http://www.jboss.org/drools, 2015.
  * [39] “JSR 94: Java Rule Engine API,” 2004.  https://www.jcp.org/en/jsr/detail?id=94.
  * [40] M. Zaharia, N. M. M. Chowdhury, M. Franklin, S. Shenker, and I. Stoica, “Spark: Cluster computing with working sets,” Tech. Rep. UCB/EECS-2010-53, EECS Department, University of California, Berkeley, May 2010.
  * [41] S. Pujari, “Apache Spark integration with DROOLS.” Medium, 2019.  https://medium.com/@saikrishna_55717/apache-spark-integration-with-drools-bb21aa706f9b.
  * [42] IBM, “IBM Operational Decision Manager 8.12.0 manual,” 2023.
  * [43] L. Kagal et al., “Rei: A policy language for the me-centric project.” HP Laboratories Palo Alto report, 2002.
  * [44] D. Brickley, R. V. Guha, and A. Layman, “Resource Description Framework (RDF) schema specification,” tech. rep., Technical report, W3C, 1999. W3C Proposed Recommendation., 1998.
  * [45] N. Garg, Apache Kafka.  Packt Publishing, 2013.
  * [46] K. Hightower, B. Burns, and J. Beda, Kubernetes: Up and running dive into the future of infrastructure.  O’Reilly Media, Inc., 1st ed., 2017.
  * [47] A. Rajasekar, M. Wan, R. Moore, and W. Schroeder, “A prototype rule-based distributed data management system,” in Workshop on Next Generation Distributed Data Management, 2006.
  * [48] R. W. Moore and A. Rajasekar, “Rule-based distributed data management,” in IEEE/ACM International Conference on Grid Computing, 2007.
  * [49] M. F. Bari, S. R. Chowdhury, R. Ahmed, and R. Boutaba, “PolicyCop: An autonomic QoS policy enforcement framework for software defined networks,” in 2013 IEEE SDN for Future Networks and Services (SDN4FNS), pp. 1–7, 2013\.