# GALOIS: A Hybrid and Platform-Agnostic
Stream Processing Architecture

Tarek Stolz<EMAIL_ADDRESS>0009-0000-5735-4318 RWTH Aachen
UniversityAhornstrasse 55AachenGermany52074 ,  István Koren<EMAIL_ADDRESS>aachen.de 0000-0003-1350-6732 Chair of Process and Data Science,
RWTH Aachen UniversityAhornstrasse 55AachenGermany52074 and  Liam Tirpitz,
Sandra Geisler<EMAIL_ADDRESS>0000-0003-0049-8112 Data
Stream Management & Analysis,
RWTH Aachen UniversityAhornstrasse 55AachenGermany52074

(2023)

###### Abstract.

With the increasing prevalence of IoT environments, the demand for processing
massive distributed data streams has become a critical challenge. Data Stream
Processing on the Edge (DSPoE) systems have emerged as a solution to address
this challenge, but they often struggle to cope with the heterogeneity of
hardware and platforms. To address this issue, we propose a new hybrid DSPoE
architecture named GALOIS, which is based on WebAssembly (Wasm) and is
hardware-, platform-, and language-agnostic. GALOIS employs a multi-layered
approach that combines P2P and master-worker concepts for communication
between components. We present experimental results showing that operators
executed in Wasm outperform those in Docker in terms of energy and CPU
consumption, making it a promising option for streaming operators in DSPoE. We
therefore expect Wasm-based solutions to significantly improve the performance
and resilience of DSPoE systems.

Edge Processing, Data Stream Processing

††journalyear: 2023††copyright: acmlicensed††price: 15.00††doi:
10.1145/3579142.3594287††isbn: 979-8-4007-0093-4/23/06††conference: The
International Workshop on Big Data in Emergent Distributed Environments; June
18, 2023; Seattle, WA, USA††booktitle: The International Workshop on Big Data
in Emergent Distributed Environments (BiDEDE ’23), June 18, 2023, Seattle, WA,
USA††ccs: Information systems Stream management

##  1\. Introduction

Modern manufacturing environments produce tremendous volumes of data at very
high frequencies. For example, in fine blanking (an industrial stamping
process) a single production line can produce data up to 6 GBit/s (Glebke et
al., 2019). In our vision of an Internet of Production (Brauner et al., 2022),
manifested by a large research cluster of excellence, the processing and
sharing of high-frequency machine data across organizations is fundamental to
enable large-scale near real-time analysis and fuel digital shadows to
facilitate smart manufacturing applications. However, the high degree of
heterogeneity of machines, programmable logic controllers, and network
connectivity creates a challenging environment for data processing. In
addition, the increased number of variants within production lines and
ultimately also the need to react quickly to incidents in light of
sustainability and resilience efforts demand fast and adaptive data stream
management solutions.

Mature distributed stream processing (DSP) architectures, such as Apache Storm
or Twitter Heron, are available for several years now and have proven to
handle high amounts of streaming data. They were designed to exploit the rich
resources of nodes in data centers on-site or in the cloud, distributing the
workload to homogeneously equipped servers. Transmitting the data of multiple
production lines to a central data center leads to network congestion and
intolerable latencies for near real-time applications. Edge processing
architectures move parts of the data processing (operators) to the edge and
the fog, close to the data sources (Cao et al., 2020; Varghese et al., 2016).
For edge environments, many challenges arise which are opposed to those in the
cloud (Varghese et al., 2016). For example, insecure, unstable, limited
network connections and failing or moving devices create a highly dynamic
environment for data processing. Further, edge devices have a high degree of
heterogeneity in, e.g., hardware or operating systems, and often have limited
resources and battery life. Distributed stream processing on the edge (DSPoE)
systems are designed to handle these issues. However, operators in DSPoEs are
often implemented language- or hardware-dependent which makes these systems
inflexible as well as hard to maintain and extend. Additionally, DSPoEs
usually rely on a centralized management component leading to high
communication overhead.

To address these challenges, we introduce GALOIS, a hybrid and platform-
agnostic stream processing architecture. Its architecture is multi-layered to
provide flexibility and distribute workload efficiently, employing centralized
as well as decentralized management depending on the layer. Re-optimizations
of the query execution plan are propagated from lower to higher levels to
reduce as much communication overhead as possible with a central, overarching
master node. Furthermore, we use WebAssembly (Wasm), a new byte code format
that can be run on diverse hardware, to implement operators in a hardware-,
platform-, and language-independent manner, providing the foundation of
GALOIS, which we will discuss in more detail in the following.

The paper is organized as follows. In Section 2, firstly, the requirements for
a hybrid and hardware-agnostic DSPoE are analyzed. Based on these
requirements, we present the design of the GALOIS architecture in Section 3.
In Section 4 a first prototype implementing this concept is described, which
is evaluated in Section 5. Finally, in Section 6, we discuss the presented
architecture in light of the current state of the art and draw conclusions in
Section 7.

##  2\. Requirements

In the following, we outline the key requirements that a DSPoE system should
meet to effectively operate within highly dynamic environments, accommodating
massive data streams and heterogeneous devices at the edge. These requirements
have been derived through a thorough analysis of edge environment
characteristics in manufacturing, as well as by examining the necessary
attributes and addressing unresolved challenges in existing DSPoE systems.

Platform-agnostic operator execution (R1): To accommodate the diversity of
edge environments, stream operators should be designed for execution
irrespective of hardware, language, and platform constraints.
Native edge streaming and processing (R2): To enable fast response times, the
data should be streamed directly point-to-point between processing edge
devices without any intermediaries. Furthermore, the execution of operators
should take place on the edge level.
Network-aware management and administration (R3): The edge network’s topology
and properties, such as link quality and device distance, should be considered
during query processing to minimize network-induced latencies and
backpressure.
Global logical optimization (R4): Logical query optimization should be done on
a global level based on heuristics, as it will influence the number and types
of operators which are subsequently be distributed to edge nodes in operator
placement.
Local physical optimization (R5): To leverage the latest, contextually
relevant information for query optimization and decrease communication
overhead, physical query optimization should occur at the same level as
operator execution, thereby enhancing layer autonomy and modularity.
Spatial organization in cluster units (R6): The system should take into
account spatial constraints when deploying to and managing edge devices. The
system should prefer to deploy adjacent operators in dense accumulation points
of edge devices (cluster units) to establish the shortest possible
communication paths.
Decentralized query reconfiguration (R7): If query plan adjustments are
necessary, these should be carried out in a decentralized and independent
manner without pausing the complete system.
Administrative escalation model (R8): Higher-level administrative units in the
architecture should only intervene when re-optimization at lower levels is
unfeasible.
Preventive fault tolerance (R9): Resilient fault tolerance is important to
elevate the robustness of a query. In the event of a client failure, a query
should be able to deliver results. This can be achieved by increasing the
robustness of a query through more selective operator placement and providing
multiple paths between operators (O’Keeffe et al., 2018; Chao and Stoleru,
2020). These core requirements inform the design of our architecture.

#### Resulting design decisions

As edge environments can be highly dynamic, the metrics used for operator
placement in DSPoEs may already be outdated when the query plan has been
deployed (Zeuch et al., 2020; Liu et al., 2021). Hence, the development of
DSPoE systems is tending more and more towards a decentralized deployment of
queries and functionalities (Dustdar and Murturi, 2021; Zeuch et al., 2020).
As a result, Pinchao et al. (Liu et al., 2021) organize edge devices within a
P2P network and perform query optimization not at deployment time but at
runtime. Because the query is optimized incrementally, there is a lower risk
that query execution is no longer optimal as time progresses (R7). However,
many P2P networks abstract away from the underlying network topology, which
can lead to non-optimal operator placement (Kunz et al., 2020; Ruslan et al.,
2019). In order to achieve low latencies, it is important that operators are
positioned as close as possible to the sources and adjacent operators are also
positioned in near vicinity to each other. Therefore, _cluster-based methods_
for organizing the underlying physical devices on the lowest level seem to be
a viable solution to this problem (R6, R3) (Kunz et al., 2020; Wang and Peh,
2014; Ning et al., 2015; Chao and Stoleru, 2020; Chao et al., 2018).
In this context, Mobile Ad-hoc Networks (MANETs) are attractive, as they
enable cluster computing on the edge (Kunz et al., 2020). MobiStream (Wang and
Peh, 2014) defines fixed locations for clusters in MANETs, but does not
accommodate dynamically forming, disappearing, and reorganizing clusters. For
this reason, clusters should support mobility and be able to be registered
autonomously (R6). For GALOIS , we opted to organize spatially separated
clusters in regions for scalability and flexibility, similar to Mobile Storm’s
concept (Ning et al., 2015) (R6). Furthermore, additional regions can be added
and thus a spatial scaling as in SpanEdge is enabled (R4,R8) (Sajjad et al.,
2016). Cluster-to-cluster communication and region-to-region communication
overhead should be kept as low as possible and implemented in a P2P fashion to
provide high flexibility and autonomy. Therefore, query operator execution
should be done on cluster and edge device level (R2) (Kunz et al., 2020).
During query runtime, continuous query optimization shall be done in the
cluster which should be as autonomous as possible and only be escalated
upwards to managing nodes if necessary (e.g., due to unresolvable overload
situations) to reduce the administrative overhead (R8) (Zeuch et al., 2020).
The query’s physical and logical optimization should be divided between local
and global layers, ensuring global instances are not burdened with complete
knowledge of the underlying edge devices and operators. Conversely, physical
optimization, heavily reliant on dynamic metrics, should be performed as close
as possible to the execution site (R6,R7).
To maximize fault-tolerance, GALOIS will use a probability model superior to
checkpoint- or replicant-based schemes (Chao and Stoleru, 2020) and select
only particularly fail-safe nodes in the operator placement (R9) (O’Keeffe et
al., 2018) making the architecture more robust.
Further, the architecture is designed to be extensible in terms of different
query languages similar to Calcite or Governor (Begoli et al., 2018; Chaudhary
et al., 2020). Finally, to execute operators on a heterogeneous system
landscape, operators should be executable independent of hardware, platform,
and language. All DSPoE we are aware of to date are language-dependent (R1)
(Grulich et al., 2020; Liu et al., 2021; Ning et al., 2015), which we address
with the architecture of GALOIS.

##  3\. Concept

Based on the requirements and design decisions explained in the previous
section, we conceptualized a system architecture, which is described in detail
in the following.

Figure 1. The GALOIS architecture

Figure 1 shows an overview of the various components in GALOIS. The system is
hierarchically structured, employing four levels of communication (Zeuch et
al., 2020) (R6). The lowest level is the _Cluster Level_ where nodes are
organized in clusters and transfer tuples in data streams. Clusters represent
accumulation points of nodes that have a physical proximity which is exploited
(R2). Within the cluster, the data streams are exchanged between the nodes. In
Figure 1, the clusters are represented as circles and network connections as
solid lines between the nodes, which are shown as black dots in the clusters.
Cluster-to-cluster data streaming is depicted as solid lines between the
clusters and is performed via gateways. The yellow squares represent the
_Cluster Managers_ (CM) which are interconnected by a P2P network (shown as a
dashed ellipse) and, if necessary, are communicating with the _Region Manager_
(RM) of their region (shown as a triangle). To increase scalability and build
local, independent subsystems, each cluster is assigned to a RM. Regions can
be defined based on the needs of the application, e.g., by grouping
geographically close clusters. The RM and GM are organized in a master (GM)
worker (RM) layer.

The _Global Manager_ (GM, depicted as pentagon) is primarily responsible for
three tasks: (1) registration of logical data sources (schema), (2) receipt of
the logical query plan, and (3) logical query optimization (R4). The
registration of a schema can be done in two ways: (1) either a human user
registers the schema and the corresponding data sources or (2) the data
sources act autonomously through corresponding interfaces and register
themselves and their schema. There are several formal query languages that can
be used to register and describe a query. Similar to Apache Calcite or
NebulaStream the GM should be able to support multiple languages (Begoli et
al., 2018; Zeuch et al., 2020). The optimizer in the GM can optimize the
logical query plan utilizing an optimization strategy. Unlike other systems,
no physical optimization is performed in the GM and a strict separation
between logical and physical optimization is enforced. A successfully deployed
query is stored in the _Query Manager_ of the GM to enable multi-query
optimization. Furthermore, the _Query Manager_ can provide the user with
information about the currently running queries and their state. The
optimization strategy should be easily interchangeable to allow different
optimization strategies, comparable to the Governor system (Chaudhary et al.,
2020). After the query has been optimized, it is divided into subqueries and
sent to the _Region Managers_ (RMs) by the _Distributor_. In addition to the
subqueries, information important for interregional communication is provided
to the RMs to enable interregional data transfer between nodes in different
regions. The splitting into subqueries is done using metadata which is stored
in the _Metadata Store_. These comprise region-specific metadata, e.g., the
source nodes in a region, region performance statistics, such as free resource
capacities, running subqueries, or key metrics such as the average tuple
throughput. The _Orchestrator_ monitors the region topology and the start of
new regions, which can be dynamically added or deleted.
_Region Managers (RMs)_ are introduced as the next layer of abstraction and
enable scalability and locality, by independently managing groups of clusters.
A region can be specified geographically or on the basis of latency zones, for
example, in order to identify the most latency-optimal regions, similar to
SpanEdge (Sajjad et al., 2016) (R3). It is assumed that latency for
geographically distant points is significantly higher than for spatially
closer nodes. The notion of proximity can be defined on the implementation
level and depends on the application. Therefore, it is important that the
subqueries are chosen such that there is only as much cross-region
communication as necessary. The core task of the RM is to distribute the
subqueries to the _Cluster Manager_ based on summary statistics, e.g.,
cluster-specific operator throughput or average queue size. The RM distributes
the logical subqueries to the _Cluster Manager_ accordingly and manages the
assignment in the local query manager for possible reconfigurations (R7). In
addition, the RM also serves as a bootstrap node for clusters that appear in
the region providing the new _Cluster Manager_ with all necessary information
to connect to the P2P network.
The _Cluster Managers_ (CM) are organized in a P2P network. The most important
tasks of the CM are operator placement, collection of metadata, monitoring of
the GALOIS clients, and communication with other CM within a P2P network to
distribute workloads at runtime. Based on the gathered metrics, subquery
reoptimization is organized autonomously by the CMs within the P2P network and
thereby metadata exchange is reduced (R8). Furthermore, the P2P network should
ensure that new clusters can be set up mobile (e.g., for vehicles or
smartphones) and dynamically over time. Node failures in the cluster should
not stop the processing of the entire query. Thus, delays like in MobiStream
(stop the hole query on failure) have to be avoided (Wang and Peh, 2014). At
the lowest level of abstraction GALOIS clients are located consisting of edge
devices with the DSPoE client installed. Each node is assigned to a cluster by
node discovery. If there is no CM nearby, an isolated node can take over the
CM role building a new cluster. The new cluster remains inactive until a
sufficiently large number of nodes are registered in the cluster (the cluster
size is configurable to reduce management overhead).

Beside being the CM (h), nodes can take six additional roles in a cluster:
Sleeper (s), Producer (p), Consumer (c), Replica (r), Gateway (g), and Sink
(z). This role scheme allows for probabilistic models which assign roles
according to reliability for fault tolerance (Chao and Stoleru, 2020). Nodes
which are not the CM are initialized as s nodes. Sleepers do not yet have a
differentiated role and are available to the CM as a resource. As soon as the
operators of a query are distributed corresponding nodes get the producer (p)
role assigned and which are the source of a data stream. All subsequent
operators in the subquery are consumers (c) which are processing the data
stream incrementally. To implement a fault tolerance scheme nodes get the role
r (replica) assigned, which are statistically particularly fail-safe. These
nodes regularly check the c node assigned to them and take over its
functionality in the event of a failure. In order to transfer data between
clusters, cluster-to-cluster communication is required; this functionality is
provided by g nodes. They act as gateways between the clusters. The z token is
assigned to sink nodes by the CM, which provide APIs for data consumption by
users or applications. These node types and the general architecture are
realized in a prototypical implementation.

##  4\. Prototype

In order to realize the requirement of platform independence in the prototype,
a form of abstraction is needed. Common solutions to enable platform
independence are virtual machines or Docker containers. Since the stream
operators should be executed on edge devices, we target an execution
environment requiring minimal resources and startup effort. With the binary
instruction format WebAssembly (Wasm)111https://webassembly.org, especially
suited for serverless scenarios (Gackstatter et al., 2022), operators can be
executed as functions within the Wasm runtime environment. In literature, Wasm
is regarded as a potential alternative to Docker, especially in edge
scenarios, as it proved to be more memory-efficient than Docker (Gackstatter
et al., 2022; Hampau et al., 2022). Hence, we implement the stream operators
in our system targeting Wasm to make their execution platform-, language-, and
hardware-agnostic (R1). We chose Rust as the implementation language also for
the client and the managers, due to its native Wasm support and integration,
as well as its ability to produce small binaries. However, any other language
could be chosen for the operators and the choice is independent of the client
and managers’ implementations.

The implementation of the first prototype followed a bottom up approach,
concentrating on the development of operators in Wasm as a proof-of-concept.
To test the operator execution, only one cluster with multiple nodes is
created. Data transfer between nodes is implemented using Kafka, but will be
replaced in the next step by efficient inter-node communication. In the
prototype, a single server component represents GM, RM, and CM in unity (GRC
server). For scaling, the components can be separated later. Currently, the
GRC server comprises modules for query, schema, and node management, as well
as physical query plan creation and a simple operator placement (R5).
Additionally, Data Fusion is integrated to execute logical query optimization.
The components will be complemented in the future with additional features,
such as cluster and resource management and fault tolerance protocols
including backpressure mechanisms suited for edge environments (R9). The GRC
server communicates with the nodes via server-sent events (SSE) including the
nodeID and the corresponding command. Commands are also used to deploy
operators to nodes after the GRC server has completed the operator placement.
The GRC server is deployed as a Docker container in a Kubernetes cluster and
the workers run on Raspberry Pis model B Rev 1.5. Figure 2 provides an
overview of the most important components.

Figure 2. Overview of the prototype and its components

##  5\. Evaluation

In the preliminary phase of our research, we conducted an assessment of the
prototype system, with a primary focus on comparing the performance of
implementing operators in Wasm to that of Docker containerization. This
evaluation aimed to explore the differences in latency, energy consumption,
throughput, and CPU load between the two approaches. Our hypothesis posited
that Wasm, due to its near-native exploitation of hardware characteristics and
reduced initialization overhead, would exhibit superior efficiency in these
performance metrics. The evaluation utilized the ELEC dataset (Harries, 1999),
comprising 45,312 tuples on half-hourly electricity prices from New South
Wales, Australia. It covers a period of 942 days and contains the following
attributes: Date, Day, Period, NSWprice, NSWdemand, VICprice, VICdemand,
transfer. The evaluation setup is shown in Figure 3.

Figure 3. Evaluation Setup

The setup comprises two Raspberry Pis 4, Model B Rev 1.5, running Linux
raspberrypi 5.15.76-v8+ on 64-bit basis. They represent the edge devices in
one cluster to which operators can be distributed. Besides the Raspberry Pis,
a Windows 10 computer was used, which acted as the producer, creating a data
stream from the aforementioned dataset, running a local, dockerized Kafka
instance. Further two DollaTek UM25C USB power meters were used to record the
power consumption of the Raspberry Pis. The power meter were connected to two
separate smartphones as the monitoring software only allows one Bluetooth
connection to a power meter at a time. The GRC server was deployed to the
Kubernetes infrastructure of the Internet of Production (IoP) located in a
different network. A continuous query, including a projection onto the date
and period attributes and 18 selection conditions (to increase query
complexity), was registered at the GRC server, which distributed the
selections to one Raspberry Pi and the projection to the second Raspberry Pi
to measure the processing costs for each operator individually. We simulated
varying system load, by generating tuples with different data rates at the
producer, specifically 125 T/s (Tuples/second), 500 T/s, and 1000 T/s. For
each input rate, power consumption and CPU load were measured at the edge
devices, as well as throughput (T/s at each operator and for the complete
query), and query latency (time of a tuple between entering and leaving the
system).

###  5.1. Results

We present the results of the described experiments in the following, where
percentages indicate average values. Operator Processing Time The Operator
processing time (OPT) is the difference between the timestamp directly before
the execution of the operator and the timestamp afterwards. Our experiments
show, that for selection Wasm outperforms Docker for each of the input rates
with 9% less OPT.

#### Power Consumption

Energy consumption is recorded in Watts per second by the power meters at
operator execution. Figure 4 contains six boxplots showing the power
consumption of the selection execution. On the x-axis the boxplots are labeled
with the experiments - each combination of execution environment (Wasm or
Docker) and input rate.

Figure 4. Energy Consumption Comparison

The energy consumption for the Docker experiment was 5.72% higher than for
Wasm at the 1000 T/s, 6.56% percent higher than for Wasm at 500 T/s, and
10.09% percent higher than for Wasm at the 125 T/s input rate. CPU Load The
average CPU load was recorded as percentage of CPU Idle Time (CIT) during
tuple processing only, where higher values indicate a lower CPU load and vice
versa. The CPU load is considered as an indicator for the potential additional
workload for Docker virtualization. Figure 5 shows six boxplots also labeled
on the x-axis with the combination of execution environment and input rate. On
the y-axis the percentage of CIT during selection execution is plotted. The
CIT for the Wasm experiment was 11.36% percent higher than for Docker at 1000
T/s, 18.89% percent higher than for Docker at 500 T/s, and 23.05% percent
higher than for Docker at the 125 T/s input rate. The projection showed no
significant differences, only the standard deviation was smaller for all
measurements.

Figure 5. Idle CPU Time Comparison

###  5.2. Discussion

Our results support the assumption that Wasm’s processing time per tuple is
more efficient than Docker’s due to its lower CPU intensity. This efficiency
stems from several factors: Docker runs as a standalone process, managing its
environment and virtual network, while Wasm, embedded in Rust, only executes
operators. Despite expecting a more significant difference in selection, the
OPT remains relatively constant, possibly due to operational and CPU
limitations. Notably, Wasm’s OPT exhibits less fluctuation for both operators
compared to Docker, likely due to the latter’s additional CPU overhead.

As expected, Docker consumes more power due to containerization overhead.
Interestingly, the CPU and energy consumption difference between Docker and
Wasm decreases at higher input rates, with the largest gap at 125 T/s. Further
experiments are needed to investigate the relationship between these
differences and input rates. Our results align with Hampau et al. (Hampau et
al., 2022), as Wasm consumes less energy than Docker. The highest difference
at 125 T/s might result from Wasm’s better OPT and increased waiting time for
tuples, which vanishes at higher input rates due to backpressure.

The OPT, limited by CPU, could be reduced with multi-threading but at the cost
of increased energy consumption, presenting a tradeoff (Zeuch et al., 2020).
Our experiments aimed to evaluate Wasm’s feasibility as a streaming operator
library. Results indicate Wasm as a potential lightweight, hardware-
independent FaaS alternative to Docker for DSPoE. We plan to extend the
operator library, including standard operators like joins and sophisticated
features like machine learning. In-depth experiments will assess complex
queries, operator placement, query reoptimization, and communication costs.
This will provide a comprehensive assessment of the proposed operator
network’s effectiveness and benefits for data streaming.

##  6\. Related Work

The DSPoE systems NebulaStream, MobiStream, DART, Apache Nifi, and MobileStorm
are related systems that enable stream processing employing concepts from edge
computing (Zeuch et al., 2020; Wang and Peh, 2014; Liu et al., 2021; Dautov et
al., 2017; Ning et al., 2015) and we analyzed them according our posed
requirements. Many of these systems, such as MobileStorm or Apache Nifi, use a
central administrative component, e.g., responsible for monitoring, query
distribution, client management, and fault handling, while the clients only
execute the operators. In contrast, DART uses a purely decentralized approach
and offloads many of the functions to the individual clients organized in a
P2P network. In GALOIS, we employ a hybrid model to combine the best of both
approaches, offering high scalability and flexibility. Almost all of the above
systems use the Java Virtual Machine (JVM) as execution environment for
platform- and hardware-agnostic operator execution. However, the JVM is not
language-independent. NebulaStream uses query compilation, i.e., queries are
distributed to the edge devices as generic query plans, where it is converted
into a C++, compiled, and executed (Grulich et al., 2020). None of the
discussed approaches are language-independent. Instead of relying on the JVM,
the online Wasm compiler presented by Groppe and Reimer (Groppe and Reimer,
2019) contributes towards language-independent query processing in networks of
internet browsers, but does not focus on data stream processing.

Cloud-based Data Stream Processors (CDSP) have been widely used and adapted as
the basis for DSPoE systems in the past. However, most DSPoE systems in this
area were developed in Java lacking hardware- or language-independent
execution of operators. CDSPs also miss a concept for fault handling that fits
the dynamics of the edge environment and they usually do not allow runtime
optimization of queries, since the performance in the cloud is assumed to be
stable. Due to the contradictory requirements of cloud and edge, it makes
sense to develop a DSPoE framework which does not to build on an existing
CDSP.

##  7\. Conclusion and Outlook

In this paper we present a new concept for a hybrid multi-layer DSPoE
architecture GALOIS. Its architecture facilitates cluster communication on the
edge using a P2P network concept enabling re-optimizations on a local level
and thereby reducing communication overhead. Also, nodes in a cluster can take
different roles, which will guide the operator placement. At the same time
GALOIS offers scalability by introducing a region management layer containing
several clusters and a global manager implemented in a master-worker fashion.
Opposed to existing DSPoEs, GALOIS is hardware-, platform, and language-
agnostic by using WebAssembly as execution environment for streaming operators
distributed to the edge. We showed in experiments on a prototypical
implementation of our architecture, that operator execution in Wasm is more
resource-efficient than in Docker in terms of Operator Processing Time and CPU
and energy consumption. The lower throughput observed in our experiments can
be attributed to the conservative use of the CPU, but this issue could be
resolved in the future by leveraging higher parallelism with multi-threading.
To summarize, Wasm represents a viable alternative to Docker for executing
operators in DSPoE environments. However, since Wasm is still in development,
it has some inherent limitations that need to be considered. For example, it
is currently not possible to pass Wasm complex data objects without additional
glue code. We plan to extend GALOIS by (1) expanding the Wasm stream operator
library to allow for more complex queries, (2) integrate a flexible QoS and
data quality monitoring to enable a dynamic re-optimization of queries, (3)
implement a full-fledged prototype including all hierarchy levels based on the
envisioned communication infrastructure also considering in-network
processing.

###### Acknowledgements.

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy - EXC-2023 Internet of
Production - 390621612.

## References

  * (1)
  * Begoli et al. (2018) Edmon Begoli, Jesús Camacho-Rodríguez, Julian Hyde, Michael J. Mior, and Daniel Lemire. 2018\.  Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources. In _Proceedings of the 2018 International Conference on Management of Data_ _(SIGMOD ’18)_. ACM, 221–230.
  * Brauner et al. (2022) Philipp Brauner et al. 2022\.  A Computer Science Perspective on Digital Transformation in Production.  _ACM Trans. on Internet of Things_ 3, 2 (2022), 1–32.
  * Cao et al. (2020) Keyan Cao, Yefan Liu, Gongjie Meng, and Qimeng Sun. 2020\.  An overview on edge computing research.  _IEEE Access_ 8 (2020), 85714–85728.
  * Chao and Stoleru (2020) Mengyuan Chao and Radu Stoleru. 2020.  R-MStorm: A Resilient Mobile Stream Processing System for Dynamic Edge Networks. In _2020 IEEE International Conference on Fog Computing (ICFC)_. IEEE, Sydney, NSW, Australia, 64–72.
  * Chao et al. (2018) Mengyuan Chao, Chen Yang, Yukun Zeng, and Radu Stoleru. 2018\.  F-MStorm: Feedback-Based Online Distributed Mobile Stream Processing. In _2018 IEEE/ACM Symposium on Edge Computing (SEC)_. IEEE, Seattle, WA, USA, 273–285.
  * Chaudhary et al. (2020) Ankit Chaudhary, Steffen Zeuch, and Volker Markl. 2020\.  Governor: Operator Placement for a Unified Fog-Cloud Environment. In _Proceedings of the 23rd International Conference on Extending Database Technology_. EDBT.
  * Dautov et al. (2017) Rustem Dautov et al. 2017\.  Pushing Intelligence to the Edge with a Stream Processing Architecture. In _2017 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData)_. 792–799.
  * Dustdar and Murturi (2021) Schahram Dustdar and Ilir Murturi. 2021.  Towards IoT Processes on the Edge.  In _Next-Gen Digital Services. A Retrospective and Roadmap for Service Computing of the Future_ , Marco Aiello, Athman Bouguettaya, Damian Andrew Tamburri, and Willem-Jan van den Heuvel (Eds.). Springer, Cham, 167–178.
  * Gackstatter et al. (2022) Philipp Gackstatter, Pantelis A. Frangoudis, and Schahram Dustdar. 2022.  Pushing Serverless to the Edge with WebAssembly Runtimes. In _2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid)_. 140–149.
  * Glebke et al. (2019) René Glebke, Martin Henze, Klaus Wehrle, Philipp Niemietz, Daniel Trauth, Patrick Mattfeld, and Thomas Bergs. 2019.  A Case for Integrated Data Processing in Large-Scale Cyber-Physical Systems. In _Hawaii International Conference on System Sciences 2019 (HICSS)_. 7252–7261.
  * Groppe and Reimer (2019) Sven Groppe and Niklas Reimer. 2019.  Code Generation for Big Data Processing in the Web using WebAssembly.  _Open Journal of Cloud Computing (OJCC)_ 6, 1 (2019), 1–15.
  * Grulich et al. (2020) Philipp M Grulich, Sebastian Breß, Steffen Zeuch, Jonas Traub, Janis von Bleichert, Zongxiong Chen, Tilmann Rabl, and Volker Markl. 2020.  Grizzly: Efficient Stream Processing Through Adaptive Query Compilation. In _Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD 2020)_. ACM.
  * Hampau et al. (2022) Raluca Maria Hampau, Maurits Kaptein, Robin van Emden, Thomas Rost, and Ivano Malavolta. 2022\.  An Empirical Study on the Performance and Energy Consumption of AI Containerization Strategies for Computer-Vision Tasks on the Edge. In _The International Conference on Evaluation and Assessment in Software Engineering 2022_ _(EASE 2022)_. ACM, New York, USA, 50–59.
  * Harries (1999) Michael Bonnell Harries. 1999\.  _Splice-2 Comparative Evaluation: Electricity Pricing_.  University of New South Wales, School of Computer Science and Engineering.
  * Kunz et al. (2020) Thomas Kunz, Silas Echegini, and Babak Esfandiari. 2020\.  A P2P Approach to Routing in Hierarchical MANETs.  _Communications and Network_ 12 (2020), 99–121.
  * Liu et al. (2021) Pinchao Liu, Dilma Da Silva, and Liting Hu. 2021.  DART: A Scalable and Adaptive Edge Stream Processing Engine. In _2021 USENIX Annual Technical Conference (USENIX ATC 21)_. USENIX Association, 239–252.
  * Ning et al. (2015) Qian Ning, Chien-An Chen, Radu Stoleru, and Congcong Chen. 2015\.  Mobile Storm: Distributed Real-time Stream Processing for Mobile Clouds. In _2015 IEEE 4th International Conference on Cloud Networking (CloudNet)_. IEEE, Niagara Falls, ON, Canada, 139–145.
  * O’Keeffe et al. (2018) Dan O’Keeffe, Theodoros Salonidis, and Peter Pietzuch. 2018\.  Frontier: Resilient Edge Processing for the Internet of Things.  _Proceedings of the VLDB Endowment_ 11, 10 (2018), 1178–1191.
  * Ruslan et al. (2019) Rafiza Ruslan, Ayu Shaqirra Mohd Zailani, Nurul Hidayah Mohd Zukri, Nur Khairani Kamarudin, Shamsul Jamel Elias, and R. Badlishah Ahmad. 2019\.  Routing performance of structured overlay in Distributed Hash Tables (DHT) for P2P.  _Bulletin of Electrical Engineering and Informatics_ 8, 2 (2019), 389–395.
  * Sajjad et al. (2016) Hooman Peiro Sajjad et al. 2016\.  SpanEdge: Towards Unifying Stream Processing over Central and Near-the-Edge Data Centers. In _2016 IEEE/ACM Symposium on Edge Computing (SEC)_. 168–178.
  * Varghese et al. (2016) Blesson Varghese, Nan Wang, Sakil Barbhuiya, Peter Kilpatrick, and Dimitrios S Nikolopoulos. 2016\.  Challenges and Opportunities in Edge Computing. In _2016 IEEE International Conference on Smart Cloud (SmartCloud)_. IEEE, 20–26.
  * Wang and Peh (2014) Huayong Wang and Li-Shiuan Peh. 2014.  MobiStreams: A Reliable Distributed Stream Processing System for Mobile Devices. In _2014 IEEE 28th International Parallel and Distributed Processing Symposium_. IEEE, Phoenix, AZ, USA, 51–60.
  * Zeuch et al. (2020) Steffen Zeuch et al. 2020\.  The NebulaStream Platform: Data and Application Management for the Internet of Things. In _10th Biennial Conference on Innovative Data Systems Research (CIDR ‘20)_. CIDR.


# Nonzero $\theta_{13}$ and CP Violation from Broken $\mu-\tau$ Symmetry

Asan Damanik111E-mal<EMAIL_ADDRESS>
Faculty of Science and Technology
Sanata Dharma University
Kampus III USD Paingan Maguwoharjo Sleman Yogyakarta
Indonesia

###### Abstract

Nonzero and relatively large of $\theta_{13}$ mixing angle has some
phenomenological consequences on neutrino physics beyond the standard model.
One of the consequences if the mixing angle $\theta_{13}\neq 0$ is the
possibility of the CP violation on the neutrino sector. In order to obtain
nonzero $\theta_{13}$ mixing angle, we break the neutrino mass matrix that
obey $\mu-\tau$ symmetry by introducing a complex parameter and determine the
Jarlskog invariant as a measure of CP violation existence. By using the
experimental data as input, we determine the Dirac phase $\delta$ as function
of mixing angle $\theta_{13}$.

Keywords: Nonzero $\theta_{13}$; CP Violation; Broken $\mu-\tau$ Symmetry
PACS No: 14.60.Pq

##  1 Introduction

Recently, there are two unsolved major problems related to neutrino physics
i.e. underlying symmetry of neutrino mass matrix and neutrino mixing matrix
when confronted to the experimental results. One of the interesting and
popular underlying symmetry for neutrino mass matrix is the $\mu-\tau$
symmetry. The $\mu-\tau$ symmetry reduces the number of parameters in
symmetric neutrino mass matrix from 6 parameters to 4 parameters. The neutrino
mass matrix with $\mu-\tau$ symmetry can also obtained from three well-known
neutrino mixing matrices i.e. tribimaximal (TBM), bimaximal (BM), dan
democratic (DC). But, the three well-known of neutrino mixing matrices predict
mixing angle $\theta_{13}=0$ and hence Dirac phase $\delta$ can be arbitrary.
The standard parametrization of neutrino mixing matrix ($V$) is given by:

$V=\bordermatrix{&&&\cr&c_{12}c_{13}&s_{12}c_{13}&s_{13}e^{-i\delta}\cr&-s_{12}c_{23}-c_{12}s_{23}s_{13}e^{i\delta}&c_{12}c_{23}-s_{12}s_{23}s_{13}e^{i\delta}&s_{23}c_{13}\cr&s_{12}s_{23}-c_{12}c_{23}s_{13}e^{i\delta}&-c_{12}s_{23}-s_{12}c_{23}s_{13}e^{i\delta}&c_{23}c_{13}}$
(1)

where $c_{ij}$ is the $\cos\theta_{ij}$, $s_{ij}$ is the $\sin\theta_{ij}$,
and $\theta_{ij}$ are the mixing angles. In the basis where the charged lepton
mass matrix is diagonal, the neutrino mass matrix $M_{\nu}$ can be
diagonalized by mixing matrix $V$ as follow:

$\displaystyle M_{\nu}=VMV^{T},$ (2)

where the diagonal neutrino mass matrix $M$ is given by:

$\displaystyle M=\bordermatrix{&&&\cr&m_{1}&0&0\cr&0&m_{2}&0\cr&0&0&m_{3}}.$
(3)

If we put the mixing angle $\theta_{13}=0$ and consequently $s_{13}=0$ and
$c_{13}=1$ in Eq. (1), then the neutrino mass matrix $M_{\nu}$ in Eq. (2)
read:

$\displaystyle M_{\nu}=\bordermatrix{&&&\cr&P&Q&Q\cr&Q&R&S\cr&Q&S&R},$ (4)

where:

$\displaystyle P=m_{1}c_{12}^{2}+m_{2}s_{12}^{2},$ (5) $\displaystyle
Q=(m_{2}-m_{1})c_{12}s_{12}c_{23},$ (6) $\displaystyle
R=(m_{1}s_{12}^{2}+m_{2}c_{12}^{2})c_{23}^{2}+m_{3}s_{23}^{2},$ (7)
$\displaystyle S=(-m_{1}s_{12}^{2}-m_{2}c_{12}^{2}+m_{3})s_{23}c_{23}.$ (8)

It is apparent from neutrino mass matrix ($M_{\nu}$) in Eq. (4) that the
neutrino mass matrix deduced from neutrino mixing matrix with assumption
$\theta_{13}=0$ is $\mu-\tau$ symmetry. But, as dictated from the experimental
results, the mixing angle $\theta_{13}\neq 0$ and relatively large [1, 2, 3,
4, 5] which imlply that the asumption $\theta_{13}=0$ in formulating the
neutrino mixing matrix must be rule out and hence the exact $\mu-\tau$
symmetry as the underlying symmetry of neutrino mass matrix is no longer
adequate to accommodate the recent experimental results.

In order to accommodate the nonzero $\theta_{13}$ and Jarlskog rephasing
invariant $J_{\rm CP}$ as a measure of CP violation in neutrino sector, in
this paper we break the $\mu-\tau$ symmetry sofly by introducing one complex
parameter to break the neutrino mass matrix with $\mu-\tau$ symmetry softly.
The paper is organized as follow: in section 2 we break the neutrino mass
matrix with $\mu-\tau$ symmetry by introducing one complex parameter into
neutrino mass matrix and in section 3 we determine the mixing angle
$\theta_{13}$ and $J_{\rm CP}$ by using the experimental results as input.
Finally, section 4 is devoted to conclusions.

##  2 Broken $\mu-\tau$ symmetry

Concerning the neutrino mass matrix that obey the $\mu-\tau$ symmetry and
mixing angle $\theta_{13}$, Mohapatra [6] stated explicitly that neutrino mass
matrix that obey $\mu-\tau$ symmetry to be the reason for maximal $\mu-\tau$
mixing and one gets $\theta_{13}=0$, conversely if $\theta_{13}\neq 0$ can
provide the $\mu-\tau$ symmetry beraking manifests in the case of normal
hierarchy. Aizawa and Yasue [7] analysis complex neutrino mass texture and the
$\mu-\tau$ symmetry which can yield small $\theta_{13}$ as a $\mu-\tau$
breaking effect. The $\mu-\tau$ symmetry breaking effect in relation with the
small $\theta_{13}$ also discussed in [8]. Analysis of the correlation between
CP violation and the $\mu-\tau$ symmetry breaking can be read in [9, 10, 11,
12].

Now, we are in position to study the effect of neutrino mass matrix that obey
the $\mu-\tau$ symmetry breaking in relation to the nonzero $\theta_{13}$ and
Jarlskog rephasing invariant $J_{\rm CP}$ by breaking the neutrino mass matrix
in Eq. (4). We break the neutrino mass matrix in Eq. (4) by introducing a
complex paramater $ix$ with the constraint that the trace of the broken
neutrino mass matrix is remain constant or equal to the trace of the unbroken
one. This scenario of breaking has been applied by Damanik [13] to break the
neutrino mass matrix invariant under a cyclic permutation. In this breaking
scenario, the broken neutrino mass matrix reads:

$M_{\nu}=\bordermatrix{&&&\cr&P&Q&Q\cr&Q&R-ix&S\cr&Q&S&R+ix}.$ (9)

As stated previously that the CP violation can be determined from the Jarlskog
rephasing invariant $J_{\rm CP}$. Alternatively, Jarlskog rephasing invariant
$J_{\rm CP}$ can be determined using the relation [14]:

$J_{\rm CP}=-\frac{{\rm
Im}\left[(M_{\nu}^{{}^{\prime}})_{e\mu}(M_{\nu}^{{}^{\prime}})_{\mu\tau}(M_{\nu}^{{}^{\prime}})_{\tau
e}\right]}{\Delta m_{21}^{2}\Delta m_{32}^{2}\Delta m_{31}^{2}},$ (10)

where $(M_{\nu}^{{}^{\prime}})_{ij}=(M_{\nu}M_{\nu}^{\dagger})_{ij}$ with
$i,j=e,\nu,\tau$, and $\Delta m_{kl}^{2}=m_{k}^{2}-m_{l}^{2}$ with
$k=2,3,~{}{\rm and}~{}l=1,2$. From Eq. (9), we have:

$\displaystyle
M_{\nu}^{{}^{\prime}}=\bordermatrix{&&&\cr&P^{2}+2Q^{2}&Q(P+S+R+ix)&Q(P+R+S-ix)\cr&Q(P+S+R-ix)&Q^{2}+R^{2}+S^{2}+x^{2}&Q^{2}+2S(R-ix)\cr&Q(P+S+R+ix)&Q^{2}+2S(R+ix)&Q^{2}+R^{2}+S^{2}+x^{2}}.$
(11)

##  3 Nonzero $\theta_{13}$ and Jarlskog rephasing invariant

From Eqs. (10) and (11) we have the Jarlskog rephasing invariant as follow:

$J_{\rm
CP}=\frac{2Q^{2}\left[S(S+P)^{2}-Q^{2}(R+S+P)-R^{2}S\right]x-(2Q^{2}S)x^{3}}{\Delta
m_{21}^{2}\Delta m_{32}^{2}\Delta m_{31}^{2}}.$ (12)

If we insert Eqs. (5)-(8) into Eq. (12), then we have the $J_{\rm CP}$ as
follow:

$\displaystyle J_{\rm
CP}=\frac{2(m_{2}-m_{1})^{2}c_{12}^{2}s_{12}^{2}c_{23}^{2}}{\Delta
m_{21}^{2}\Delta m_{32}^{2}\Delta
m_{31}^{2}}[[(-m_{1}s_{12}^{2}-m_{2}c_{12}^{2}+m_{3})s_{23}c_{23}$
$\displaystyle\times((-m_{1}s_{12}^{2}-m_{2}c_{12}^{2}+m_{3})s_{23}c_{23}+m_{1}c_{12}^{2}+m_{2}s_{12}^{2})^{2}$
$\displaystyle-(m_{2}-m_{1})^{2}c_{12}^{2}s_{12}^{2}c_{23}^{2}((m_{1}s_{12}^{2}+m_{2}c_{12}^{2})c_{23}^{2}+m_{2}s_{23}^{2}$
$\displaystyle-(m_{1}s_{12}^{2}+m_{2}c_{12}^{2}-m_{3})s_{23}c_{23}+m_{1}c_{12}^{2}+m_{2}s_{12}^{2})$
(13)
$\displaystyle-((m_{1}s_{12}^{2}+m_{2}c_{12}^{2})c_{23}^{2}+m_{2}s_{23}^{2})^{2}(-m_{1}s_{12}^{2}-m_{2}c_{12}^{2}$
$\displaystyle+m_{3})s_{23}c_{23}]x-c_{23}s_{23}[-m_{1}s_{12}^{2}-m_{2}c_{12}^{2}+m_{3}]x^{3}].$

From Eq. (13), one can see that the exact $\mu-\tau$ symmetry ($x=0$) lead to
$J_{\rm CP}=0$ as claimed by many authors, and if we still want to have
$J_{\rm CP}\neq 0$ from the $\mu-\tau$ symmetry, then we must break it softly.

From Eq. (13) one can see that in this breaking scenario, the Jarlskog
rephasing invariant ($J_{\rm CP}$) in neutrino sector can be proceed if we
break the neutrino mass matrix of $\mu-\tau$ symmetry by introducing complex
parameter $ix$ parameter. It is also apparent from Eq. (13) that $m_{1}\neq
m_{2}$ as a constraint to the existence of the $J_{\rm CP}\neq 0$.

In order to get the value of Jarlskog rephasing invariant $J_{\rm CP}$ of Eq.
(13), we use the experimental values of neutrino oscillation as input. The
current combined world data for neutrino squared-mass difference are given by
[15, 16]:

$\displaystyle\Delta m_{21}^{2}=7.59\pm 0.20(_{-0.69}^{+0.61})\times
10^{-5}~{}\rm{eV^{2}},$ (14) $\displaystyle\Delta m_{32}^{2}=2.46\pm 0.12(\pm
0.37)\times 10^{-3}~{}\rm{eV^{2}},~{}\rm(for~{}NH)$ (15) $\displaystyle\Delta
m_{32}^{2}=-2.36\pm 0.11(\pm 0.37)\times
10^{-3}~{}\rm{eV^{2}},~{}\rm(for~{}IH)$ (16) $\displaystyle\theta_{12}=34.5\pm
1.0(_{-2.8}^{3.2})^{o},~{}~{}\theta_{23}=42.8_{-2.9}^{+4.5}(_{-7.3}^{+10.7})^{o},~{}~{}\theta_{13}=5.1_{-3.3}^{+3.0}(\leq
12.0)^{o},$ (17)

at $1\sigma~{}(3\sigma)$ level. We also put $m_{1}=0$ as a first approximation
and within this scenario, for normal hierarchy (NH):

$\displaystyle m_{2}^{2}=\Delta m_{21}^{2},$ (18) $\displaystyle
m_{3}^{2}=\Delta m_{32}^{2}+\Delta m_{21}^{2}.$ (19)

By inserting the valeus of Eqs. (14)-(19), we have:

$\displaystyle J_{\rm CP}=0.4644x-832.9790x^{3}.$ (20)

From Eq. (20), we can determine the maximun value of $J_{\rm CP}$ by using the
relation:

$\displaystyle\frac{dJ_{\rm CP}}{dx}=0,$ (21)

which proceed $x=0.0167$. By substituting the value of $x=0.0167$ into Eq.
(20), we have the maximum value of Jarlskog rephasing invariant:

$\displaystyle J_{\rm CP}\approx 0.004.$ (22)

It is also possible to determine the Jarlskog rephasing invariant $J_{\rm CP}$
from the neutrino mixing matrix by using the relation:

$\displaystyle J_{\rm CP}={\rm Im}(V_{11}^{*}V_{23}^{*}V_{13}V_{21}).$ (23)

From neutrino mixing matrix of Eq. (1), the Jarlskog rephasing invariant
$J_{\rm CP}$ reads:

$\displaystyle J_{\rm
CP}=c_{12}s_{12}c_{23}s_{23}c_{13}^{2}s_{13}\sin{\delta}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}=c_{12}s_{12}c_{23}s_{23}\left(s_{13}-s_{13}^{3}\right)\sin{\delta}.$
(24)

If we put the experimental values of mixing angles $\theta_{12}$ and
$\theta_{23}$ of Eq. (17) into Eq. (24), then we have:

$\displaystyle J_{\rm CP}=0.2327\left(s_{13}-s_{13}^{3}\right)\sin{\delta}.$
(25)

Since the value of $s_{13}^{3}<<s_{13}$, the Eq. (25) can be approximated as
follow:

$\displaystyle J_{\rm CP}\approx 0.2327s_{13}\sin{\delta}.$ (26)

If we insert the value of Jarlskog rephasing invariant of Eq. (22) into Eq.
(26), then we have the Dirac phase $\delta$ as follow:

$\displaystyle\delta\approx\arcsin\left(\frac{0.0172}{\sin\theta_{13}}\right).$
(27)

By inserting the value of mixing angle $\theta_{13}$ as shown in Eq. (17), we
have:

$\displaystyle\delta\approx 11.2^{\rm o}$ (28)

##  4 Conclusions

We have studied sistematically the effect of breaking on neutrino mass matrix
that obey $\mu-\tau$ symmetry by introducing a complex parameter $ix$ with the
requirement that the trace of the broken $\mu-\tau$ symmetry is remain
constant. By using the experimental data as input, we can obtain the Jarlskog
rephasing invariant $J_{\rm CP}\neq 0$ that indicate the existence of CP
violation in neutrino sector and hence the Dirac phase $\delta$ which also
depend on the mixing angle $\theta_{13}$ for neutrino in normal hierarchy for
the case: $m_{1}=0$.

## References

  * [1] M. Apollonio et al. [Double Chooz Collab.], Phys. Lett. B 466, 415 (1999)
  * [2] P. Adamson et al. [MINOS Collab.], Phys. Rev. Lett. 107, 181802 (2011).
  * [3] K. Abe et al. [T2K Collab.], Phys. Rev. Lett. 107, 041801 (2011).
  * [4] F. P. An et al., Phys. Rev. Lett. 108, 171803 (2012).
  * [5] J. K. Ahn et al. [RENO Collab.], Phys. Rev. Lett. 108, 191802 (2012).
  * [6] R. N. Mohapatra, JHEP 0410, 027 (2004).
  * [7] I. Aizawa and M. Yasue, Phys. Rev. D 73, 015002 (2006).
  * [8] F. Fuki and M. Yasue, Phys. Rev. D 73, 055014 (2006).
  * [9] R. N. Mohapatra and W. Rodejohann, Phys. Rev. D 72, 053001 (2005).
  * [10] T. Baba and M. Yasue, Phys. Rev. D 75, 055001 (2007).
  * [11] H.-J. He and X.-J Xu, Phys. Rev. D 86, 111301 (R) (2012).
  * [12] A. Damanik, arXiv:1305.6900 [hep-ph].
  * [13] A. Damanik, Mod. Phys. Lett. A 26, 567 (2011).
  * [14] C. G. Branco, R. Gonzalez Felipe, F. R. Joaquim, I. Masina, M. N. Rebelo, and C. A. Savoy, Phys. Rev. D 67, 073025 (2003).
  * [15] M. Gonzales-Carcia, M. Maltoni and J. Salvado, JHEP 056, 1004 (2010).
  * [16] G. Fogli et al., J. Phys. Con. Ser. 203, 012103 (2010).


# Analyzing the Robustness of PECNet

Aryan Garg
Indian Institute of Technology, Mandi
<EMAIL_ADDRESS>Renu Rameshan
Indian Institute of Technology, Mandi
<EMAIL_ADDRESS>
###### Abstract

We perform a robustness analysis on PECNet [25], a pedestrian trajectory
prediction system for autonomous vehicles, with various clustering analyzes,
novel metrics for distribution analysis and classifications. We use synthetic
data augmentation techniques ranging from Newtonian mechanics to Deep
Reinforcement Learning based simulations to improve and test the system. We
also decouple the system and perform various ablation studies alongside other
experiments. We improved the state-of-the-art results on the Final
Displacement Error(FDE) metric by 9.5%. We introduce a novel architectural
change using implicit neural representations: Sinusoidal Representation
Networks (SIRENs) [33] for higher precision results to validate our robustness
hypothesis surrounding PECNet. We propose a novel, built upon PECNet, multi-
modal system as well.

##  1 Introduction

Autonomous driving cars [35] is a booming concept and the race to go all the
way from SAE level 0 to level 5 [36] is always on. Autonomous Vehicles aim to
solve multiple issues like traffic congestions, improve driver comfort levels,
and in the possible future make traveling on the roads safer and faster.
However, there exist many social dilemmas and ethical questions about their
incorporation in society [9]. Nonetheless, we don’t aim to answer these
questions through this study. To accomplish the aforementioned goals, we need
extremely robust, adaptive, flexible and general systems that additionally,
react quickly due to the high stakes’ dynamic environment. Pedestrians add to
that complexity due to their unknown destination and constantly evolving
individual route planning and course correction based on the circumstances in
their field of view. This rules out using simple Newtonian mechanics [6] to
predict their motion. Autonomous cars are essentially social agents [8] and
for that they need excellent anticipation systems to ensure people’s safety
without compromising on efficiency and performance. The problem is popularly
known as Human Motion / Pedestrian Trajectory Prediction in the literature.

With the unceasing acceleration in the research, design and deployment of
better artificially intelligent systems, we need a deeper understanding and
sanity checks on all state-of-the-art models to ensure robustness and bridge
the gap between deployment and research. Here robustness shouldn’t be confused
with adversarial robustness [39].

However, recently there have been multiple state-of-the-art benchmarks that
grossly overfit by finding a pareto optimal solution(parameters) for a
specific dataset. These models prove to be simply undeployable. The process of
creating a model that benchmarks is given higher importance (for the sake of
publications) than actual robust and potentially deployable system[29]. We
propose a robustness/deployment fitness metric in section Discussion 5.

One such benchmark is: PECNet [25] that could not even generalize on Newtonian
Trajectories and did even worse on noisy Newtonian Trajectories which are
closer to the distribution of actual pedestrian trajectories.

##  2 Related Work

Variational Autoencoders(VAEs) introduced by Kingma et al [20] fall under the
umbrella of generative models that are far from standard autoencoders [26, 22,
37, 7] as they first estimate the underlying distribution and not a fixed
feature space using the encoder. Then the decoder samples, which is inherently
stochastic, from this distribution and gives a generated output by reducing
the Estimated Lower Bound (ELBO). The bottleneck has a stochastic node, from
which the decoder queries, through which gradients are made to flow using the
reparameterization trick. A great tutorial to understand vanilla VAEs was
given by Carl Doersch in 2016 [11]. Few improvements and/or variations of the
vanilla VAEs are as follows: $\beta$-VAE [17], Vector Quantized
VaritionalAutoEncoder (VQ-VAE) by Oord _et al_. in 2017 [27] followed by VQ-
VAE2 by Razavi _et al_. in 2019 [30], Temporal Difference Variational
AutoEncoder (TD-VAE) by Gregor _et al_. in 2019 [15]. The inspiration to
create the Reinforcement Learning based dataset was derived from the
underlying ideas of TD-VAE.

Generally, the assumptions of a VAE are weak, and training is fast. They are
able to generalize well unlike generative adversarial networks [13] and hence
have become quite popular within three years of their birth. Even though VAEs
have been extremely successful in unsupervised learning applications such as
generating faces [31], physical models [21], house numbers [14], segmentation
[34] and forecasting from static images [38], (which is the primary
application discussed here) they commonly suffer from the problem of vanishing
Kullback-Leibler(KL) divergence. Cyclic annealing schedule as introduced by Fu
_et al_. [12] is useful to mitigate this issue and we implement the same for
certain experiments.

Moving forward to the ’human future prediction task’ we take note of the most
common datasets. See table 1

Dataset | Trajectories(Pedestrian) | FPS
---|---|---
ETH | 750 | 2.5
UCY | 786 | 2.5
SDD | 5232 | 30
Table 1: Most common datasets [4]. Other popular ones include: CITR, Town
Center, KITTI, Waymo and Forking Garden Paths (synthetic) [24]

State-of-the-art benchmarks on the Stanford Drone Dataset(SDD) are briefly
mentioned in table 2.

Model | Year | ADE | FDE | K
---|---|---|---|---
DESIRE | 2017 | 19.25 | 34.05 | 20
Social GAN | 2018 | 27.23 | 41.44 | 20
Sophie | 2018 | 16.27 | 29.38 | 20
CGNS | 2019 | 15.6 | 28.2 | 20
CF-VAE | 2019 | 12.60 | 22.30 | 20
P2TIRL | 2020 | 12.58 | 22.07 | 20
PECNet | 2020 | 9.96 | 15.88 | 20
Y-Net | 2021 | 7.85 | 11.85 | 20
$V^{2}$-Net | 2021 | 7.12 | 11.39 | 20
NSP-SFM | 2022 | 6.52 | 10.61 | 20
Table 2: ADE - Average Displacement Error. FDE - Final Displacement Error. K -
number of samples allowed for multi-modal prediction

##  3 Methodology

A clearly outlined plan for PECNet was initially proposed. It incorporated
analyzing PECNet, Stanford Drone Dataset(SDD), ETH [28] & UCY [23] datasets
and all the literature around the work, in the first phase. Then, dependent on
the analysis, the second phase would potentially include improving PECNet if
deemed fit.

###  3.1 Clustering Analysis

We carried out an analysis of the Stanford Drone Dataset[32] first, by
visualizing and clustering the trajectories, using various norms (see figures
1, 2, 3, 4). It is evident that most trajectories are not spatially spread out
and the dataset possesses rotational equivariance.

Figure 1: Full training dataset clustered based on the Frobenius norm, with
black trajectories representing the cluster. Figure 2: Clustering based on
L1-norm. Black trajectories are representatives of the cluster. Figure 3:
Clustering based on L2-norm. Black trajectories are representatives of the
cluster. Figure 4: Clustering based on L-$\inf$ norm. Black trajectories are
representatives of the cluster.

###  3.2 Non-linearity analysis: Abruptness-Score

Then we clustered trajectories based on the bounding boxes to get an idea of
the maximal displacement and turn in each. Also, came up with a novel and
simple metric: Abruptness Score to measure the turns and variability or non-
linearity in each trajectory. An areal-scaled and and unscaled version of the
metric is used for analysis and outlier detection. The intuition and
mathematical formulation are as follows:

In the example figure 5, the trajectories $\zeta_{1}=\\{A,B,C1\\}$ and
$\zeta_{2}=\\{A,B,C2\\}$ are shown. The dotted blue line is the normal to the
red danger zone. Points that fall under this danger zone will form an obtuse
turn trajectory like $\zeta_{2}$. Naturally, we want the score to assign a
larger value to $\zeta_{2}$ than $\zeta_{1}$ since the turn is huge and the
trajectory (more) abruptly changes direction.

Figure 5: Defining Turns

Mathematically,

$AbScore=\left\lceil\frac{180.\theta}{10\pi}\right\rceil|\vec{a}\times\vec{b}|$
(1)

where

$\vec{a}=\vec{AB},\vec{b}=\vec{BC}$ (2)

$\theta=|\arcsin\frac{\vec{a}\times\vec{b}}{|\vec{a}||\vec{b}|}|$ (3)

If $\theta$ is obtuse, we add $pi/2$ to $\theta$ before sending it to equation
1.

For scaling we simply divide the abruptness score by the area of the tightest-
bounding-box of the trajectory or divide by
$(max(\zeta_{x})-min(\zeta_{x}))*(max(\zeta_{y})-min(\zeta_{y})$. For
perfectly linear trajectories, we use the length of the trajectory for
scaling.

Why do we need such areal-scaling?

To compare trajectories that span different sizes or regions in terms of their
non-linearity.

Based on this metric we analyze SDD and report the findings in table 3 and
figure 6.

Statistic | Value
---|---
Maximum | 494866.374
Minimum | 0.0
Mean | 3430.665
Std. Deviation | 11987.34
Table 3: SDD: AbScore Non-linearity analysis Figure 6: SDD’s Non Linearity
Distribution Tail. (Thresholded to remove outliers)

###  3.3 Unique-points & Qualitative Analysis

We perform a classification of the training dataset based on the number of
unique points in each trajectory. See table 4

Then we manually identify 7 qualitative classes for each trajectory as
follows:

  * •

Type 1: Stationary pedestrians

  * •

Type 2: 3-8 unique points in trajectory bounded in a 5x5 box

  * •

Type 2F: F means category Flying: A straight line trajectory will be shifted
into the line starting from a new point if the perspective of the viewer
(here: drone) changes. We call this scenario the flying category trajectory.
The drone usually translates along an axis here.

  * •

Type 3: 3-9 unique points loosely bounded in a 100x100 box

  * •

Type 3F: (Flying) Same as 2F

  * •

Type 4: Start and Goal points are within a 5x5 box

  * •

Type 5: Flying randomly. This is different from 2F and 3F in the sense that
the drone translated haphazardly here.

  * •

Type 6: Backtracker: The pedestrian re-traces his steps after a while. Usually
after 6-7 steps.

  * •

Type 7: Perfectly Linear to Moderately linear trajectories that could be
modelled by simple Newtonian mechanics.

These two classifications were done to emulate the statistical properties of
the training dataset for augmentation purposes.

Unique Points | Trajectories | % dataset
---|---|---
1 | 145 | 5.13%
2 | 62 | 2.19 %
3 | 71 | 2.51 %
4 | 69 | 2.44%
5 | 57 | 2.01%
6 | 41 | 1.45%
7 | 51 | 1.80%
8 | 28 | 0.99%
9 | 26 | 0.92%
10 | 24 | 0.85%
11 | 22 | 0.78%
12 | 25 | 0.88%
13 | 17 | 0.60%
14 | 24 | 0.85%
15 | 22 | 0.78%
16 | 22 | 0.78%
17 | 39 | 1.38%
18 | 30 | 1.06%
19 | 76 | 2.69%
20 | 1978 | 69.92%
Table 4: Trajectories’ unique points. 145 trajectories had 1 unique point,
i.e, the goal and starting point as the same with all other points being
sampled there itself: A stationary pedestrian.

These two analyses gave us a clear idea about the trajectories that the model
was learning on.

###  3.4 Benchmark Validation & Sanity Runs

Moving onto the system, we successfully reproduce the benchmark-S (without
social pooling layer) metricS: Average Displacement Error (ADE) and Final
Displacement Error (FDE) [3, 2]. This was done as a sanity check for our
experiments and tests. Figure: 7

Figure 7: ADE & FDE Loss vs Epochs. Reproduction of the benchmark. Adam
Optimizer, learning rate = 0.0003, epochs = 1000, $K=20$. Best $FDE:$ 15.60.
$ADE:$ 10.47. Model converges at $556^{th}$ epoch and the results are as
quoted in [25].

We also change the learning rates for another sanity run. Figure 8

Figure 8: ADE & FDE Loss vs Epochs plots. Sanity-runs. See table 5. 1000 epochs. Learning Rate | Best ADE | Best FDE
---|---|---
0.001 | 11.01 | 15.62
0.0005 | 12.52 | 15.68
0.0003 | 10.47 | 15.60
0.0002 | 10.65 | 15.78
0.0001 | 10.65 | 15.78
Table 5: Sanity Run on different learning rates. Figure 8. Bold - benchmark
reproduction.

###  3.5 Improvements

After this analysis, we moved on to the second phase of the plan: improvement.
Naturally, the first idea for us was data augmentation due to the intensive
clustering and classification of trajectories already performed. We created
five different synthetic datasets to augment.

  1. 1.

Purely Newtonian Trajectories: Consists of two types of accelerations: static
and variable but within bounds. 111Code of clustering analysis and the
synthetic datasets’ creation can be found at: Synthetic-Dataset-Creation-
notebook

  2. 2.

Noisy Newtonian Trajectories The synthetic dataset created in the former was
re-created with noise 11footnotemark: 1.

  3. 3.

Trajectories that followed circuits, loops, spirals, and other geometric
curves with both variable and fixed sampling of the points 11footnotemark: 1.

  4. 4.

Interaction modeling of trajectories using a novel Hidden Markov Model(HMM):
HMMs were considered for the interaction modeling due to their probabilistic
nature and success in the tasks which depict structure in time [10]
11footnotemark: 1. See figure 9 for an overview of the HMM model with states
and action spaces and figure 10 for the samples generated.

Figure 9: Interaction modeling graph of the hidden markov model (HMM) to
emulate multiple human trajectories as produced in any real static-drone
dataset.

Figure 10: Interaction modeling samples generated.

  5. 5.

Reinforcement Learning based agents that interacted with each other based on
two inherent goals that model humans: Reaching the goal fast and avoiding
collisions with fellow pedestrians. Agents were given control of their
acceleration instead of their velocity to get smoother, human-like
trajectories. Apart from acceleration, stopping for another crossing
pedestrian, or being considerate which was observed in many actual
trajectories, was implicitly decided by the agent’s sociability, fitness, and
patience attributes. These attributes remain fixed over a simulation and are
randomly initialized for diversity. We observe that agents initially learn the
greedy approach to reach their goals as fast as possible leading to multiple
collisions which are then heavily penalized by the learning objective. This in
turn leads to agents choosing sub-optimal paths that avoid maximal collisions.
We add a detection mechanism to the agents so they can observe other agents in
the scene too. This leads into super-natural trajectories where agents slow
down or speed up when in the proximity of other agents who might potentially
collide 11footnotemark: 1.

Mathematically, the reward function is formulated as follows:

$R_{t}\>=\>\frac{AF^{t}\,(n_{ICS}+1)^{(AS+AP)}}{t^{2}\,(1+\|G-x_{t}\|)}$ (4)

$where\;AF,\>AP,\>and\>AS\;\epsilon\;[0,1]$ (5)

where,
$n_{ICS}$ := Number of times agent entered impending collision state but
didn’t enter collision state,

$AS$ := Agent’s sociability (an inherent and unchanging quality of the agent),

$AP$ := Agent’s Patience (an inherent and unchanging quality of the agent)

$AF$ := Agent’s Fitness. Usually a discount factor in standard RL reward
function formulation, but here it signifies how quickly the agent gets tired.
If an agent it extremely unfit, the more it walks, the lesser rewards it gets
as it gets fatigued and naturally prefers walking less.

$t$ := Time elapsed

$G$ := Goal or end-point

$x_{t}$ := Current position

$\|.\|$ := $L_{2}\,norm$

$R_{t}$ := Reward received at time $t$

and loss function is the one used in standard deep-policy gradients methods:

$J(\phi)=-\,\sum_{t=1}^{t=N}\,log(P_{\phi}(a_{t}|s_{t}))\,R_{t}$ (6)

where,
$P(.)$ is parameterized by $\phi$ modelled by a function approximator(DNN)
that outputs $\mu\>and\>\sigma^{2}$ vectors corresponding to actions and state
at time ’t’.

Some samples produced using this approach are produced in figure 11

Figure 11: Agent inserted and trained in the HMM interaction model. Agent’s
trajectory is turquoise. Evolution of the samples produced(L-to-R): First,
Initial Stages of Training: Agent learns to turn. Second depicts a complicated
scene where the agent learns to avoid multiple collisions.Third depicts a
scene where the agent successfully avoids a collision and reaches it’s goal
for the first time.

An overview of the generation process is in figure 12

Figure 12: Deep Policy Gradients Agents for emulating a live drone-shot scene
with human-like agents reaching their goals.

  6. 6.

Synthetic SDD(syn-SDD) is the re-creation of the SDD based on the qualitative
classification in the same proportions 11footnotemark: 1.

We augment SDD with each category of synthetic dataset in different
proportions, and note significant worsening in performance on both ADE and
FDE. However, the FDE improved at the cost of ADE, when SDD was augmented with
specific proportions of syn-SDD.

Assuming that the dataset doesn’t represent the trajectories accurately enough
and that the gradient terrain doesn’t have a sharp creek which Adam[18](which
is generally considered very sensitive) finds early on due to the hyper-
parameter fine tuning methods, (which are not discussed in the paper: PECNet),
we moved towards deep-learning based improvements. The improvements that we
proposed, implemented and tested were as follows:

  1. 1.

Hyper-parameter tuning (it was soon discovered that the model was using a
pareto optimal solution).

  2. 2.

Cyclical annealing schedule or more colloquially, $\beta$-annealing for stable
training was implemented. KL vanishing problem was observed on some augmented
datasets.

  3. 3.

Replacing the standard multilayer perceptron layers(MLPs) with Sinusoidal
Representational Networks (SIRENs) [33]

We also speculated using a Partially Observable Markov Decision Process for
the same task [5]. More on this in Discussion section: 5

##  4 Experiments

With the objectives of testing the robustness of PECNet and understanding how
the model behaves under new and challenging situations, which a real system
would undergo on a daily basis, we perform experiments where we generate
synthetic datasets for augmentation and testing, replace standard Multi-Layer-
Perceptrons(MLPs) with SIREN MLPs. We finally carry out an ablation study.

###  4.1 Synthetic Datasets

Based on the quantitative (table: 4) and qualitative analysis(7 representative
classes) under section Methodology: 3, we augment the training dataset while
keeping the statistical properties of the training dataset intact.
Augmentations vary from $1\%$ to $18\%$. We report the results in table 6 and
observe that even small quantities of noise in the original dataset, due to
augmentation, make the system worthless for deployment. PECNet heavily
overfits on the SDD dataset and it is highly probable that Adam [19] finds a
deep crevice in the gradient surface. An evolutionary optimization strategy
like Covariance Matrix Adaptation (CMA-ES) [16] would highlight the
shortcomings of the robustness of PECNet.

% Augmented | Total Trajectories | ADE | FDE
---|---|---|---
1% | 18328 | 64.34 | 19.18
3% | 19048 | 53.22 | 15.63
5% | 19766 | 45.17 | 15.72
6% | 20126 | 51.73 | 15.43
8% | 20844 | 46.37 | 15.75
10% | 21564 | 51.51 | 15.54
13% | 22642 | 40.76 | 15.90
15% | 23360 | 51.40 | 18.36
18% | 24438 | 56.56 | 15.90
Table 6: Standard learning rate: $3e-4$, 1000 epochs and no social
pooling.Note that the training dataset has 18k trajectories due to simple
augmentations(rotations & translations) introduced in the original paper [25].

We create a pure synthetic dataset and perform three experiments:

  1. 1.

Testing on ours where the test-set is generated as per the qualitative
statistics of the testing set.

  2. 2.

Purely synthetic dataset training and testing on original test-set.

  3. 3.

Augmented training-set and testing on ours.

###  4.2 SIREN Improvement

Human Trajectories are inherently physical and natural entities with spatial
and temporal derivatives. A trajectory, therefore, can be treated as a
physical signal as well. The paradigm of implicit neural representation of
continuous and derivable physical signals has been on the rise in the recent
times. One such weight-initialization and periodic activation function, for
finer detail representation in natural signals, is a method introduced by
Sitzmann _et al_. [33] with impressive results on a wide-range of tasks and a
thorough explanation of the mathematical background in the appendix.

We implement the idea in [33] and replace all standard MLPs of PECNet with
SIREN MLPs. We expected an improvement or atleast the constancy of the
benchmark results. However, we observed an adverse effect on both metrics
which reinforced our belief in the hypothesis that PECNet abysmally overfits
on SDD.

We present the results of our novel architectural contribution for ’human
trajectory prediction task’ in table 7.

Learning Rate | ADE | FDE | Best FDE epoch
---|---|---|---
0.001 |  $>$50 | 15.91 | 724
0.0005 | $>50$ | 15.83 | 345
0.0003 | $>50$ | 15.77 | 480
0.0002 | $>50$ | 15.66 | 445
0.0001 | $>50$ | 15.80 | 588
Table 7: SIREN-PECNet Results. No decoupling. 1000 epochs. For Loss vs Epoch
plots, see figure 13

Figure 13: ADE & FDE Loss vs Epochs plots. Non-Decoupled, SIREN upgraded
PECNet. Learning rates: 0.001, 0.0005, 0.0003, 0.0002 and 0.0001 respectively.
See table 7

###  4.3 Ablation Studies

We performed two ablation studies. First, by decoupling the ADE and the FDE
metrics. See table 8 and figure 14.

Learning Rate | ADE | FDE | Best FDE epoch
---|---|---|---
0.001 |  $>$50 | 15.68 | 457
0.0005 |  $>$50 | 15.76 | 301
0.0003 |  $>$50 | 15.9 | 541
0.0002 |  $>$50 | 15.65 | 420
0.0001 |  $>$50 | 15.92 | 391
Table 8: Decoupled system (ADE and FDE). No social pooling. See log loss plots
(figure: 14)

Second, without the standardization parameter on the input trajectories. We
note an adverse effect on ADE and a significant state-of-the-art improvement
on FDE, by removing standardization. See table 9 and figure 15.

Learning Rate | ADE | FDE | Best FDE epoch
---|---|---|---
0.001 | 22.20 | 9.32 | 915
0.0005 | 29.91 | 9.05 | 834
0.0003 | 25.92 | 9.37 | 998
0.0002 | 26.75 | 9.04 | 908
0.0001 | 25.57 | 9.05 | 235
Table 9: State-of-the-art FDE. No Standardization. No decoupling. No social
pooling. See log-loss plots (figure: 15)

We notice a striking absurdity in the codebase where the standardization
parameter($=1.86$) is also dividing the loss. See figure 16. This is a grave
mistake since a Variational Autoencoder using ReLU [1] as an activation
funciton, is by no means a linear predictive machine.

Figure 14: ADE & FDE Loss vs Epochs plots. Decoupled system. Learning rates:
0.001, 0.0005, 0.0003, 0.0002 and 0.0001 respectively. See table 8

Figure 15: ADE & FDE Loss vs Epochs plots. No Standardization. Learning rates:
0.001, 0.0005, 0.0003, 0.0002 and 0.0001 respectively. See table 9

Figure 16: Repository code of PECNet: Misuse of hyper-parameter:
$data_{s}cale$ to divide the ADE loss metric.

##  5 Discussion

We propose three further changes that could help in further testing PECNet:
Employing CMA-ES instead of Adam to skip deep crevices and explore the
gradient surface more naturally. The second addition would be implementing a
cyclic annealing scheduling to stabilize the Variational AutoEncoder training.
The third would be to add another generative model in conjunction to produce
multi-modal outputs, see Figure and also introduce a confidence metric for
better predictions and larger controllability.

Figure 17: Proposed multi-modal PECNet

We also lay emphasis on the widening bridge between academic research and
real-world applications of the same based on over-fitting of the datasets to
set benchmarks for publications. For the same reasons we call for a robustness
and deployment fitness score based on thorough code-reviewing and testing
under simulations that would be encountered in a real world scenario.

A deep learning based idea, for the same, is to let all models(academic and
industrial) exist as test-sets for a generative model, let’s say robGAN,
trained on deployed systems. When robGAN sees a model, it’s discriminator
should be able to tell whether the model is fit for deployment.

## References

  * [1] Abien Fred Agarap.  Deep learning using rectified linear units (relu), 2018.
  * [2] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese.  Social lstm: Human trajectory prediction in crowded spaces.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [3] Alexandre Alahi, Vignesh Ramanathan, and Li Fei-Fei.  Socially-aware large-scale crowd forecasting.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
  * [4] Javad Amirian, Bingqing Zhang, Francisco Valente Castro, Juan Jose Baldelomar, Jean-Bernard Hayet, and Julien Pettre.  Opentraj: Assessing prediction complexity in human trajectories datasets.  In Asian Conference on Computer Vision (ACCV), number CONF. Springer, 2020.
  * [5] Haoyu Bai, Shaojun Cai, Nan Ye, David Hsu, and Wee Sun Lee.  Intention-aware online pomdp planning for autonomous driving in a crowd.  In 2015 ieee international conference on robotics and automation (icra), pages 454–460. IEEE, 2015.
  * [6] Chris L. Baker, Rebecca Saxe, and Joshua B. Tenenbaum.  Action understanding as inverse planning.  Cognition, 113(3):329–349, 2009.  Reinforcement learning and higher cognition.
  * [7] Yoshua Bengio, Éric Thibodeau-Laufer, Guillaume Alain, and Jason Yosinski.  Deep generative stochastic networks trainable by backprop, 2013.
  * [8] Maren Bennewitz, Wolfram Burgard, and Sebastian Thrun.  Learning motion patterns of persons for mobile service robots.  In In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA, pages 3601–3606, 2002.
  * [9] Jean-François Bonnefon, Azim Shariff, and Iyad Rahwan.  The social dilemma of autonomous vehicles.  Science, 352(6293):1573–1576, 2016.
  * [10] M. Brand, N. Oliver, and A. Pentland.  Coupled hidden markov models for complex action recognition.  In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 994–999, 1997.
  * [11] Carl Doersch.  Tutorial on variational autoencoders, 2016.
  * [12] Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin.  Cyclical annealing schedule: A simple approach to mitigating kl vanishing, 2019.
  * [13] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks, 2014.
  * [14] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra.  Draw: A recurrent neural network for image generation, 2015.
  * [15] Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, and Theophane Weber.  Temporal difference variational auto-encoder, 2018.
  * [16] Nikolaus Hansen.  The cma evolution strategy: A tutorial, 2016.
  * [17] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-VAE: Learning basic visual concepts with a constrained variational framework.  In International Conference on Learning Representations, 2017.
  * [18] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2014.
  * [19] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2017.
  * [20] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2013.
  * [21] Tejas D Kulkarni, William F. Whitney, Pushmeet Kohli, and Josh Tenenbaum.  Deep convolutional inverse graphics network.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [22] Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Ng.  Efficient sparse coding algorithms.  In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
  * [23] Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski.  Crowds by example.  Computer Graphics Forum, 26, 2007.
  * [24] Junwei Liang, Lu Jiang, Kevin Murphy, Ting Yu, and Alexander Hauptmann.  The garden of forking paths: Towards multi-future trajectory prediction, 2019.
  * [25] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon.  It is not the journey but the destination: Endpoint conditioned trajectory prediction.  CoRR, abs/2004.02025, 2020.
  * [26] Field D. Olshausen, B.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images, 1996.
  * [27] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning, 2017.
  * [28] S. Pellegrini, A. Ess, K. Schindler, and L. van Gool.  You’ll never walk alone: Modeling social behavior for multi-target tracking.  In 2009 IEEE 12th International Conference on Computer Vision, pages 261–268, 2009.
  * [29] Heimann R.  Doing ai: A business-centric examination of ai culture, goals, and values, 2021.
  * [30] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2, 2019.
  * [31] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models, 2014.
  * [32] A. Robicquet, A. Sadeghian, A. Alahi, and S. Savarese.  Stanford drone dataset.
  * [33] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions, 2020\.
  * [34] Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [35] Sebastian Thrun, Wolfram Burgard, and Dieter Fox.  Probabilistic robotics (intelligent robotics and autonomous agents).  2005\.
  * [36] David Ticoll.  Driving changes: Automated vehicles in toronto.  10 2015.
  * [37] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, page 1096–1103, New York, NY, USA, 2008. Association for Computing Machinery.
  * [38] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert.  An uncertain future: Forecasting from static images using variational autoencoders, 2016.
  * [39] Qingzhao Zhang, Shengtuo Hu, Jiachen Sun, Qi Alfred Chen, and Z. Morley Mao.  On adversarial robustness of trajectory prediction for autonomous vehicles.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15159–15168, June 2022.