# Evaluating Digital Work Instructions with Augmented Reality versus Paper-
based Documents for Manual, Object-Specific Repair Tasks in a Case Study with
Experienced Workers∗ ††thanks: ∗This version of the article has been accepted
for publication, after peer review but is not the Version of Record and does
not reflect post-acceptance improvements, or any corrections. The Version of
Record is available online at: http://dx.doi.org/10.1007/s00170-023-11313-4

Leon Eversberg1, Jens Lambrecht1 1Technische Universität Berlin, Chair
Industry Grade Networks and Clouds, Straße des 17. Juni 135, 10623 Berlin,
Germany<EMAIL_ADDRESS>
###### Abstract

Manual repair tasks in the industry of maintenance, repair, and overhaul
require experience and object-specific information. Today, many of these
repair tasks are still performed and documented with inefficient paper
documents. Cognitive assistance systems have the potential to reduce costs,
errors, and mental workload by providing all required information digitally.
In this case study, we present an assistance system for object-specific repair
tasks for turbine blades. The assistance system provides digital work
instructions and uses augmented reality to display spatial information. In a
user study with ten experienced metalworkers performing a familiar repair
task, we compare time to task completion, subjective workload, and system
usability of the new assistance system to their established paper-based
workflow. All participants stated that they preferred the assistance system
over the paper documents. The results of the study show that the manual repair
task can be completed 21 % faster and with a 26 % lower perceived workload
using the assistance system.

###### Index Terms:

Assistive technology, Augmented reality, Case study, Digital work
instructions, Maintenance repair overhaul

##  I Introduction

In contrast to the paradigm of workerless factories from the 1980s, humans
remain the most flexible entities in the era of Industry 4.0 and Smart
Manufacturing [1]. However, complexity is ever-increasing due to new
manufacturing paradigms such as shorter product life cycles, increasing
product variety, and mass customization [2, 3]. To overcome these challenges,
industrial assistance systems can be utilized in order to enhance a person’s
physical, sensorial or cognitive capabilities, leading to a so-called Operator
4.0 [4]. Cognitive assistance systems (CAS) enhance cognitive capabilities by
providing digital work instructions, helping users make decisions, and
assisting them in learning new tasks [5]. The digitization of paper-based
documents can lead to significant increases in efficiency, as providing
digital work instructions removes the need for printing, searching, filing,
and retrieving paper-based documents [6].

With traditional paper-based work instructions, e.g., for assembly or
maintenance tasks, the user must first search for relevant documents. Then, he
has to switch between the two-dimensional work instructions, which tell him
how to perform the task, and the execution of the task on the three-
dimensional physical object [7]. According to Cognitive Load Theory, splitting
the user’s attention between spatially or temporally separated information
sources leads to additional cognitive load [8]. This so-called Split-Attention
Effect can be reduced with Augmented Reality (AR) [9], which superimposes
virtual objects onto the real world.

Research on industrial CAS with AR has seen a steady increase in the number of
publications over the last decade and focuses primarily on manual assembly and
maintenance tasks [10, 11, 12]. For those applications, digital work
instructions can be semi-automatically generated, given that each task follows
a standardized meta-model of a workflow [13, 14, 15].

Manual repair tasks in the industry of maintenance, repair, and overhaul (MRO)
are less standardized and instead require intuition- and experienced-based
decision-making from experienced workers [16, 17], e.g., for the repair of
turbine blades or fiber-reinforced composite structures in an aircraft. In our
previous work [18], we described the system architecture of a CAS based on the
digital twin for manual, object-specific repair tasks. In this present work,
we describe the evaluation of our CAS in a case study with experienced workers
on a familiar turbine blade repair task. The main contributions of this work
are the following:

  * •

the presentation of a CAS for manual, object-specific repair tasks which are
frequently performed in the MRO industry, and

  * •

experimental results from a user study with experienced shop floor
metalworkers comparing the presented CAS to their current paper-based
workflow.

##  II Related Works

Research on CAS with AR has been focusing primarily on manual assembly and
maintenance tasks [10, 11, 12]. Multiple studies have shown the benefits of
CAS with AR, such as reducing the time to complete manual tasks [19, 20],
human errors [21, 22], mental workload [7], or improving the learning curve
for new tasks [23, 24]. The three most commonly used metrics for evaluating
industrial assistance systems are task completion time (TCT), the number of
human errors in the process, and the subjectively perceived workload with the
NASA-TLX survey [11, 7].

Funk et al. [25] developed an assistance system for manual assembly equipped
with a Microsoft Kinect 3D camera and a projector for AR work instructions.
Their assistance system recognizes the current assembly step by evaluating
pick locations, the current assembly workpiece, and a predefined tool zone. In
an 11-day user study at an assembly line of a car manufacturing company, they
found that the in-situ projections were useful during the learning phase of
untrained workers. However, for expert workers, the assistance system had a
negative effect, i.e., perceived workload and TCT increased [26].

In [27], Lai et al. developed a CAS for manual assembly tasks. They equipped a
workbench with two webcams and a display for AR. A neural network was trained
on synthetic images to detect relevant tools with a webcam and highlight their
position in digital work instructions. In a user study with 20 university
students with no prior experience, participants were asked to carry out a
spindle motor assembly task. Compared to paper-based documents, the CAS
reduced the TCT by 33 % and the number of errors during the task by 32 %.

Uva et al. [28] developed a projective AR workbench for assembly and
maintenance tasks. A projector was used to visualize text and 2D symbols on
the workbench and on the maintenance object. The maintenance object was
mounted on a movable tracking board with markers. A user study was conducted
in [29] with 16 untrained engineering students to compare projective AR with
paper-based documents. For a combination of assembly and disassembly tasks on
a motorbike engine, they found a 20 % reduction in TCT using the assistance
system, as well as an improvement in error rate and subjectively better ease
of use, satisfaction level, and intuitiveness. In conclusion, they point out
that the assistance system is particularly beneficial for tasks with high
complexity.

Kästner et al. [24] presented an assistance system with monitor-based AR for
complex manual tasks. They used deep learning models for object detection and
action recognition to automatically detect the current work step and display
corresponding work instructions. In a user study with 30 inexperienced
participants, they found that the group with AR assistance had a lower TCT
than the group without AR assistance in the first 13 iterations. However, for
more than 13 iterations, the group without AR assistance performed better than
the AR group. Therefore, it was concluded that AR assistance is beneficial for
untrained workers in the first iterations when learning a new task.

Havard et al. [30] conducted a user study with 20 engineering students to
compare PDF maintenance instructions with AR maintenance instructions on a
mobile tablet. The maintenance process consisted of 27 actions to replace two
springs in a machine. Several markers were placed on the machine for spatial
registration of the AR work instructions. In the evaluation, no statistically
significant difference was found for either group in terms of TCT or mental
workload using NASA-TLX. In summary, they recommend AR work instructions for
training inexperienced workers in sufficiently complex tasks and tasks that
are frequently performed.

In [31], AR work instructions using a projector were compared to paper and
oral work instructions for three different assembly tasks. 44 participants,
some of whom were cognitively or physically impaired, participated in the user
study. The evaluation found that with AR work instructions, participants had a
lower perceived task complexity and made fewer errors. However, there was no
significant difference in TCT. Vanneste et al. point out that the advantages
of AR diminish over time with repetition, therefore AR should be used for less
experienced workers.

In summary, current research on CAS in manufacturing has focused primarily on
manual assembly and maintenance. In the application area of maintenance,
research on assistance systems has been focusing on standardized replacement
repair according to the scheme of step-by-step disassembly, component
replacement, and step-by-step assembly [32, 33, 29, 30, 34]. Evaluations of
industrial assistance systems mostly use inexperienced participants in their
user study, e.g., computer science students, who are not familiar with the
evaluated task [35]. Evaluation results often show that positive effects from
the usage of CAS diminish over time, thus most industrial assistance systems
are found to be useful for training new employees but not necessarily for
daily use by experienced workers.

##  III Research Questions

In order to be used in daily productive operations by experienced workers, a
new assistance system must provide a noticeable improvement over the
established workflow. Based on the literature from Section II, there is a gap
in research regarding the evaluation of CAS with experienced workers on
familiar real-world tasks.

In [18] we described the development of a CAS for the manual repair process of
turbine blades in MRO. The system was developed with a human-centered design
approach specifically for experienced workers. To evaluate the developed
assistance system, we formulate the following two research questions (RQ) for
this case study:

  * •

RQ 1: What effect does the developed CAS have on the TCT of experienced
workers when performing familiar repair tasks on turbine blades?

  * •

RQ 2: What effect does the developed CAS have on the perceived workload of
experienced workers when performing familiar repair tasks on turbine blades?

##  IV Material and methods

The system architecture of the developed CAS has been described in our
previous work in [18]. Thus, only a brief overview of the system and its
assistance functions will be given in Subsection IV-A. Afterward, the
conducted user study will be described in detail in Subsection IV-B.

###  IV-A The Cognitive Assistance System

As shown in Fig. 1, we equipped a manual workstation for grinding work on
turbine blades with the following additional hardware:

  * •

two Microsoft Azure Kinect 3D cameras for context awareness,

  * •

a Cognex DataMan DM 8600 scanner to read 1D and 2D bar codes,

  * •

a 27-inch touchscreen monitor for user interaction and

  * •

a 43-inch monitor to show screen-based AR content.

Figure 1: Our CAS supports shop floor metalworkers during the manual grinding
of turbine blades. Based on the scanned serial number of a turbine blade,
digital work instructions are presented on the touchscreen and on the AR
display.

We chose the Robot Operating System (ROS) as a framework for asynchronous
many-to-many communication between our software modules through ROS topics.
The system’s components, depicted in Fig. 2, can be grouped into ROS nodes, a
dynamic web application as a human-machine interface, and a digital twin for
object-specific data.

Figure 2: Overview of the system components and their communication

We use the Asset Administration Shell (AAS) [36] to represent digital
industrial assets. The AAS metamodel is documented in [37] and the HTTP/REST
API is documented in [38]. Digital twins according to the AAS specifications
can be created using available open-source AASX tools from the Industrial
Digital Twin Association, i.e., the AASX Package Explorer and the AASX
Server111https://github.com/admin-shell-io [accessed: 15-03-2023]. Given an
object’s serial number, object-specific information can be accessed via the
AASX server’s REST API using the HTTP GET method and updated using the HTTP
PUT method. Based on the data from the AASX Server, we implemented the
following assistance functionalities for this case study:

####  IV-A1 Automatic Part Identification

Currently, the shop floor workers must manually read the serial number on each
turbine blade and then look it up in a paper file, which is slow and prone to
error. Therefore, we added a barcode scanner for automatic part
identification. Specifically, we use 2D Data Matrix codes [39] to encode the
serial number, which is applied onto each turbine blade via direct part
marking. Each time a Data Matrix code is scanned, its serial number is
published to the assistance system in a ROS topic. The serial number is then
used to access data from the AASX Server.

####  IV-A2 Digital Work Instructions

We developed a web application to display digital work instructions on the
touchscreen. In general, the web application first provides an overview of the
open tasks for the given serial number, then detailed information about the
selected task, and the ability to provide digital documentation at the end of
the task. The developed web application is depicted in Fig. 11 in the
appendix.

After scanning a serial number, defects for the current object are shown in a
list. The task of each metalworker in this case study is to repair all defects
classified as open. All instructions are linked to the serial number of the
current object and are dynamically requested using the REST API of the AASX
server. The digital work instructions include the following core features:

  * •

an interactive 3D model of the current object with spatial information about
defects, zones, and residual wall thickness measurements,

  * •

detailed text information about the selected defect, such as its type, length,
and additional comments,

  * •

opening PDF documents linked to the serial number,

  * •

a glossary for common abbreviations, and

  * •

digital documentation of the performed repair task.

####  IV-A3 Screen-Based Augmented Reality

According to the pinhole camera model, the relationship between a 3D point
$\tilde{\mathbf{M}}=[X,Y,Z,1]^{T}$ and its 2D image projection
$\tilde{\mathbf{m}}=[u,v,1]^{T}$ is given by Eq. 1, where $s$ is an arbitrary
scale factor, $\mathbf{A}$ is the camera intrinsic matrix and
$(\mathbf{R},~{}\mathbf{t})$ is the extrinsic rotation and translation between
world coordinate system and camera coordinate system [40].

$s\tilde{\mathbf{m}}=\mathbf{A}\left[\mathbf{R}~{}\mathbf{t}\right]\tilde{\mathbf{M}}$
(1)

Based on Eq. 1, we use the top-mounted RGB camera for screen-based AR by
projecting three-dimensional AR objects into the live camera image. In
general, we use AR to help the worker locate spatial information about the
object being repaired. For the manual repair of turbine blades, we superimpose
defects, zones, and residual wall thickness measurements on the live image of
the work area, see Fig. 3. The user can toggle which information is displayed.

Figure 3: Screen-based AR superimposing a defect (left), its zone (middle) and
nearby wall thickness measurements (right) on the live camera image

Each time the object being repaired is moved, we have to recompute the
extrinsic parameters $(\mathbf{R},~{}\mathbf{t})$. This problem is known as
object pose estimation. Using homogeneous transforms $\mathbf{T}$, we can
rewrite Eq. 1 to Eq. 2 with a dynamic transformation $\mathbf{T}_{O}^{W}$ from
object coordinates to world coordinates and a static transformation
$\mathbf{T}_{W}^{C_{1}}$ from world coordinates to the top-mounted camera
coordinates. The camera intrinsic matrix $\mathbf{A}$ and the transformation
$\mathbf{T}_{W}^{C_{1}}$ can be obtained using standard camera calibration
techniques [40].

$s\tilde{\mathbf{m}}=\mathbf{A}\begin{bmatrix}1&0&0&0\\\ 0&1&0&0\\\
0&0&1&0\end{bmatrix}\mathbf{T}_{W}^{C_{1}}~{}\mathbf{T}_{O}^{W}~{}\tilde{\mathbf{M}}$
(2)

To obtain the transformation $\mathbf{T}_{O}^{W}$, we use the markerless pose
estimation process depicted in Fig. 4. A new pose estimation is triggered by
the user’s hands leaving the work area. Then, the object is first localized
using 2D object detection on RGB images [41]. The obtained bounding boxes are
used to crop the point cloud data from both 3D cameras. Based on the cropped
point cloud, the 6D object pose $\mathbf{T}_{O}^{W}$ is estimated using point
pair feature matching [42] and then refined using the point-to-plane iterative
closest point algorithm [43].

Figure 4: Overview of the pose estimation process

###  IV-B User Study

####  IV-B1 Participants

In order to perform the evaluated manual repair task, study participants had
to be professionals with a certain level of work experience. Therefore, a
total of ten male metalworkers with a median age of 46 years, ranging from 33
to 57 years, were recruited to participate in the user study. All participants
were experienced in the grinding repair process of turbine blades, with a
median work experience of 14 years, ranging from a minimum of 3 years to a
maximum of 28 years.

####  IV-B2 Study Design

In a between-subjects experimental study design, each participant is assigned
to a single group, such as a control group or the CAS group. In a within-
subjects study design, participants are assigned to multiple groups. Because
within-subjects study designs usually have greater statistical power, they
require fewer participants [44].

Because of the limited number of study participants, a within-subjects study
design was performed, i.e., each participant used both systems (paper file and
assistance system). To reduce practice effects, the sequence of the two
systems was counterbalanced. That is, the first participant started with the
CAS and then performed the same task with the paper file. The second
participant started with the paper file and then used the CAS.

In the first phase, all participants watched a four-minute-long introduction
video explaining all the relevant features of the CAS. Afterward, each
participant had up to ten minutes to get used to working with the assistance
system. Next, all participants were given up to ten minutes to check the
paper-based work instructions contained in a file. Because the file was
designed to replicate their current work instructions, most participants did
not require much time. During this phase, the CAS and the paper-based file did
not include the relevant defects of the evaluation.

In the second phase, each participant was asked to complete the given task
according to Subsection IV-B3 consecutively with both systems. Because the
given repair task requires mostly cognitive reasoning, and it was not feasible
to physically repair real defects, participants were asked to realistically
indicate the physical grinding process with unplugged tools. During the task,
we measured the TCT for each turbine blade as a dependent variable. This phase
was recorded on video with sound for further evaluation.

In the final phase, each participant was asked to fill out standardized
questionnaires for each system and provide feedback on the CAS. We used the
NASA-TLX questionnaire [45] to measure cognitive load and the UMUX-LITE
questionnaire [46] to measure system usability.

####  IV-B3 Task

The aim of the given task was to simulate a familiar real-world repair job of
multiple turbine blades. To this end, we visited the shop floor and reviewed
the metalworker’s labor practices and required documents.

For this study, participants were given three different serial numbers with
one defect each. The participants were asked to repair any assigned defects on
the surface of the three corresponding turbine blades. The necessary amount of
material that has to be removed depends on the defect type, its assigned zone,
and nearby residual wall thickness measurements. Additionally, they were asked
to document their work and the required amount of time. All required
information was either contained in a paper file or provided digitally by the
assistance system. The study set-up for the paper file is depicted in Fig.
5(a) and the CAS is shown in Fig. 5(b). For a more detailed workflow for both
systems, see Figs. 10 and 11 in the appendix.

(a) Set-up with the well-known paper file

(b) Set-up with the new digital assistance system

Figure 5: Study set-up for both systems

To be consistent with the participants’ usual work routine, we did not impose
step-by-step work instructions for the task. To complete the given task, all
workers performed the actions depicted in Fig. 6 based on their work
experience.

Figure 6: Observed workflow of the given task for three turbine blades

##  V Results

###  V-A Task Completion Time

Overall, 7 out of 10 participants completed the task faster with the CAS
compared to the paper file. The time of the paper-based workflow was strongly
dependent on whether action 3 and action 4 were carried out or skipped.
Additionally, we observed that finding the defect information page in the
paper file (action 2) often took a long time.

For the CAS, time was often lost due to the unfamiliar navigation of the
digital work instructions. Furthermore, after moving the turbine blade,
participants had to wait each time roughly one second for the pose estimation
of the AR.

The mean results for TCT are shown in Fig. 2 with 95 % confidence intervals
(CI). With the CAS, participants were on average 21.2 % faster than the
participants with the paper file.

Figure 7: Average TCT for both systems with 95 % CI222Within-subject
confidence intervals were calculated according to [47]

We performed a paired t-test to compare the overall TCT for both systems
(significance level $\alpha=0.05$). The test’s normality assumption was
checked with the Shapiro-Wilk test. Results of the two-tailed paired t-test
indicate that there is a non-significant medium difference between the paper
file ($M=466.1$, $SD=171.3$), and the CAS ($M=367.3$, $SD=95.5$), $t(9)=2.2$,
$p=0.054$. According to Cohen’s $d_{z}$, which measures the standardized mean
difference [48], the effect size of the CAS on the TCT was medium with
$d_{z}=0.7$.

###  V-B Perceived Workload

The perceived workload was measured with the NASA Task Load Index
questionnaire [45]. A meta-analysis of 556 studies [49] found that the average
score for this questionnaire is at RTLX = 42.

Fig. 2 shows the RTLX values for both systems. Using the CAS, participants
reported on average a 26.8 % reduction in perceived workload compared to the
paper-based workflow. We statistically compared RTLX values for both systems
with a paired t-test ($\alpha=0.05$). The test’s normality assumption was
checked with the Shapiro-Wilk test. Results of the two-tailed paired t-test
indicate that there is a significant large difference between the paper file
($M=44.4$, $SD=16.2$) and the CAS ($M=32.5$, $SD=12.6$), $t(9)=3.1$,
$p=0.012$. The effect size of the CAS on the perceived workload was large with
Cohen’s $d_{z}=0.9$.

Figure 8: Perceived workload according to NASA-RTLX for both systems with 95 %
CI22footnotemark: 2

###  V-C System Usability

To rate the usability of both systems, we used the UMUX-LITE [46]
questionnaire. UMUX-LITE is a short questionnaire that rates the perceived
usefulness (PU) and the perceived ease of use (PEU) of a system on a seven-
point Likert scale and then combines them to a score ranging from 0 to 100.
Table I shows the UMUX-LITE scores and the corresponding school grades on the
Sauro/Lewis curved grading scale (CGS) [50].

The UMUX-LITE shows poor ratings for the paper-based workflow, both for PU and
PEU. The CAS was perceived as both very useful and easy to use. During the
feedback, all ten participants stated that they would prefer to use the CAS
over the paper file in production.

TABLE I: Average UMUX-LITE scores for the usability of both systems | PU | PEU | UMUX-LITE | CGS Grade
---|---|---|---|---
Paper | 3.9 | 4.8 | 55.8 | D
CAS | 6.2 | 6.1 | 85.8 | A+

###  V-D Ranking of Individual Assistance Functions

We asked all participants to rate each implemented assistance function
according to how important it was to them. The average results from the five-
point Likert scale are presented in Fig. 9. All assistance functions were
rated above important. The most important function was to see the defects on
the touchscreen and on the AR display. Although the glossary was not used by
any participants during the task, they rated the feature as nice to have and
important for new employees.

Figure 9: Average ranking of individual assistance functions on a five-point
Likert scale

##  VI Discussion

The results from Section V showed better efficiency, perceived workload, and
usability with the CAS, compared to the paper file. However, results from the
user study are limited to experienced male metalworkers with a median age of
46. We did not test the CAS on workers with less than three years of work
experience.

Regarding RQ 1, the TCT was reduced by an average of 21.2 % for a familiar
turbine blade repair task by experienced workers using the CAS. The paired
t-test showed a non-significant medium effect for the time difference between
the CAS and the paper file. We observed that some participants skipped action
3 and action 4 for the paper-based workflow, i.e., they did not check zoning
information and residual wall thickness measurements in the paper file (see
Figs. 10(b), 10(c), and 10(d)). Action 3 can be skipped with enough work
experience, but neglecting the measurements from action 4 can potentially lead
to errors. In comparison, the CAS always displayed all required information by
default. Therefore, we presume that the CAS will reduce manual errors in the
long term.

The group of participants had a median work experience of 14 years repairing
turbine blades with the paper file, whereas the time to practice the usage of
the CAS was limited to a maximum of ten minutes during the user study. We
observed that some participants lost time during the task due to the
unfamiliar navigation of the digital work instructions. To this effect,
multiple participants stated that if they had more time to get used to working
with the assistance system, the CAS would be much faster than their current
paper-based workflow.

Because all participants had to have a certain level of work experience in the
evaluated manual repair task, we were unable to recruit more than ten study
participants. The small sample size of ten resulted in a low statistical power
of the conducted t-tests. The statistical power is the probability to reject a
null hypothesis, i.e., the probability to find a statistically significant
difference in the observed groups [48]. Statistically significant results for
the TCT might be found by repeating the study with a bigger sample size, or by
giving participants more time to get used to working with the CAS.

As for RQ 2, according to NASA-TLX, the perceived workload was reduced by an
average of 26.8 % with the CAS. The paired t-test showed a significant large
effect for the RTLX difference between the CAS and the paper-based workflow.
Feedback from participants indicated that searching for documents in the paper
file was a major problem that could be eliminated through the use of the CAS.
Furthermore, participants stated that evaluating the wall thickness
measurement table (Figs. 10(c) and 10(d)) is very challenging, whereas the CAS
was easy to use in this regard. This might be the reason action 4 was often
skipped in the paper-based workflow.

All three AR functions were rated by the participants on average between
important and very important. However, we observed that some participants
mostly ignored the screen-based AR during the task and focused on the
touchscreen with the 3D model instead. During the feedback, they stated that
the AR technology was very new to them. Looking at the turbine blade’s 3D
model on the touchscreen monitor was more familiar to them. The AR assistance
functions might be more useful during a real grinding process. In the present
user study, participants could only indicate the physical repair process.

##  VII Conclusion

All ten participants preferred the CAS over their current paper-based
workflow. The UMUX-LITE ratings showed better perceived usefulness and
perceived ease of use for the CAS. Additionally, all assistance functions were
rated on average between important and very important.

In conclusion, the user study showed that the developed CAS can reduce the
high-cost manual repair time for turbine blades of experienced metalworkers.
Additionally, the digital work instructions with AR reduced the perceived
workload. Furthermore, we could observe that the participants skipped
inconvenient work steps with the paper file. With the assistance system, on
the other hand, all the required information was conveniently displayed by
default.

In future work, we would like to measure long-term effects by repeating the
study after the workers had more time to get used to working with the
assistance system. Additionally, we would like to test if the assistance
system has a positive effect on the learning curve and the number of errors of
new employees. We hypothesize that new employees who are not yet familiar with
the paper file might learn the turbine blade repair process faster with the
assistance system than without it. Finally, we would like to take a closer
look at the advantages of the AR display compared to the digital work
instructions shown on the touchscreen.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Acknowledgements

This work was part of the project MRO 2.0 - Maintenance, Repair, and Overhaul
(ProFIT-10167454) and was supported in part by the European Regional
Development Fund (ERDF).

We would like to thank our project partners from Siemens Energy, Gestalt
Robotics, YOUSE, and Fraunhofer Institute for Production Systems and Design
Technology for their work in the development process of the assistance system.

## References

  * [1] D. Gorecky, M. Schmitt, M. Loskyll, and D. Zuhlke, “Human-machine-interaction in the industry 4.0 era,” in _2014 12th IEEE International Conference on Industrial Informatics (INDIN)_. IEEE, 2014.
  * [2] B. Brinzer and K. Schneider, “Complexity assessment in production: linking complexity drivers and effects,” _Procedia CIRP_ , vol. 93, pp. 694–699, 2020.
  * [3] B. Alkan, D. A. Vera, M. Ahmad, B. Ahmad, and R. Harrison, “Complexity in manufacturing systems and its measures: a literature review,” _European J. of Industrial Engineering_ , vol. 12, no. 1, p. 116, 2018.
  * [4] D. Romero, P. Bernus, O. Noran, J. Stahre, and Å. Fast-Berglund, “The operator 4.0: Human cyber-physical systems & adaptive automation towards human-automation symbiosis work systems,” in _Advances in Production Management Systems. Initiatives for a Sustainable World_ , I. Nääs, O. Vendrametto, J. Mendes Reis, R. F. Gonçalves, M. T. Silva, G. von Cieminski, and D. Kiritsis, Eds. Cham: Springer International Publishing, 2016, pp. 677–686.
  * [5] B. G. Mark, E. Rauch, and D. T. Matt, “Industrial assistance systems to enhance human–machine interaction and operator’s capabilities in assembly,” in _Implementing Industry 4.0 in SMEs_. Springer International Publishing, 2021, pp. 129–161.
  * [6] B. J. M. Heng, A. K. Ng, and R. K. H. Tay, “Digitization of work instructions and checklists for improved data management and work productivity,” in _2019 4th International Conference on Intelligent Transportation Engineering (ICITE)_. IEEE, 2019.
  * [7] N. F. S. Jeffri and D. R. A. Rambli, “A review of augmented reality systems and their effects on mental workload and task performance,” _Heliyon_ , vol. 7, no. 3, 2021.
  * [8] J. Sweller, P. Ayres, and S. Kalyuga, “The split-attention effect,” in _Cognitive Load Theory_. Springer New York, 2011, pp. 111–128.
  * [9] D. Dixon, U. Terton, and R. Greenaway, “Reducing the split-attention effect in assembly based instruction by merging physical parts with holograms in mixed reality.” in _CSEDU (1)_ , 2018, pp. 235–244.
  * [10] E. Bottani and G. Vignali, “Augmented reality technology in the manufacturing industry: A review of the last decade,” _IISE Transactions_ , vol. 51, no. 3, pp. 284–310, feb 2019.
  * [11] J. Egger and T. Masood, “Augmented reality in support of intelligent manufacturing – a systematic literature review,” _Computers & Industrial Engineering_, vol. 140, p. 106195, 2020.
  * [12] L. F. de Souza Cardoso, F. C. M. Q. Mariano, and E. R. Zorzal, “A survey of industrial augmented reality,” _Computers & Industrial Engineering_, vol. 139, p. 106159, 2020.
  * [13] R. Lindorfer, R. Froschauer, and G. Schwarz, “ADAPT - a decision-model-based approach for modeling collaborative assembly and manufacturing tasks,” in _2018 IEEE 16th International Conference on Industrial Informatics (INDIN)_. IEEE, 2018.
  * [14] F. Quint, F. Loch, M. Orfgen, and D. Zuehlke, “A system architecture for assistance in manual tasks,” _Ambient Intelligence and Smart Environments_ , vol. 21, pp. 43–52, 2016.
  * [15] J. Geng, X. Song, Y. Pan, J. Tang, Y. Liu, D. Zhao, and Y. Ma, “A systematic design method of adaptive augmented reality work instruction for complex industrial operations,” _Computers in Industry_ , vol. 119, no. 103229, 2020\.
  * [16] M. Esposito, M. Lazoi, A. Margarito, and L. Quarta, “Innovating the maintenance repair and overhaul phase through digitalization,” _Aerospace_ , vol. 6, no. 5, p. 53, 2019.
  * [17] P. Bertram, C. Kränzler, P. Rübel, and M. Ruskowski, “Development of a context-aware assistive system for manual repair processes - a combination of probabilistic and deterministic approaches,” _Procedia Manufacturing_ , vol. 51, pp. 598–604, 2020.
  * [18] L. Eversberg, P. Ebrahimi, M. Pape, and J. Lambrecht, “A cognitive assistance system with augmented reality for manual repair tasks with high variability based on the digital twin,” _Manufacturing Letters_ , vol. 34, pp. 49–52, 2022.
  * [19] J. Illing, P. Klinke, U. Grünefeld, M. Pfingsthorn, and W. Heuten, “Time is money! evaluating augmented reality instructions for time-critical assembly tasks,” in _19th International Conference on Mobile and Ubiquitous Multimedia_. ACM, 2020.
  * [20] F. Ababsa, “Augmented reality application in manufacturing industry: Maintenance and non-destructive testing (NDT) use cases,” in _Augmented Reality, Virtual Reality, and Computer Graphics_ , L. T. De Paolis and P. Bourdot, Eds. Cham: Springer International Publishing, 2020, pp. 333–344.
  * [21] M. Funk, T. Kosch, and A. Schmidt, “Interactive worker assistance: comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions,” in _Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing_. ACM, 2016.
  * [22] C.-H. Chu, C.-J. Liao, and S.-C. Lin, “Comparing augmented reality-assisted assembly functions—a case study on dougong structure,” _Applied Sciences_ , vol. 10, no. 10, p. 3383, 2020.
  * [23] L. Hou, X. Wang, L. Bernold, and P. E. D. Love, “Using animated augmented reality to cognitively guide assembly,” _Journal of Computing in Civil Engineering_ , vol. 27, no. 5, pp. 439–451, 2013.
  * [24] L. Kästner, L. Eversberg, M. Mursa, and J. Lambrecht, “Integrative object and pose to task detection for an augmented-reality-based human assistance system using neural networks,” in _2020 IEEE Eighth International Conference on Communications and Electronics (ICCE)_ , 2021, pp. 332–337.
  * [25] M. Funk, L. Lischke, S. Mayer, A. S. Shirazi, and A. Schmidt, “Teach me how! interactive assembly instructions using demonstration and in-situ projection,” in _Assistive Augmentation_. Springer Singapore, 2017, pp. 49–73.
  * [26] M. Funk, A. Bächler, L. Bächler, T. Kosch, T. Heidenreich, and A. Schmidt, “Working with augmented reality?” in _Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments_. ACM, 2017.
  * [27] Z.-H. Lai, W. Tao, M. C. Leu, and Z. Yin, “Smart augmented reality instructional system for mechanical assembly towards worker-centered intelligent manufacturing,” _Journal of Manufacturing Systems_ , vol. 55, pp. 69–81, 2020.
  * [28] A. E. Uva, M. Fiorentino, M. Gattullo, M. Colaprico, M. F. de Ruvo, F. Marino, G. F. Trotta, V. M. Manghisi, A. Boccaccio, V. Bevilacqua, and G. Monno, “Design of a projective AR workbench for manual working stations,” in _Lecture Notes in Computer Science_. Springer International Publishing, 2016, pp. 358–367.
  * [29] A. E. Uva, M. Gattullo, V. M. Manghisi, D. Spagnulo, G. L. Cascella, and M. Fiorentino, “Evaluating the effectiveness of spatial augmented reality in smart manufacturing: a solution for manual working stations,” _The International Journal of Advanced Manufacturing Technology_ , vol. 94, no. 1-4, pp. 509–521, 2017.
  * [30] V. Havard, D. Baudry, B. Jeanne, A. Louis, and X. Savatier, “A use case study comparing augmented reality (AR) and electronic document-based maintenance instructions considering tasks complexity and operator competency level,” _Virtual Reality_ , 2021.
  * [31] P. Vanneste, Y. Huang, J. Y. Park, F. Cornillie, B. Decloedt, and W. V. den Noortgate, “Cognitive support for assembly operations by means of augmented reality: an exploratory study,” _International Journal of Human-Computer Studies_ , vol. 143, p. 102480, 2020.
  * [32] M. Fiorentino, A. E. Uva, M. Gattullo, S. Debernardis, and G. Monno, “Augmented reality on large screen for interactive maintenance instructions,” _Computers in Industry_ , vol. 65, no. 2, pp. 270–278, 2014\.
  * [33] S. Henderson and S. Feiner, “Exploring the benefits of augmented reality documentation for maintenance and repair,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 17, no. 10, pp. 1355–1368, 2011.
  * [34] F. Obermair, J. Althaler, U. Seiler, P. Zeilinger, A. Lechner, L. Pfaffeneder, M. Richter, and J. Wolfartsberger, “Maintenance with augmented reality remote support in comparison to paper-based instructions: Experiment and analysis,” in _2020 IEEE 7th International Conference on Industrial Engineering and Applications (ICIEA)_. IEEE, 2020.
  * [35] L. Merino, M. Schwarzl, M. Kraus, M. Sedlmair, D. Schmalstieg, and D. Weiskopf, “Evaluating mixed and augmented reality: A systematic literature review (2009-2019),” in _2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)_. IEEE, 2020\.
  * [36] J. Neidig, A. Orzelski, and S. Pollmeier, _Asset Administration Shell Reading Guide_ , Plattform Industrie 4.0 and IDTA, 2022. [Online]. Available: https://www.plattform-i40.de/IP/Redaktion/DE/Downloads/Publikation/AAS-ReadingGuide_202201.html
  * [37] Plattform Industrie 4.0, “Details of the asset administration shell - part 1\. the exchange of information between partners in the value chain of industrie 4.0 (version 3.0rc02),” Spezifikation, 2022. [Online]. Available: https://www.plattform-i40.de/IP/Redaktion/DE/Downloads/Publikation/Details_of_the_Asset_Administration_Shell_Part1_V3.pdf?__blob=publicationFile&v=10
  * [38] ——, “Details of the asset administration shell - part 2. interoperability at runtime – exchanging information via application programming interfaces (version 1.0rc02),” Spezifikation, 2021. [Online]. Available: https://www.plattform-i40.de/IP/Redaktion/DE/Downloads/Publikation/Details_of_the_Asset_Administration_Shell_Part_2_V1.pdf?__blob=publicationFile&v=7
  * [39] ISO/IEC 16022:2006-09, “Information technology - automatic identification and data capture techniques - data matrix bar code symbology specification,” International Organization for Standardization, 2006.
  * [40] Z. Zhang, “A flexible new technique for camera calibration,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 22, no. 11, pp. 1330–1334, 2000.
  * [41] L. Eversberg and J. Lambrecht, “Generating images with physics-based rendering for an industrial object detection task: Realism versus domain randomization,” _Sensors_ , vol. 21, no. 23, 2021.
  * [42] B. Drost, M. Ulrich, N. Navab, and S. Ilic, “Model globally, match locally: Efficient and robust 3D object recognition,” in _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_. IEEE, 2010, pp. 998–1005.
  * [43] Y. Chen and G. Medioni, “Object modelling by registration of multiple range images,” _Image and Vision Computing_ , vol. 10, no. 3, pp. 145–155, 1992\.
  * [44] G. Keren, “Between- or within-subjects design: A methodological dilemma,” in _A Handbook for Data Analysis in the Behaviorial Sciences_ , G. Keren and C. Lewis, Eds. Psychology Press, 1993, pp. 257–272.
  * [45] S. G. Hart and L. E. Staveland, “Development of NASA-TLX (task load index): Results of empirical and theoretical research,” in _Advances in Psychology_. Elsevier, 1988, pp. 139–183.
  * [46] J. R. Lewis, B. S. Utesch, and D. E. Maher, “UMUX-LITE,” in _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. ACM, 2013.
  * [47] D. Cousineau and F. O’Brien, “Error bars in within-subject designs: a comment on baguley (2012),” _Behavior Research Methods_ , vol. 46, no. 4, pp. 1149–1151, 2014.
  * [48] J. Cohen, _Statistical power analysis for the behavioral sciences_. L. Erlbaum Associates, 1988.
  * [49] M. Hertzum, “Reference values and subscale patterns for the task load index (TLX): a meta-analytic review,” _Ergonomics_ , vol. 64, no. 7, pp. 869–878, 2021.
  * [50] J. R. Lewis, “Measuring perceived usability: SUS, UMUX, and CSUQ ratings for four everyday products,” _International Journal of Human–Computer Interaction_ , vol. 35, no. 15, pp. 1404–1419, 2018\.

(a) Defect information for one specific turbine blade (action 2)

(b) Zoning documentation for the product family (action 3)

(c) Wall thickness measurement spots for the product family (action 4)

(d) Wall thickness measurements for multiple serial numbers (action 4)

(e) User input to document finished repair work for one turbine blade (action
6)

Figure 10: Repairing a turbine blade with the established paper file

(a) Overview of all defects for one turbine blade (action 2)

(b) Digital work instructions for the selected defect (actions 2, 3, 4)

(c) User input to document performed repair work (action 6)

(d) AR display with spatially registered work instructions (action 2, 3, 4)

Figure 11: Repairing a turbine blade with the CAS


11institutetext: Faculty of Mathematics and Computer Science, Jagiellonian
University,
Kraków, Poland
22institutetext: Mila - Quebec AI Institute, Montreal, Quebec, Canada
33institutetext: Université de Montréal, Montreal, Quebec, Canada
33email<EMAIL_ADDRESS>
# ChiENN: Embracing Molecular Chirality with Graph Neural Networks

Piotr Gaiński(✉) 11 Michał Koziarski 22 3 3 Jacek Tabor 11 Marek Śmieja 11

###### Abstract

Graph Neural Networks (GNNs) play a fundamental role in many deep learning
problems, in particular in cheminformatics. However, typical GNNs cannot
capture the concept of chirality, which means they do not distinguish between
the 3D graph of a chemical compound and its mirror image (enantiomer). The
ability to distinguish between enantiomers is important especially in drug
discovery because enantiomers can have very distinct biochemical properties.
In this paper, we propose a theoretically justified message-passing scheme,
which makes GNNs sensitive to the order of node neighbors. We apply that
general concept in the context of molecular chirality to construct Chiral Edge
Neural Network (ChiENN) layer which can be appended to any GNN model to enable
chirality-awareness. Our experiments show that adding ChiENN layers to a GNN
outperforms current state-of-the-art methods in chiral-sensitive molecular
property prediction tasks.

###### Keywords:

Graph Neural Networks GNN Message-passing Chirality Molecular Property
Prediction

##  1 Introduction

Recent advances in Graph Neural Networks (GNNs) have revolutionized
cheminformatics and enabled learning the molecular representation directly
from chemical structures [9, 30]. GNNs are widely used in molecular property
prediction [34, 20, 36, 31], synthesis prediction [17, 3], molecule generation
[21, 2, 22], or conformer generation [19, 11, 33, 6]. Surprisingly, typical
GNNs cannot capture the concept of chirality, roughly meaning they do not
distinguish between a molecule and its mirror image, called enantiomer (see
1). Although enantiomers share many physical, chemical, and biological
properties, they may behave remarkably differently when interacting with other
chiral molecules, e.g. chiral proteins. For this reason, capturing chirality
is critical in the context of drug design [23, 12, 10, 25, 16] and should not
be ignored by the design of GNN architecture.

Figure 1: An example of a chiral molecule (left) and its mirror image (right).

A chiral molecule is a molecule with at least one chiral center which is
usually a carbon atom with four non-equivalent constituents. The mirror image
of a chiral molecule, called an enantiomer, cannot be superposed back to the
original molecule by any combination of rotations, translations, and
conformational changes (see 1). Therefore, enantiomers are molecules with
different bond arrangements and the same graph connectivity. There are many
examples of chiral drugs used in pharmacy whose enantiomers cause
substantially different effects [23]. For instance (S)-penicillamine is an
antiarthritic drug while its enantiomer (R)-penicillamine is extremely toxic
[15]. Actually, chirality can be a characteristic of any class of graphs
embedded in euclidean space (where we have an intuitive notion of reflection).
For instance, 2 shows two 2D road maps that are mirror images of each other
and possess different properties. For this reason, modeling chirality in GNNs
is not restricted to the chemical domain.

Figure 2: An illustration of a road map (left) and its mirror image (right).
We see that the maps share the same connectivity between cities, however, to
get from city $A$ to city $B$ one has to take the second exit on a roundabout
$D$ for the left map, and the first exit for the right map.

In this paper, we propose and theoretically justify a novel order-sensitive
message-passing scheme, which makes GNNs sensitive to chirality. In contrast
to existing methods of embracing chirality, our framework is not domain
specific and does not rely on arbitrary chiral tagging or torsion angles (see
2). The only inductive bias our method introduces to a GNN is the dependency
on the orientation of the neighbors around a node, which lies at the core of
chirality. The key component of the proposed framework is the message
aggregation function. In a typical GNN, the messages incoming to a node from
its neighbors are treated as a set and aggregated with a permutation-invariant
function (sum, max, etc.). It makes the model unable to distinguish between
chiral graphs with the same connectivity, but with different spatial
arrangements. We re-invent this approach and introduce a message aggregation
function that is sensitive to the spatial arrangement (order) of the
neighbors. Our approach can be used in any chiral-sensitive graph domain where
chirality can be expressed by an order of the neighboring nodes. We apply that
general order-sensitive message-passing framework in the context of molecular
chirality to construct Chiral Edge Neural Network (ChiENN) layer. The ChiENN
layer can be appended to most molecular GNN models to enable chirality
sensitiveness. Our experiments show that ChiENN can be successfully used
within existing GNN models and as a standalone model consisting of stacked
ChiENN layers. In both cases, ChiENN outperforms current state-of-the-art
methods in chiral-sensitive molecular property prediction tasks by a large
margin. We make our code publicly available111https://github.com/gmum/ChiENN.
Our contributions are as follows:

  1. 1.

We propose and theoretically justify a general order-sensitive message-passing
scheme. Our method can be adapted to any chiral-sensitive graph domain where
chirality can be expressed by an order of the neighboring nodes (3).

  2. 2.

We use the proposed framework to construct a novel ChiENN layer that enables
chirality awareness in any GNN model in the domain of molecular graphs (4).
The proposed ChiENN can be applied to any 3D graph task with the notion of
chirality.

  3. 3.

We evaluate and analyze the ChiENN layer and show that it outperforms current
state-of-the-art methods in chiral-sensitive molecular property prediction
tasks (5).

##  2 Related Work

####  2.0.1 Explicit Tagging of Chiral Center.

The most common approach for incorporating chirality into GNN is to use local
or global chiral tags [27, 18, 13]. Both local and global tagging can be seen
in the following way. Every carbon atom with four non-equivalent constituents,
called a chiral center, is given a tag (CCW or CW) describing the orientation
of its constituents. The orientation is defined using the enumeration of
constituents computed by an arbitrary algorithm. The constituent with the
highest number (4) is positioned so that it points away from the observer. The
curve passing through the constituents with numbers 1, 2, and 3 respectively
determines a clockwise (CW) or counterclockwise (CCW) orientation of the
chiral center. Although enumeration algorithms for global and local tagging
differ (the latter is not explicitly used in practice), the expressivity of
both methods is limited, as we show in 5.

####  2.0.2 3D GNNs with torsion angles.

Some recent GNN models enrich graphs with 3D information, like distances
between atoms [4, 20], angles between bonds [28, 8], and torsion angles
between two bonds joined by another bond [5, 7]. As distances and angles are
invariant to chirality, the torsion angles (that are negated upon reflection)
are required for 3D GNN to express the chirality. However, even access to a
complete set of torsion angles does not guarantee expressivity in chiral-
sensitive tasks as shown in [1]. Torsion angles are sensitive to bond
rotations and can also be negated by the reflection of a non-chiral molecule.
In [1], the authors proposed the ChIRo model that instead of embedding single
torsion angles, embeds sets of torsion angles with a common bond. ChIRo is the
current state-of-the-art method for chiral-sensitive tasks. In contrast to
ChIRo, our proposed method does not incorporate distances, angles, or torsion
angles. It only relies on the orientation of neighbors around a node, making
it more general and easily adaptable to other chiral graph domains. Moreover,
our experiments show that the ChiENN layer outperforms ChIRo by a large margin
on chiral-sensitive molecular tasks (see 5).

####  2.0.3 Changing Aggregation Scheme.

The method most related to our approach is the Tetra-DMPNN model from [24]
which replaces a classic message-passing scheme with a chiral-sensitive one.
The proposed aggregation scheme is guided by local chiral tags, meaning that
it relies on some arbitrary rules for enumerating neighbors and cannot be
applied to nodes other than chiral centers. Moreover, the Tetra-DMPNN method
is computationally expensive and does not scale with the number of possible
neighbors of a chiral center, making the model useful only in the context of
chemistry. Our approach provides a general, efficient, and scalable chiral-
sensitive message passing and outperforms the Tetra-DMPNN on chiral-sensitive
molecular tasks by a large margin (see 5).

##  3 Order-Sensitive Message-Passing Scheme

####  3.0.1 Setting.

Let us consider a directed graph $G=(X,E)$ in which every node $x_{i}\in X$ is
represented by a $N$-dimensional encoding ($x_{i}\in\mathbb{R}^{N}$). Edge
$e_{ij}$ connects nodes $x_{i}$ and $x_{j}$ and is represented by
$M$-dimensional encoding ($e_{ij}\in\mathbb{R}^{M}$). In addition, we assume
that for every node $x\in X$, we are given an order $o$ of all its neighbors
$o=(x_{0},x_{1},\ldots,x_{d-1})$. The order of neighbors forms a sequence,
which stands in contrast to typical graphs, where neighbors are treated as an
unordered set. Given a permutation $\pi$ on $\\{0,1,\ldots,d-1\\}$, we assume
that two orders $o_{1}=(x_{0},\ldots,x_{d-1})$ and
$o_{2}=(x_{\pi(0)},\ldots,x_{\pi(d-1)})$ are equivalent if and only if $\pi$
is a shift i.e. $\pi(i)=(i+k)\ \mathrm{mod}\ d$, for a fixed $k\in\mathbb{Z}$.
In other words, the neighbors form the sequence on a ring. One of the most
common mechanisms in GNN is message-passing, which updates the representation
of a node $x$ by the information coming from its neighbors
$(x_{0},\ldots,x_{d-1})$, which can be written as:

$x^{\prime}=f(x;x_{0},\ldots,x_{d-1}).$

In this paper, we are going to describe the general message-passing scheme,
which is aware of the neighbors’ order. Before that, we discuss possible
choices of the aggregation function $f$.

####  3.0.2 Vanilla message-passing as a permutation-invariant transformation.

Let us first discuss a basic case, where $f$ is a permutation-invariant
function, i.e.

$f(x;x_{0},\ldots,x_{d-1})=f(x;x_{\pi(0)},\ldots,x_{\pi(d-1)}),$

for every permutation $\pi$ of $\\{0,1,\ldots,d-1\\}$. This aggregation
ignores the order of neighbors and lies in a heart of typical GNNs. Let us
recall that $f$ is permutation-invariant with respect to
$\\{x_{0},x_{1},\ldots,x_{d-1}\\}$ if and only if it can be decomposed in the
form [35]:

$f(x_{0},x_{1},\ldots,x_{d-1})=\rho(\sum_{i=0}^{d-1}\phi(x_{i})),$

for suitable transformations $\phi$ and $\rho$. In the context of graphs, a
general form of a permutation-invariant aggregation of neighbors
$\\{x_{0},x_{1},\ldots,x_{d-1}\\}$ of $x$ is:

$x^{\prime}=f(x;x_{0},\ldots,x_{d-1})=\rho(x;\sum_{i=0}^{d-1}\phi(x;x_{i})),$
(1)

for suitable transformations $\phi$ and $\rho$. By specifying $\rho,\phi$ as
neural networks, we get the basic formula of vanilla message-passing.

####  3.0.3 Shift-invariant aggregation.

Vanilla message-passing relies on permutation-invariant aggregation and it
does not take into account the neighbor’s order. Thus we are going to discuss
the weaker case of aggregation function $f$ and assume that $f$ is shift-
invariant, i.e.

$f(x;x_{0},\ldots,x_{d-1})=f(x;x_{0+p},\ldots,x_{d-1+p}),$

for any shift by a number $p\in{0,\ldots,d-1}$, where the additions on indices
are performed modulo $d$. This assumption is consistent with our initial
requirement that shifted orders are equivalent. The following theorem gives a
general formula for shift-invariant mappings.

###### Theorem 3.1

The function $f$ is shift-invariant if and only if $f$ can be written as:

$f(x_{0},\ldots,x_{d-1})=\sum_{p=0}^{d-1}g(x_{0+p},\ldots,x_{d-1+p})$

for an arbitrary function $g$.

###### Proof

If $f$ is shift invariant, then
$f(x_{0},\ldots,x_{d-1})=f(x_{0+p},\ldots,x_{d-1+p})$ for every $p$, and
consequently

$f(x_{0},\ldots,x_{d-1})=\sum_{p=0}^{d-1}\frac{1}{d}f(x_{0+p},\ldots,x_{d-1+p}).$

On the other hand, if the function $f$ can be written as
$\sum_{p=0}^{d-1}g(x_{0+p},\ldots,x_{d-1+p})$, then it is shift-invariant for
arbitrary function $g$.

Following the above theorem, we get a general formula for shift-invariant
aggregation applicable to graphs:

$x^{\prime}=\rho(x;\sum_{p=0}^{d-1}\psi(x;x_{{0+p}},\ldots,x_{d-1+p})),$ (2)

for suitable $\rho$ and $\psi$, where all additions are performed modulo $d$.
Now, we want to ensure that our function $f$ is not only shift-invariant but
also order-sensitive

####  3.0.4 Order-sensitive message-passing.

Let us assume that we are in the class of shift-invariant transformations. We
are going to specify the formula (2) to obtain ab aggregation, which is
sensitive to any permutation other than shift. More precisely, we say that $f$
is order-sensitive if and only if for every permutation $\pi$, we have:

$f(x;x_{0},\ldots,x_{d-1})=f(x;x_{\pi(0)},\ldots,x_{\pi(d-1)})\iff\pi(i)=(i+k)\
\mathrm{mod}\ d.$

Let us investigate typical functions $\psi$ in formula (2), which can be
implemented using neural networks. We start with the simplest case, where
$\psi$ is linear. Then

$\sum_{p=0}^{d-1}\psi(x_{0+p},\ldots,x_{d-1+p})=\sum_{p=0}^{d-1}\sum_{i=0}^{d-1}w_{i}x_{i+p}=\sum_{i=0}^{d-1}w_{i}\sum_{p=0}^{d-1}x_{i+p}=\sum_{i=0}^{d-1}w_{i}\sum_{j=0}^{d-1}x_{j},$

does not depend on the order of the neighbors $(x_{0},...,x_{d-1})$. To
construct more complex functions, we use an arbitrary Multi-Layer Perceptron
(MLP) as $\psi$. Since MLPs are universal approximators (for a sufficiently
large number of hidden units), we can find such parameters $\theta$ that
$\sum_{p=0}^{d-1}\psi_{\theta}(x_{\pi(0)+p},\ldots,x_{\pi(d-1)+p})$ returns a
different value for every permutation $\pi$ that is not a shift. Therefore our
aggregation scheme with $\psi$ given by MLP can learn order-sensitive mapping.
Following the above observations, we implement our order-sensitive message-
passing using MLP as $\psi$. To match our construction to various numbers of
neighbors in a graph, we restrict $\psi$ to be $k$-ary (denoted as $\psi^{k}$)
for some fixed $k>1$ and overload it so that:

$\psi^{k}(x_{0},\ldots,x_{d-1})=\psi^{k}(x_{0},\ldots,x_{d-1},\underbrace{0,\ldots,0)}_{k-d}\text{
for }d<k.$

. Given that, we implement the 2 with the following neural network layer:

$\begin{gathered}x^{\prime}=Wx+\sum_{p=0}^{d-1}\psi^{k}(x_{0+p},...,x_{k-1+p}),\\\
\psi^{k}(x_{0+p},...,x_{k-1+p})=W_{1}\sigma(W_{2}(x_{0+p}|...|x_{k-1+p})).\end{gathered}$
(3)

Our $k$-ary message function $\psi^{k}$ is composed of concatenation operator
$|$ and two-layer MLP with ELU as $\sigma$. Intuitively, the output of
$\psi^{k}(x_{0+p},...,x_{k-1+p})$ can be seen as a message obtained jointly
from $k$ consecutive neighbors starting from a neighbor $p$ in order
$(x_{0},...,x_{d-1})$ which is illustrated in 3.

Figure 3: An illustration of our update rule for node $x$ with 3 ordered
neighbors $(x_{0},x_{1},x_{2})$ and for $k=2$. We see that $\psi^{k}$ is used
to embed pairs of consecutive nodes.

##  4 ChiENN: Chiral-Aware Neural Network

In this section, we apply the order-sensitive message-passing framework to
molecular graphs. We show that order-sensitive aggregation is a key factor for
embracing molecular chirality. Roughly speaking, in contrast to vanilla
message-passing, the proposed ChiENN (Chiral-aware Edge Neural Network) is
able to distinguish enantiomers, where one molecule is a mirror image of the
second. Although we evaluate the ChiENN model in the context of molecular
property prediction, the proposed model can be applied to any 3D graph task
with the notion of chirality. To construct ChiENN based on our order-sensitive
message-passing scheme from 3, we need to define a notion of neighbors’ order
in molecular graphs that grasps the concept of chirality (see 4). We introduce
this notion of order for edge (dual) molecular graphs and provide a simple
transformation from standard molecular graphs to edge molecular graphs.
Therefore the rest of the section is organized into three subsections:

  1. 1.

Edge Graph describing the transformation from a molecular graph to its edge
(dual) form used in our ChiENN model,

  2. 2.

Neighbors Order defining the order of the neighbors in an edge graph,

  3. 3.

Chiral-Aware Update constructing order-sensitive update rule using our order-
sensitive framework from the 3.

Figure 4: An intuitive illustration of the neighbor ordering in a molecule.
First, we pick a directed bond from atom C to H and then order the rest of the
neighbors around that bond. We observe that for a chiral molecule (left) and
its mirror image (right), we obtain different orders of the $COOH$, $CH_{3}$,
and $OH$ constituents.

###  4.1 Edge Graph

Let us suppose, we have a directed graph $G=(X,C,E)$ that represents a
concrete conformation (3D embedding) of a molecule. The node encoding
$x_{i}\in X$ corresponds to an $i$-th atom from a molecule, $c_{i}\in
C\subseteq\mathbb{R}^{3}$ are its coordinates in 3D space, and the edge
encoding $e_{ij}\in E$ represents a bond between $i$-th and $j$-th atoms. To
make the definition of neighbor order straightforward, our ChiENN model
operates on an edge (dual) graph
$G^{\prime}=(X^{\prime},C^{\prime},E^{\prime})$ which swaps nodes with edges
from the original graph $G$. It means that the node $x_{ij}\in X^{\prime}$
represents the edge $e_{ij}\in E$, while the edge $e_{ij,jk}\in E^{\prime}$
represents the node $x_{j}$ that connects edge $e_{ij}\in E$ with $e_{jk}\in
E$. Similarly, $c^{\prime}_{ij}\in\mathbb{R}^{3}\times\mathbb{R}^{3}$ is now a
3D coordinate vector that links positions $c_{i}$ and $c_{j}$. Formally, we
have:

$\displaystyle X^{\prime}$ $\displaystyle=\\{x_{ij}=e_{ij}:e_{ij}\in E\\},$
$\displaystyle C^{\prime}$ $\displaystyle=\\{c_{ij}=c_{i}|c_{j}:c_{i},c_{j}\in
C,e_{ij}\in E\\},$ $\displaystyle E^{\prime}$
$\displaystyle=\\{e_{ij,jk}=e_{ij}|x_{j}|e_{jk}:e_{ij},e_{jk}\in E,x_{j}\in
X\\},$

where $|$ stands for a concatenation operator. Clearly, the constructed edge
graph $G^{\prime}=(X^{\prime},C^{\prime},E^{\prime})$ can be fed to any GNN
that can take as an input the original graph $G=(X,C,E)$.

###  4.2 Neighbors Order

In an edge molecular graph $G=(X,C,E)$, a node $x_{jk}\in E$ represents a
directed bond from atom $j$ to atom $k$ in the original molecule. It is
assigned with a 3D vector $c_{jk}\in
C\subseteq\mathbb{R}^{3}\times\mathbb{R}^{3}$ spanned from atom $j$ to atom
$k$. Therefore, we will sometimes refer to nodes as if they were 3D vectors.
Let us consider the node $x_{jk}$ and the set of its incoming neighbors:
$N(x_{jk})=\\{x_{i_{1}j},x_{i_{2}j},...,x_{i_{d}j}\\}$. By construction of
$G$, every node $x_{jk}$ has a corresponding parallel node $x_{kj}$. For
simplicity, we will treat this parallel node separately and exclude it from
the set of neighbors, i.e. $x_{kj}\notin N(x_{jk})$. The construction of the
neighbors $N(x_{jk})$ order is illustrated in 5 and consists of two steps:

  1. 1.

Transformation: first, we perform a sequence of 3D transformations on $x_{jk}$
and $N(x_{jk})$ to make $x_{jk}$ anchored to coordinate origin, perpendicular
to $yz$ plane and pointed away from the observer (see 5 b)).

  2. 2.

Sorting: second, we project the transformed neighbors $N(x_{jk})$ to the $yz$
plane and sort the projections by the angle to the $y$ axis.

Details of the above construction are presented in the supplementary
materials.

Figure 5: An illustration of ordering the neighbors $\\{x_{0},x_{1},x_{2}\\}$
of $x_{jk}$ for a chiral molecule (top) and its mirror image (bottom) around
the chiral center $j$. First, we perform a sequence of 3D transformations on
$x_{jk}$ and its neighbors to make $x_{jk}$ anchored to coordinate origin,
perpendicular to $yz$ plane and pointed away from the observer. Next, we
project the transformed neighbors to the $yz$ plane and sort the projections
by the angle to the $y$ axis. We see that for the chiral molecule (top) and
its mirror image (bottom), we obtained non-equivalent orders
$(x_{1},x_{0},x_{2})$ and $(x_{0},x_{1},x_{2})$.

Two observations can be made regarding the above construction:

###### Observation 1

The above construction returns non-equivalent orders for a chiral center and
its mirror image.

###### Observation 2

Any $SE(3)$ transformation of a molecule coordinates $C$ and any internal
rotation of its bonds (conformation) can only change the shift of the order
$o$, resulting in equivalent order $o^{\prime}$.

Therefore, the above construction grasps the notion of chirality in a molecule
and is additionally $SE(3)$\- and conformation-invariant. We artificially
excluded $x_{kj}$ from a set of $x_{jk}$ neighbors, because its parallel to
$x_{jk}$ and therefore its angle to $y$ axis after the sequence of
transformations is undefined. In theory, another neighbor $x_{ij}$ can also be
parallel to $x_{jk}$ and should also be excluded from the neighbor set, but we
have not observed such a case in our experiments and decided not to take it
into account.

###  4.3 Chiral-Aware Update

Once we transformed a molecular graph to edge (dual) molecular graph
$G=(X,E,C)$ using transformation from 4.1 and assigned every node $x_{jk}$
with an order of its neighbors $(x_{1},...,x_{d})$ using construction from
4.2, we can define the order-sensitive update rule of our ChiENN model:

$\begin{gathered}x_{jk}^{\prime}=W_{1}x_{jk}+W_{2}x_{kj}+\sum_{p=0}^{d-1}\psi^{k}(x_{0+p},...,x_{k-1+p}),\\\
\psi^{k}(x_{0+p},...,x_{k-1+p})=W_{3}\sigma(W_{4}(x_{0+p}|...|x_{k-1+p})),\end{gathered}$
(4)

where $\psi^{k}$ is $k$-ary message function and $\sigma$ is ELU non-linear
activation. The update rule is almost the same as that from 3, but here we add
a term that explicitly embeds $x_{kj}$ node, which was artificially excluded
from the order of the $x_{jk}$ neighbors.

##  5 Experiments

We compare ChiENN with several state-of-the-art models on a variety of chiral-
sensitive tasks. Details of experiments are described in Section 5.1, while
the results can be found in Section 5.2. Furthermore, to validate design
choices behind ChiENN we also conducted an ablation study, presented in
Section 5.3.

###  5.1 Set-up

####  5.1.1 Datasets.

We conduct our experiments on five different datasets affected by molecule
chirality. First, two datasets proposed in [1] which are designed specifically
to evaluate the capability of a model to express chirality: classification of
tetrahedral chiral centers as R/S (which should be a necessary, but not
sufficient, condition to learn meaningful representations of chiral
molecules); and enantiomer ranking, in which pairs of enantiomers with
enantioselective docking scores were selected, and the task was to predict
which molecule of the pair had a lower binding affinity in a chiral protein
pocket. Second, the binding affinity dataset, which is an extension of the
previously described enantiomer ranking, with the same underlying molecules,
but the task being regression of the binding affinity. Additionally, we take
two datasets from the MoleculeNet benchmark [32] that do not explicitly
require prediction of molecule chirality, but contain some percentage of
molecules with chiral centers, and the underlying biological task in principle
might be chirality-dependant: BACE, a binary classification dataset for
prediction of binding results for a set of inhibitors of human
$\beta$-secretase 1 (BACE-1) [29]; and Tox21, a multilabel classification
dataset containing qualitative toxicity measurements on 12 different targets,
including nuclear receptors and stress response pathways.

Table 1: Comparison of ChiENN-based approaches with the reference methods on chiral-sensitive tasks. Methods are split into groups by the underlying base model, except for the bottom group which includes models specifically designed to be chiral-sensitive. We bold the best results in every group and underline the best results across all groups. All variations of our method (ChiENN, SAN+ChiENN, and GPS+ChiENN) significantly outperform current state-of-the-art chiral-sensitive models. Note that for the R/S task, we omitted the results for models with chiral tags encoded in node features, for which the task is trivial. Model |  R/S |  Enantiomer |  Binding
---|---|---|---
|  ranking |  affinity
Accuracy $\uparrow$ |  R. Accuracy $\uparrow$ |  MAE $\downarrow$
DMPNN |  0.500±0.000 |  0.000±0.000 |  0.310±0.001
DMPNN+tags |  - |  0.701±0.003 |  0.285±0.001
GPS |  0.500±0.000 |  0.000±0.000 |  0.330±0.003
GPS+tags |  - |  0.669±0.037 |  0.318±0.004
GPS+ChiENN |  0.989±0.000 |  0.753±0.004 |  0.258±0.001
SAN |  0.500±0.000 |  0.000±0.000 |  0.317±0.004
SAN+tags |  - |  0.722±0.004 |  0.278±0.003
SAN+ChiENN |  0.987±0.001 |  0.764±0.005 |  0.257±0.002
ChIRo |  0.968±0.019 |  0.691±0.006 |  0.359±0.009
Tetra-DMPNN |  0.935 ±0.001 |  0.690±0.006 |  0.324±0.026
ChiENN |  0.989±0.000 |  0.760±0.002 |  0.275±0.003

####  5.1.2 Reference methods.

As reference models we consider several state-of-the-art neural network
architectures for processing graphs, both chirality-aware and general: GPS
[26], SAN [14], DMPNN [34], ChIRo [1], and Tetra-DMPNN [24]. For models not
designed to process chirality, that is DMPNN, GPS, and SAN, we additionally
considered their variants with chiral atom tags included in the node features,
similar to [1]. For the proposed approach we consider both a pure model
obtained by stacking several ChiENN layers, as well as combining ChiENN layers
with other architectures (ChiENN+GPS and ChiENN+SAN).

####  5.1.3 Training details.

All models were trained using Adam optimizer for up to 100 epochs, with a
cosine learning rate scheduler with 10 warm-up epochs and gradient norm
clipping, following the set-up of [26]. Cross-entropy and L1 loss functions
were used for classification and regression, respectively. Note that in
contrast to [1], to keep the set-up consistent across models we did not use
triplet margin loss for ChIRo, and observed worse results than reported in
[1]. We also performed a grid search with the identical budget (see 0.B.1).
For all datasets and models, we reported results averaged from three runs. For
enantiomer ranking, binding affinity, and R/S, we used data splits provided by
[1] and for BACE and Tox21, we used random splits with a train-valid-test
ratio of 7:1:2. For each model and dataset, we report mean results from 3
independent runs with the best parameters picked by grid search.

####  5.1.4 Evaluation.

Note that for the binding rank task we used accuracy modified with respect to
[1]. We required the difference between the predicted affinity of two
enantiomers to be higher than the threshold of 0.001. This led to ranking
accuracy being equal to 0 for models unable to distinguish chiral molecules.

###  5.2 Comparison with Reference Methods

In this section, we compare ChiENN-based networks with state-of-the-art
reference architectures using the experimental setting described in 5.1.

####  5.2.1 Chiral-sensitive tasks.

The results on chiral-sensitive tasks are presented in Table 1. For both the
enantiomer ranking and binding affinity, ChiENN-based approaches achieved the
best results, producing a significant improvement in performance over the
state-of-the-art chiral-aware architectures, that is ChIRo and Tetra-DMPNN.
For both GPS and SAN, there was a significant improvement in performance due
to the addition of ChiENN layers when compared to chiral tag inclusion. It
demonstates that ChiENN model can enable chiral-awareness demonstrating the
general usefulness of the proposed layer, and the fact that it can be combined
with a model preferred in a given task. Finally, as expected, all of the
chirality-aware methods can properly distinguish chiral centers in the R/S
task, while the baselines that do not capture the concept of chirality (DMPNN,
GPS and SAN) cannot. Note that for this task, we omitted the results for
models with chiral tags encoded in node features, for which the task is
trivial.

####  5.2.2 Remaining tasks.

The results on BACE and Tox21 tasks are in Table 2. We see that the ChiENN
model achieves results comparable to state-of-the-art models, however the
influence of chirality-sensitiveness on these tasks is not clear. For SAN we
actually observed a slight drop in performance when using ChiENN layers, and
for GPS the results remained roughly the same. The possible explanations for
that might be either 1) lack of importance of chirality on predicted tasks, or
2) small dataset size, leading to overfitting in presence of chiral
information. Our conclusion is that ChiENN layers significantly improve the
performance in chiral-sensitive tasks, and produce comparable results in the
other tasks, where the influence of chirality is not clear. We believe that
further investigation on the influence of chirality on the tasks commonly used
in the molecular property prediction domain would be beneficial and we leave
it for future work.

Table 2: Comparison of ChiENN-based approaches with the reference methods on tasks not explicitly requiring chirality. We see that the ChiENN model achieves results comparable to state-of-the-art models. Model |  BACE |  Tox21
---|---|---
AUC $\uparrow$ |  AUC $\uparrow$
DMPNN |  0.847±0.015 |  0.813±0.008
DMPNN+tags |  0.840±0.004 |  0.824±0.006
GPS |  0.841±0.004 |  0.821±0.000
GPS+tags |  0.812±0.017 |  0.825±0.002
GPS+ChiENN |  0.839±0.008 |  0.821±0.007
SAN |  0.846±0.012 |  0.842±0.007
SAN+tags |  0.829±0.009 |  0.841±0.004
SAN+ChiENN |  0.826±0.014 |  0.834±0.005
ChIRo |  0.815±0.010 |  0.847±0.005
Tetra-DMPNN |  0.824±0.017 |  0.807±0.003
ChiENN |  0.838±0.003 |  0.838±0.003

###  5.3 Ablation Studies

####  5.3.1 Comparison of $k$-ariness of the message function.

We began with an analysis of the impact of $k$-ariness (Equation 3) of the
message function used by ChiENN. Specifically, in this experiment, we used the
pure variant of ChiENN, which is a graph neural network using ChiENN layers as
message-passing layers. We varied $k\in\\{1,2,3\\}$, where $k=1$ disables the
ability of the network to distinguish enantiomers as it collapses our order-
sensitive message passing scheme from 3 to vanilla message-passing from 1. We
considered values of $k$ up to 3 since it corresponds to the airiness of
standard chiral centers observed in the edge graphs (see 4.1) of molecules.
The results are presented in Table 3. As expected, choosing $k=1$ leads to a
failure in distinguishing enantiomers (makes message passing permutation
invariant), as demonstrated by minimum performance in R/S and enantiomer
ranking tasks. Interestingly, for most datasets choosing $k=2$ was sufficient,
leading to a comparable performance to $k=3$. The only exception to that was
BACE dataset, for which a noticeable drop in performance was observed when
using $k=2$. We used $k=3$ in the remainder of this paper.

Table 3: Comparison of $k$-ariness of the message function. $k$-ary |  R/S |  Enantiomer |  Binding |  BACE |  Tox21
---|---|---|---|---|---
|  ranking |  affinity |  |
Accuracy $\uparrow$ |  R. Accuracy $\uparrow$ |  MAE $\downarrow$ |  AUC $\uparrow$ |  AUC $\uparrow$
$k=1$ |  0.500±0.000 |  0.000±0.000 |  0.328±0.000 |  0.831±0.028 |  0.833±0.005
$k=2$ |  0.989±0.001 |  0.759±0.003 |  0.267±0.001 |  0.788±0.014 |  0.836±0.004
$k=3$ |  0.989±0.000 |  0.760±0.002 |  0.275±0.003 |  0.838±0.003 |  0.838±0.003

####  5.3.2 Using ChiENN layer with existing models.

Secondly, we conducted an ablation of different design choices that can be
made to enable enantiomer recognition within the existing architectures.
Specifically, we focused on the GPS model and considered using three different
strategies: conversion to edge graph proposed in this paper, the inclusion of
chiral tags in the node features of the graph, and finally, replacement of
message passing layers with ChiENN layers. The results are presented in Table
4. Several observations can be made: first of all, in the case of R/S task, we
can see that both using chiral tags and the ChiENN layers allows us to
properly recognize chiral centers (and as stated before, due to the simplicity
of the task, good performance here is a necessary, but not sufficient,
requirement for learning meaningful chiral representations). Secondly, using
ChiENN layers significantly improves the performance in the enantiomer ranking
(explicitly requiring chirality) and binding affinity (implicitly requiring
it) tasks, more than simply including chiral tags. Interestingly, combining
chiral tags with edge graph transformation improves the performance compared
to using the tags alone (though not as much as using ChiENN layers),
suggesting that it might be a feasible general strategy. Finally, the results
on two remaining tasks, that is BACE and Tox21, for which the impact of
chirality is unclear, are less straightforward: in the case of BACE, GPS with
edge graph transformation achieves the best performance, and in the case of
Tox21, using both the edge graph transformation and including the chiral tags.
However, we can conclude that using ChiENN layers outperforms simply including
chiral tags in tasks requiring chirality, and have comparable performance to
the baseline GPS in other tasks.

Table 4: Ablation study of different design choices for GPS+ChiENN model. The ”Graph” column indicates conversion to the edge graph; the ”Tags” column indicates the inclusion of the chiral tags as node features; the ”ChiENN” column indicates the usage of ChiENN as a message-passing layer. Graph | Tags | ChiENN |  R/S |  Enantiomer |  Binding |  BACE |  Tox21
---|---|---|---|---|---|---|---
|  ranking |  affinity |  |
Accuracy $\uparrow$ |  R. Accuracy $\uparrow$ |  MAE $\downarrow$ |  AUC $\uparrow$ |  AUC $\uparrow$
No | No | No |  0.500±0.000 |  0.000±0.000 |  0.330±0.003 |  0.841±0.004 |  0.821±0.000
Yes | No | No |  0.500±0.000 |  0.000±0.000 |  0.306±0.001 |  0.851±0.007 |  0.821±0.007
No | Yes | No |  1.000±0.000 |  0.669±0.037 |  0.318±0.004 |  0.812±0.017 |  0.825±0.002
Yes | Yes | No |  1.000±0.000 |  0.720±0.002 |  0.283±0.008 |  0.802±0.019 |  0.838±0.003
Yes | No | Yes |  0.989±0.000 |  0.753±0.004 |  0.258±0.001 |  0.839±0.008 |  0.821±0.007

##  6 Conclusions

In this paper, we proposed and theoretically justify a general order-sensitive
message-passing scheme that can be applied to any chiral-sensitive graph
domain where chirality can be expressed by an order of the neighboring nodes.
We used the proposed framework to construct a novel ChiENN layer that enables
chirality awareness in any GNN model in the domain of molecular graphs, where
chirality plays an important role as it can strongly alter the biochemical
properties of molecules. Our experiments showed that the ChiENN layer allows
to outperform the current state-of-the-art methods in chiral-sensitive
molecular property prediction tasks.

## Acknowledgements

The research of J. Tabor was supported by the Foundation for Polish Science
co-financed by the European Union under the European Regional Development Fund
in the POIR.04.04.00-00-14DE/18-00 project carried out within the Team-Net
program. The research of P. Gaiński and M. Śmieja was supported by the
National Science Centre (Poland), grant no. 2022/45/B/ST6/01117. For the
purpose of Open Access, the author has applied a CC-BY public copyright
license to any Author Accepted Manuscript (AAM) version arising from this
submission.

## Ethical Statement

As we consider our work to be fundamental research, there are no direct
ethical risks or societal consequences; these have to be analyzed per concrete
application.

## References

  * [1] Adams, K., Pattanaik, L., Coley, C.W.: Learning 3D representations of molecular chirality with invariance to bond rotations. In: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net (2022)
  * [2] Brown, N., Fiscato, M., Segler, M.H.S., Vaucher, A.C.: Guacamol: Benchmarking models for de novo molecular design. J. Chem. Inf. Model. 59(3), 1096–1108 (2019)
  * [3] Chen, S., Jung, Y.: Deep retrosynthetic reaction prediction using local reactivity and global attention. JACS Au 1(10), 1612–1620 (2021)
  * [4] Choukroun, Y., Wolf, L.: Geometric transformer for end-to-end molecule properties prediction. arXiv preprint arXiv:2110.13721 (2021)
  * [5] Coors, B., Condurache, A.P., Geiger, A.: Spherenet: Learning spherical representations for detection and classification in omnidirectional images. In: Proceedings of the European conference on computer vision (ECCV). pp. 518–533 (2018)
  * [6] Ganea, O., Pattanaik, L., Coley, C., Barzilay, R., Jensen, K., Green, W., Jaakkola, T.: Geomol: Torsional geometric generation of molecular 3d conformer ensembles. Advances in Neural Information Processing Systems 34, 13757–13769 (2021)
  * [7] Gasteiger, J., Becker, F., Günnemann, S.: Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems 34, 6790–6802 (2021)
  * [8] Gasteiger, J., Groß, J., Günnemann, S.: Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123 (2020)
  * [9] Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E.: Neural message passing for quantum chemistry. In: International conference on machine learning. pp. 1263–1272. PMLR (2017)
  * [10] Gómez-Hortigüela, L., Bernardo-Maestro, B.: Chiral organic structure-directing agents. Insights into the chemistry of organic structure-directing agents in the synthesis of zeolitic materials pp. 201–244 (2018)
  * [11] Hawkins, P.C., Skillman, A.G., Warren, G.L., Ellingson, B.A., Stahl, M.T.: Conformer generation with omega: algorithm and validation using high quality structures from the protein databank and cambridge structural database. Journal of chemical information and modeling 50(4), 572–584 (2010)
  * [12] Jamali, F., Mehvar, R., Pasutto, F.: Enantioselective aspects of drug action and disposition: Therapeutic pitfalls. Journal of pharmaceutical sciences 78, 695–715 (09 1989)
  * [13] Kovatcheva, A., Golbraikh, A., Oloff, S., Feng, J., Zheng, W., Tropsha, A.: Qsar modeling of datasets with enantioselective compounds using chirality sensitive molecular descriptors. SAR and QSAR in Environmental Research 16(1-2), 93–102 (2005)
  * [14] Kreuzer, D., Beaini, D., Hamilton, W., Létourneau, V., Tossou, P.: Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems 34, 21618–21629 (2021)
  * [15] Krstulović, A.M.: Chiral Separations by HPLC. Chichester: Ellis Horwood, Chichester (1989)
  * [16] Liao, K., Yang, Y.F., Li, Y., Sanders, J., Houk, K., Musaev, D., Davies, H.: Design of catalysts for site-selective and enantioselective functionalization of non-activated primary c–h bonds. Nature Chemistry 10 (10 2018)
  * [17] Liu, C., Korablyov, M., Jastrzebski, S., Wlodarczyk-Pruszynski, P., Bengio, Y., Segler, M.H.S.: Retrognn: Fast estimation of synthesizability for virtual screening and de novo design by learning from slow retrosynthesis software. J. Chem. Inf. Model. 62(10), 2293–2300 (2022)
  * [18] Mamede, R., de Almeida, B.S., Chen, M., Zhang, Q., Aires-de Sousa, J.: Machine learning classification of one-chiral-center organic molecules according to optical rotation. Journal of Chemical Information and Modeling 61(1), 67–75 (2020)
  * [19] Mansimov, E., Mahmood, O., Kang, S., Cho, K.: Molecular geometry prediction using a deep generative graph neural network. Scientific Reports 9(1), 20381 (Dec 2019)
  * [20] Maziarka, L., Danel, T., Mucha, S., Rataj, K., Tabor, J., Jastrzebski, S.: Molecule attention transformer. CoRR abs/2002.08264 (2020)
  * [21] Maziarka, Ł., Pocha, A., Kaczmarczyk, J., Rataj, K., Danel, T., Warchoł, M.: Mol-cyclegan: a generative model for molecular optimization. Journal of Cheminformatics 12(1), 2 (Jan 2020)
  * [22] Maziarz, K., Jackson-Flux, H.R., Cameron, P., Sirockin, F., Schneider, N., Stiefl, N., Segler, M.H.S., Brockschmidt, M.: Learning to extend molecular scaffolds with structural motifs. In: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net (2022), https://openreview.net/forum?id=ZTsoE8G3GG
  * [23] Nguyen, L., He, H., Pham-Huy, C.: Chiral drugs: An overview. International journal of biomedical science : IJBS 2, 85–100 (06 2006)
  * [24] Pattanaik, L., Ganea, O.E., Coley, I., Jensen, K.F., Green, W.H., Coley, C.W.: Message passing networks for molecules with tetrahedral chirality. arXiv preprint arXiv:2012.00094 (2020)
  * [25] Pfaltz, A., Drury, W.: Design of chiral ligands for asymmetric catalysis: From c2-symmetric p,p- and n,n-ligands to sterically and electronically nonsymmetrical p,n-ligands. Proceedings of the National Academy of Sciences of the United States of America 101, 5723–6 (05 2004)
  * [26] Rampášek, L., Galkin, M., Dwivedi, V.P., Luu, A.T., Wolf, G., Beaini, D.: Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems 35, 14501–14515 (2022)
  * [27] Schneider, N., Lewis, R.A., Fechner, N., Ertl, P.: Chiral cliffs: investigating the influence of chirality on binding affinity. ChemMedChem 13(13), 1315–1324 (2018)
  * [28] Schütt, K.T., Sauceda, H.E., Kindermans, P.J., Tkatchenko, A., Müller, K.R.: Schnet–a deep learning architecture for molecules and materials. The Journal of Chemical Physics 148(24), 241722 (2018)
  * [29] Subramanian, G., Ramsundar, B., Pande, V., Denny, R.A.: Computational modeling of $\beta$-secretase 1 (BACE-1) inhibitors using ligand based approaches. Journal of chemical information and modeling 56(10), 1936–1949 (2016)
  * [30] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)
  * [31] Wang, Y., Wang, J., Cao, Z., Barati Farimani, A.: Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence 4(3), 279–287 (Mar 2022)
  * [32] Wu, Z., Ramsundar, B., Feinberg, E.N., Gomes, J., Geniesse, C., Pappu, A.S., Leswing, K., Pande, V.: MoleculeNet: a benchmark for molecular machine learning. Chemical science 9(2), 513–530 (2018)
  * [33] Xu, M., Luo, S., Bengio, Y., Peng, J., Tang, J.: Learning neural generative dynamics for molecular conformation generation. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net (2021)
  * [34] Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., et al.: Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling 59(8), 3370–3388 (2019)
  * [35] Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. Advances in neural information processing systems 30 (2017)
  * [36] Zhu, J., Xia, Y., Wu, L., Xie, S., Qin, T., Zhou, W., Li, H., Liu, T.: Unified 2d and 3d pre-training of molecular representations. In: Zhang, A., Rangwala, H. (eds.) KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022. pp. 2626–2636. ACM (2022)

##  Appendix 0.A Neighbor Order Construction

In this section, we describe Transformation and Sorting steps of order
construction from 4.2.

###  0.A.1 Transformation

We perform a sequence of 3D transformations on $c_{jk}$ and
$c_{i_{1}j},...,c_{i_{d}j}\in\mathds{R}^{3}\times\mathds{R}^{3}$ to make
$c_{jk}$ anchored to coordinate origin, perpendicular to $yz$ plane and
pointed away from the observer. To simplify the notation, we will perform
transformation on coordinates $c_{i}\in\mathds{R}^{3}$ that were used in the
definition of the 6D coordinates $c_{ij}=c_{i}|c_{j}$ in 4.1. We define
function to calculate angle between 2D vectors as
$angle(a,b)=arccos\left(\frac{ab^{T}}{|a||b|}\right)$.

  1. 1.

We first transpose every point with $t(c)=c-c_{j}$, so that $c_{jk}$ is
anchored to the coordinate origin, so $c_{i}=t(c_{i})$,

  2. 2.

We calculate $\alpha_{x}=angle([(c_{j})_{y},(c_{j})_{z}],[0,1])$ and matrix
$W_{x}$ representing the rotation along $x$ axis by $\alpha_{x}$ angle. We
rotate the points, so $c_{i}=W_{x}c_{i}$,

  3. 3.

We calculate $\alpha_{y}=angle([(c_{j})_{x},(c_{j})_{z}],[1,0])$ and matrix
$W_{y}$ representing the rotation along $y$ axis by $\alpha_{x}$ angle. We
rotate the points, so $c_{i}=W_{y}c_{i}$.

###  0.A.2 Sorting

After applying the transformation described in the previous versions, we can
sort the 3D coordinates $c_{i_{1}},...,c_{i_{d}}$ by the angle
$\alpha_{i}=angle([1,0],[(c_{i})_{y},(c_{i})_{z}])$ between the $y$-axis and
their projections on $yz$ axis. Therefore, we obtain an order
$(x_{i_{\pi(1)}k},...,x_{i_{\pi(d)}k})$ such that
$\alpha_{i_{\pi(l)}}\leq\alpha_{i_{\pi(l+1)}}$.

##  Appendix 0.B Experimental Details

###  0.B.1 Hyperparameter grids

We performed a grid search of parameters with identical budget: for all
models, with learning rate
$\in\\{1\mathrm{e}{-3},1\mathrm{e}{-4},1\mathrm{e}{-5}\\}$ and dropout
$\in\\{0,0.2,0.5\\}$, and with model-dependent number of layers and layer
dimensionality, chosen based on the parameters from corresponding papers: for
DMPNN and Tetra-DMPNN, with layers $\in\\{2,4,6\\}$ and dimensionality
$\in\\{300,600,900\\}$; for ChIRo, with layers $\in\\{2,3,4\\}$ and
dimensionality $\in\\{64,128,256\\}$; and for GPS, SAN and ChiENN, with layers
$\in\\{3,6,10\\}$ and dimensionality $\in\\{64,128,256\\}$. We restricted the
grid search to a subset of a dataset of size at most 10000 molecules. As the
computational costs of Tetra-DMPNN are high, for binding affinity, we took the
optimal hyperparameters for DMPNN+tags as parameters for Tetra-DMPNN. We did
similarly for SAN+ChiENN for R/S and binding rank and took the corresponding
optimal hyperparameters from SAN.