# Accelerating Laboratory Automation Through Robot Skill Learning For Sample
Scraping*

Gabriella Pizzuto1, Hetong Wang1, Hatem Fakhruldeen1, Bei Peng2, Kevin S.
Luck3 and Andrew I. Cooper1 *This work was supported by the Leverhulme Trust
through the Leverhulme Research Centre for Functional Materials Design and the
H2020 ERC Synergy Grant Autonomous Discovery of Advanced Materials under grant
agreement no. 856405.1Gabriella Pizzuto, Hetong Wang, Hatem Fakhruldeen and
Andrew I. Cooper are with the Department of Chemistry, University of
Liverpool, United Kingdom.
Corresponding author<EMAIL_ADDRESS>Peng is with the
Department of Computer Science, University of Liverpool, United Kingdom.3Kevin
S. Luck is the Finnish Center for Artificial Intelligence (FCAI), Espoo,
Finland.

###### Abstract

The potential use of robotics for laboratory experiments offers an attractive
route to alleviate scientists from tedious tasks while accelerating the
process of obtaining new materials, where topical issues such as climate
change and disease risks worldwide would greatly benefit. While some
experimental workflows can already benefit from automation, it is common that
sample preparation is still carried out manually due to the high level of
motor function required when dealing with heterogeneous systems, e.g.,
different tools, chemicals, and glassware. A fundamental workflow in chemical
fields is crystallisation, where one application is polymorph screening, i.e.,
obtaining a three dimensional molecular structure from a crystal. For this
process, it is of utmost importance to recover as much of the sample as
possible since synthesising molecules is both costly in time and money. To
this aim, chemists have to scrape vials to retrieve sample contents prior to
imaging plate transfer. Automating this process is challenging as it goes
beyond robotic insertion tasks due to a fundamental requirement of having to
execute fine-granular movements within a constrained environment that is the
sample vial. Motivated by how human chemists carry out this process of
scraping powder from vials, our work proposes a model-free reinforcement
learning method for learning a scraping policy, leading to a fully autonomous
sample scraping procedure. To realise that, we first create a simulation
environment with a Panda Franka Emika robot using a laboratory scraper which
is inserted into a simulated vial, to demonstrate how a scraping policy can be
learned successfully. We then evaluate our method on a real robotic
manipulator in laboratory settings, and show that our method can autonomously
scrape powder across various setups. This experiment is a first step towards
the automation of crystallisation workflows in chemistry labs.

##  I Introduction

Transforming materials discovery plays a pivotal role in addressing global
challenges for example renewable and clean energy storage, sustainable
polymers and packaging for consumer products towards a more circular economy,
and drugs and therapeutics. To achieve fundamental breakthroughs in this
field, human scientists need to work closely with ‘robot scientists’ to carry
out long-term experiments that are fundamental to obtaining new materials. The
goal here is that robots will carry out tedious cognitive and manual tasks,
whereas humans conduct experiments that are safe and mentally stimulating.
While there have been remarkable endeavours to introduce robotics in the field
of laboratory automation [1], [2], [3], there remains an open gap in having
robots carry out tasks beyond pre-programmed manipulation. This hinders
progress in developing autonomous robots that are capable of operating end-to-
end in human labs, or at the very least continue scientists’ experiments after
hours.

Figure 1: Overview of laboratory sample scraping, where (left) a human
scientist scrapes crystals that generally form on vial walls by transcending
along the walls vertically and (right) our robot learning to scrape via RL.

To date, laboratory robotics rely on pre-defined point-to-point motions
without any sensory feedback or learning-based methods that would increase
their adaptability to differing environmental conditions. Learning would help
to tackle the problem of environmental variation - learning algorithms are
capable of adapting to variations in lighting conditions, displacements of
material, etc. Conversely, humans are able to use different modalities and
carry out skills that gradually get better with experience. This is not, and
will never be, attainable with traditional laboratory automation due to the
required dexterity and coordination between visual inputs and motor response
that is trivial in humans [4]. Data-driven learning methods have shown
remarkable progress in robotics in recent years. In turn, robots were able to
acquire a variety of practical skills that previously were not possible [5].
Solving dexterous manipulation in laboratory tasks is an important problem as
it enables a wide range of useful robotic workflows that otherwise are not
possible. Consider the sample scraping problem as illustrated in Fig. 1. A
human chemist would scrape the vial walls by translating vertically in a
circular fashion. Solving such tasks with traditional laboratory automation
method is not possible due to the difficulty of requiring force feedback as
the walls are scraped and different crystal morphology would need an adaptive
method of scraping.

In this work, we introduce model-free reinforcement learning (RL) to
laboratory automation to overcome previous challenges in skill acquisition for
material discovery experimental workflows. We learn policies for the full
laboratory task of scraping, which map proprioceptive and force observations
to end effector displacement actions of a robotic arm. Moreover, we explore
curriculum learning [6], [7], where we progressively increase the difficulty
of the task through extending the size of the workspace as the training of our
agent improves, to reduce training costs while increasing the overall task
success. By doing so, we could successfully learn the more challenging
insertion and scraping task, while the RL method without a curriculum fails to
learn. Finally, while training and evaluation of the scraping policy is
performed in simulation, we report the required adjustments and findings from
migrating these policies to a real robotic platform.

In summary, the contributions of this work are:

  * •

A closed-loop formulation of an in-tube scraping task, a sub-task of a
crystallisation workflow.

  * •

A new benchmark task for robot learning in chemistry labs111A link to the
benchmark will be added here upon publication..

  * •

A case study of applying deep reinforcement learning to a challenging motor
skill (in-tube scraping) that is not possible with traditional laboratory
automation.

  * •

Exploiting curriculum learning to improve the learning performance of the RL
agent in a more challenging insertion and scraping task.

  * •

A demonstration of the learned robot behaviour both in simulation and in a
real-world chemistry lab.

To the best of our knowledge, this is the first research that uses learning-
based methods for acquiring human-inspired motor skills for a crystallisation
workflow in a real-world chemistry laboratory. Evaluations of the method with
an experimental setup using tools and glassware from a real-world material
discovery lab are aimed at demonstrating its applicability to real laboratory
environments.

##  II Related Work

###  II-A Laboratory Automation for Manipulation Skills

Rapid progress in robotics, automation, and algorithmic efficiency has made it
possible for autonomous robotic platforms to be used in laboratory
environments. For example, robotic manipulators have been used to measure
solubility in multiple solvents [8], [9], and for carrying out a Michael
reaction [3], while mobile manipulators have exceeded human-level ability to
search for improved photocatalysts for hydrogen production from water [10].
However, whilst these are excellent endeavours in removing these tedious and
repetitive tasks from the scientists’ daily routines, the robotic platforms do
not exhibit the same skill as a human chemist would. In fact, the robotic
skills in these experiments were pre-programmed and carried out without
sensory feedback. These were also limited to pick-and-place of vials and
racks, rather than irregular objects such as laboratory tools. In practice,
usage of robotic platforms as such would limit deployment of autonomous robots
for a vast array of experiments as they would not be capable of generalising
to new reactions, or would necessitate oversimplification of workflows leading
to poorer results.

The automation of laboratories for life sciences has stricter protocols when
compared to other fields, and as a result several works have demonstrated the
usage of dual-arm robots to mimic human way of carrying out laboratory
procedures. It was demonstrated how a dual-arm robotic manipulator can be used
for sample preparation and measurement in [11], [12] and [13]. Here, the
authors focused on automating the instrument to facilitate the task of using
laboratory instruments such as pipettes. Conversely, we concentrate our
research endeavours on having the robotic manipulator learn the task through
goal-conditioned reinforcement learning which allows us to use laboratory
tools that do not have to be automated. This approach also enables us to
address laboratory tasks that require human adaptability when dealing with
different glassware and molecules that have distinct chemical properties such
as hygroscopy, cohesiveness, hardness, amongst other factor.

###  II-B Reinforcement Learning for Contact-rich Manipulation

There have been several methods proposed to accelerate automation of contact-
rich robotic skill learning, primarily for insertion tasks such as peg-in-hole
assembly. For example, deep reinforcement learning-based assembly search
strategies to align the peg with the hole [14], [15], industrial insertion
tasks with tighter clearance [16] or using meta-reinforcement learning for
generalising across a variety of different insertion tasks [5]. Our task in
this paper is similar to robotic insertion; however, it goes beyond any
insertion task as the tool needs to maintain contact with the walls of the
cylindrical vessel, which is also reflected through the reward function,
similar to the constraint within the learning objective in [17].

While DRL-based policies are increasingly being used and shown to have the
potential to solve assembly skills in real-world applications, there still
exists an open gap in having such methods for real laboratory tasks. One
potential reason for this could be that laboratory automation has
traditionally been done in open-loop, which hinders the interaction of the
agent obtaining feedback from its environment. We aim to contribute to this
area by proposing a novel way of laboratory skill learning using goal-
conditioned reinforcement learning [18] for a task that has not been addressed
before, while demonstrating the important role that DRL solution can have when
adopted in real-world laboratories to solve tasks that are unattainable with
conventional methods.

##  III Methodology

The goal of this work is to introduce and evaluate the performance of
learning-based autonomous robotic laboratory skill acquisition, particularly
for the task of scraping a vial sample. This task is commonly performed in
laboratory environments and is fundamental for successful powder diffraction
since it maximises the transfer of powder. This is particularly crucial as
synthesising molecules is costly [19]. As illustrated in Fig. 2, a real or
simulated robotic manipulator uses a scraper that is attached to the robot’s
end effector and a vial is placed within its workspace. The robot has to learn
to insert the tool while scraping the vial walls. Our proposed method relies
on force/torque information at the robot’s end-effector which is essential to
realise contact-rich manipulation tasks; scraping glassware requires stringent
force control which can be captured through sensory feedback.

Figure 2: The overall block diagram of autonomous robotic scraping. Our method
consists of an RL policy and a robotic controller. We map the action
(cartesian pose) to joint positions using inverse kinematics to achieve the
target goal. The controller uses force/torque feedback at the end effector and
a joint space controller for motion planning.

###  III-A Problem Formulation

The task _scrape_ is defined as the following: the goal is to reach the target
position at the bottom of the vial, while maintaining contact with the vial
wall for material removal. The end-effector commands and state readings are
computed with respect to the tool central point (TCP) at the tip of the
scraper. The task is successful if the distance between the target position
and the end-effector is smaller within the goal tolerance, and contact between
the scraper and the vial is maintained when the scraper has been inserted
inside. The overall problem is illustrated in Fig. 3.

Figure 3: An illustration of the robotic scraping task. The robot inserts the
scraper inside the vial. The robot’s goal is to translate along the z-axis
while maintaining contact with the vial wall ($F_{y}\geq C_{threshold}$).

For modeling the underlying decision-making problem of an agent scraping
powder from the vial walls, we adopt the finite-horizon discounted Markov
decision process (MDP) formalism [20], [21]. It can be defined as a tuple
$M=(S,A,P,r,\gamma,T)$, consisting of a continuous state space $S$, a
continuous action space $A$, a transition function $P(s_{t+1}|s_{t},a_{t})$
defining the probability of the environment transitioning to state $s_{t+1}$
after the agent takes action $a_{t}$ in state $s_{t}$, a reward function
$r(s_{t},a_{t})$ specifying the scalar reward the agent receives after taking
action $a_{t}$ in state $s_{t}$, a discount factor $\gamma\in[0,1)$, and a
horizon $H$ specifying the number of steps per episode. The actions are drawn
from a stochastic policy which is a state-conditioned probability distribution
over actions $\pi(a_{t}|s_{t})$. The goal of the agent is to find the optimal
policy $\pi^{*}$, which maximises the expected sum of discounted future
rewards $\mathbb{E}[\sum_{i=0}^{T}\gamma^{i}r_{t+i}]$ when the agent selects
actions in each state according to $\pi^{*}$.

The details about the agent’s actions and state space (observations), as well
as the formulation of our reward function for autonomous robotic sample
scraping are outlined next.

###  III-B Agent Description

As robotic scraping is a continuous control task, where sample efficiency is
desired and we are dealing with both continuous actions and state spaces, we
opted for an off-policy actor-critic algorithm. Precisely, our method uses a
state-of-the-art method for continuous control: the Truncated Quantile Critic
(TQC) [22] which builds on Soft-Actor Critic (SAC) [23], Twin Delayed Deep
Deterministic Policy Gradient (TD3) [24] and Quantile Regression Deep
Q-network (QR-DQN) [25]. TQC takes advantage of quantile regression to predict
a distribution for the value function. Similar to TD3, it truncates the
quantiles predicted by different networks. We also compare this method in the
simulated environment to vanilla SAC. The agent also made use of hindsight
experience replay (HER) [26], a technique that increases sample efficiency by
replacing goals are by the achieved goals and in turn updating the policy with
previous states, actions, and rewards stored in a buffer.

###  III-C Model Architecture

The model architecture was implemented using the stable baselines software
package [27] and the hyperparameters chosen are based on the optimised ones in
the RL baselines zoo [28] for goal-conditioned DRL. Given that the observation
space contains the position of the end-effector, we chose a multiple input
policy network which concatenates all states into one vector that is passed to
the policy. The model hyperparameters are given in TABLE I.

TABLE I: Hyperparameters used for the DRL algorithm Hyperparameter | Selected value
---|---
Network type | Multiple Input Policy
Hidden layers | 3
Hidden layer neurons | 512
Activation hidden layers | ReLU
Learning rate | $1e^{-3}$
Replay buffer size | 10k (500 real robot)
Batch size | 2048 (64 real robot)
$\Delta$ | 0.05cm (real robot only)

###  III-D Observations

The observations used for the scraping task are proprioceptive and force
feedback. The proprioceptive observation gives the cartesian position of the
end-effector ($x$, $y$, $z$). The force feedback describes the force/torque
feedback at the end-effector ($F_{X}$, $F_{Y}$, $F_{Z}$).

###  III-E Actions

The actions used in our experiments comprise the cartesian pose where
$a=(\Delta x,\Delta y,\Delta z)$, which is the change in end-effector pose
needed to arrive at the desired goal. The $\Delta$ was limited such that the
robot does not experience sharp changes and abrupt movements. In practice, we
converted the cartesian pose to joint angles of the seven DoF robotic
manipulator via inverse kinematics to carry out the motion planning in joint
space. We also limit these values (by scaling the output of the agent) to keep
the angles within the joint limits and hence within a feasible range for the
real robot and also to assure a more stable training process.

###  III-F Reward

Our reward function for the simulated environment consists of the following
components:

$R_{simulated\\_task}=R_{goal}+w_{0}R_{contact}+w_{1}R_{vial}$ (1)

At each time step, we calculate the shaping reward $R_{goal}$ as the L2 norm
(Euclidean distance) between the target goal and current observation, defined
as:

$R_{goal}=||goal_{desired}-goal_{current}||_{2}$ (2)

$R_{goal}$ is motivated by observations of how chemists scrape samples as the
manoeuvre is more often done vertically rather than horizontally. From a
technical viewpoint, this also makes it more feasible as a robotic task to
maintain contact with the vial wall as the translation is more similar to a
plane rather than a curved surface.

The simulated agent’s actions are constrained by two additional reward
components that punish undesired behaviour: $R_{contact}$ and $R_{vial}$.
$R_{contact}$ is set to -1 if the horizontal part of the scraper does not
touch the simulated vial represented. In simulation, collision detection is
used to verify where these two bodies are in contact, whereas for the real
robot force/torque feedback at the end effector was measured directly.
$R_{vial}$ is set to -0.1 if the centre of the simulated vial is moved by 2cm.
We explored different weightings of both rewards ($w_{0}$, $w_{1}$). The
reward function for the real robotic scraping is similar to Equation 1 but
without the penalty for moving the vial:

$R_{real\\_task}=R_{goal}+R_{contact}$ (3)

In practice, vials are placed in racks; hence, the motion of the vial is
already constrained, and we did not observe large displacements.

##  IV Experimental Evaluation

We evaluated our autonomous robotic scraper in simulation and on a real
robotic platform. Our experiments aim at answering the following research
questions. First, can we learn a policy that enables a robotic manipulator to
scrape the contents of a vial? This first experiment aims to compare the
performance of different off-policy model-free RL algorithms to learn a
scraping policy for our task. The criteria for this comparison is the average
model success rate and the mean return attained by different methods. These
are both significant for the selection of the specific model that will be
deployed, as we aim at minimising real-world training and inference times and
the risk of failure. We trained directly in the real world as contact-rich
tasks are hard to model in simulated environments, in addition to the non-
trivial challenge of modelling powder and crystals in a simulated vial.
Second, given that the goal of scraping is to maximise the removal of powder
from the vial walls, how does the policy deployed on the real robot perform?
This experiment aims to illustrate that learning a good policy is not
sufficient, but it is also important to visually demonstrate the removal of
powder from the vial walls for task success. Our demonstration is vital to
understand failure cases that might hinder task completion, while giving us a
better understanding of current shortcomings of the learning algorithms and
the scraper (tool) so that we can optimise it for both our robotic manipulator
and human scientists.

###  IV-A Simulated Experiment

####  IV-A1 Experimental Setup

Our simulated environment, illustrated in Fig. 2, builds on panda-gym [29],
which is a set of DRL environments for the Franka Emika Panda robot integrated
with OpenAI gym. The panda-gym benchmark does not yet have insertion tasks and
we believe that our task provides an interesting novel environment here,
especially for in-tube manipulation. Moreover, the task of scraping is even
more challenging than insertion as contact has to be maintained; hence our
environment is new as it introduces a constrained wiping task which is
compelling to the robotics and machine learning communities alike. In our
task, the robot has 7 degrees of freedom (DoFs) and it is equipped with a
scraper attached to joint 7. This was done to leverage quick experiments while
focusing on the difficulty of contact-rich insertion and scraping rather than
grasping. The scraper consists of a white cylinder and a black box and was
modelled based on a scraping tool that is commonly found in laboratory
environments (Fig. 6). In simulation, the vial used in a laboratory setting
was modelled as a hollow tube. All experiments were run using a machine
equipped with an AMD Ryzen Threadripper 3970X 32 Core CPU and a single NVIDIA
GeForce RTX 3090 Graphical Processing Unit (GPU).

####  IV-A2 Experiment I - Learning a scraping policy

In this experiment, the scraping task needs to be learned by only relying on
the proprioceptive feedback and the contact force readings at the end-
effector, which in simulation the collision detection between two bodies is
used to monitor contact. The robot needs to learn a policy where the goal is
to minimise the distance between the start and the goal pose, while
maintaining contact with the inside of the vial wall. During training, a
target position is randomly generated within a pre-defined region on the
bottom of the vial. The start pose is fixed and is set at the top of the vial.
The length of a trajectory is 50 timesteps, where at the end of each
trajectory, the environment is reset and a new goal is randomly generated. All
these trajectories are stored in a common replay buffer. The results are the
success rate evaluated every 1000 test episodes over the course of learning.
We explore three baselines of the results for the off-policy RL algorithms
(TQC, SAC) used with HER. We report the evaluation performance of the best
performing hyper-parameters for all algorithms (based on training
performance). We run all experiments with 5 seeds and report both mean and
standard error (denoted by the shaded area on the plot) in Fig. 4.

Figure 4: Evaluation results of learning a scraping policy in simulation
across 5 random seeds.

While we explored the possibility of the robot scraping the full circular
vial, we decided that this is not the best solution as human chemists scrape
vertically and with the other arm they rotate the vial. Furthermore, this
process proved to be more robust for this task when deployed on the robot

####  IV-A3 Experiment II - Adding a curriculum

This experiment focused on increasing the challenge of the scraping task by
changing the starting pose to outside the vial, such that the task now covers
both insertion and scraping. The rest of the experiment is set up similar to
Experiment I. In this experiment we explore curriculum learning, where the key
idea is to learn first on simple examples before moving to more difficult
problems. In order to learn the target task, the agent first trains on a
curriculum that consists of a sequence of intermediate tasks with increasing
difficulty. Our curriculum consists of only a single intermediate task, where
the starting state is closer to the goal (at the top of the vial wall). This
approach makes use of transfer learning methods to effectively extract and
pass on reusable knowledge acquired in one task to the next. In our
experiment, we achieve this by transferring the policies learned. The results
are the success rate evaluated every 1000 test episodes over the course of
learning. We report the evaluation performance of the best performing hyper-
parameters for TQC (based on training performance). We run all experiments
with 5 seeds and report both mean and standard error (denoted by the shaded
area on the plot) in Fig. 5. The results obtained demonstrate that using a
curriculum with an intermediate task is necessary in order to achieve high
success rates and generalise across different, more challenging start poses.

(a)

(b)

Figure 5: Evaluation results of learning a scraping policy with a curriculum
in simulation across 5 random seeds.

###  IV-B Real Robot Experiment: An Autonomous Sample Scraping Case Study

Our developed method was evaluated in the context of an autonomous laboratory
workflow to demonstrate its potential to the challenging scraping problem.
This aim was to autonomously scrape powder off the walls of a given sample
vial in the real world.

####  IV-B1 Experimental Setup

The real setup comprised the Franka Emika Panda robot, equipped with a Robotiq
parallel gripper and force torque sensor. We trained our policy on hardware
using a plastic vial, and then tested the model on a glass vial. This was done
to minimise potential glass breakage during training exploration. Glass is
used in laboratories as it has a high chemical compatibility with organic
solvents and hence using plastic vials would limit the chemistry experiments.
Our experimental setup is shown in Fig. 6. For the real-world experiments, we
used the TQC model.

####  IV-B2 Case Study - Scraping powder

The overall scraping task is demonstrated in the video provided in the
supplementary material. To monitor the progress of the scraping task a number
of parameters were defined. We used food grade material (flour) since
experiments were carried out on an open bench. The vial was divided into
$N_{reg}$ regions, since our learnt policy was constrained to an area of the
vial equivalent to $\pi/12$ radians to avoid reaching the joint limit on the
end effector.

To complete the task, the robot takes the following steps. First, it picks up
the scraping tool and goes to the scraping start pose. The robot then follows
the scraping policy, where it picks the best sequence of actions to achieve
the target goal. The robot would repeat this step for a pre-defined duration
that was optimised to maximise scraping within the region in $N_{reg}$; in our
experiment, this was chosen to be 5 times. This is followed by the robot
putting the scraper in its holder and rotating the vial by $\pi/12$ radians,
in preparation for scraping the next region in $N_{reg}$. Following this, the
robot grasps the tool again. Once the robot has carried this out across all
regions ($N_{reg}=24$ if each region is $\pi/12$), the task is considered to
be complete. This task was carried out successfully with two different sized
vials and two scrapers of different length, as illustrated in Fig. 6.

(a)

(b)

Figure 6: An overview of (a) the robotic scraping setup (a complete video is
provided as supplementary material) and (b) scrapers and vials used.

In Fig. 7, we demonstrate the results obtained from the autonomous robotic
scraping. Fig. 7a demonstrates the vial at the start of the experiment with
all of the powder still fixed inside the vial, whereas Fig. 7c captures the
vial contents after robot has scraped the inside of the vial.

(a)

(b)

(c)

Figure 7: Qualitative results of the vials obtained from the case study, where
(a) the initial vial condition, (b) midway through (powder partially removed)
and (c) the powder scraped off.

###  IV-C Failure Cases

During the experimental trials, we analysed in-depth the failure cases. An
early failure case was when the actual robot pushed the inside of the vial
that was outside of a rack and it toppled over. In this instance, it was
impossible for it to know that it had failed the task. Hence, we made the
decision of scraping vials when they are inside a rack, which is also a more
realistic setting in a lab as it is rarely the case that vials are left
outside of a rack on a bench. We also introduced a threshold if the force
$F_{z}$ is exceeded (20Nm) the robot should return to its start position, to
minimise the chance of the robot pushing vials over and breakages. We only set
a threshold to $F_{z}$ as we did not notice the robot applying large forces
for $F_{x}$ and $F_{y}$, particularly since the translation was predominantly
in the z-direction. Another similar example was when the scraper gets stuck in
the vial and the robot could not generate enough force to reset and pull the
scraper out. In this case, the human needs to reset the robot manually. This
was the only non-recoverable failure case which required a human supervisor to
reset the task.

###  IV-D On Scraper Tool Choice

The scraper used here is a tool that is commonly found in laboratories.
However, given that a robotic manipulator does not have the same dexterity as
a human arm, the tool made the task more challenging due to the small surface
contact that can be achieved. We also limited the change in motion during the
scraping task to minimise high impact that could increase the risk of glass
breakage. As a result, this made the process slower but safer. In the future,
we will explore other tools such as brushes or even scrapers where the tool
end is similar to a 2D cone to increase task efficiency.

##  V Conclusion

In this work, we demonstrated that model-free reinforcement learning is a
viable approach for learning laboratory skills such as autonomous robotic
sample scraping. To learn the policy, we first created a simulated environment
based on our real ‘robotic scientist’ with a scraper and vial. This novel task
challenges previous insertion benchmark environments as it requires a
constrained powder wiping task during the insertion. In the simulated
environment we showed how distinct policies were learned for several off-
policy RL approaches and that using curriculum learning is beneficial to
generalise across more challenging configurations. We then demonstrated that a
data-driven model can be learned and deployed on a real robotic manipulator to
successfully scrape the inside of a sample vial. Our empirical results
demonstrate that our proposed method enables the robot to complete the task
across different vial sizes and tools. Additionally, the results show the use
of DRL enables us to complete tasks that have not been possible in chemistry
workflows. There are several future directions for extending and improving
this approach. On our immediate future plan, we want to explore how our method
performs with different scrapers and how we can combine the visual input
within the current reward. Moreover, we want to look into bi-manual
manipulation to speed up the scraping task where an arm is used to rotate the
vial while the other one scrapes.

## References

  * [1] K. Thurow and S. Junginger, Devices and Systems for Laboratory Automation.  John Wiley and Sons, 2022.
  * [2] H. Fakhruldeen, G. Pizzuto, J. Glawucki, and A. I. Cooper, “Archemist: Autonomous robotic chemistry system architecture,” IEEE International Conference on Robotics and Automation, 2022.
  * [3] J. X.-Y. Lim, D. Leow, Q.-C. Pham, and C.-H. Tan, “Development of a robotic system for automatic organic chemistry synthesis,” IEEE Transactions on Automation Science and Engineering, vol. 18, no. 4, pp. 2185–2190, 2021.
  * [4] M. Seifrid, R. Pollice, A. Aguilar-Granda, Z. Morgan Chan, K. Hotta, C. T. Ser, J. Vestfrid, T. C. Wu, and A. Aspuru-Guzik, “Autonomous chemical experiments: Challenges and perspectives on establishing a self-driving lab,” Accounts of Chemical Research, vol. 55, no. 17, pp. 2454–2466, 2022\.  PMID: 35948428.
  * [5] T. Z. Zhao, J. Luo, O. Sushkov, R. Pevceviciute, N. Heess, J. Scholz, S. Schaal, and S. Levine, “Offline meta-reinforcement learning for industrial insertion,” in 2022 International Conference on Robotics and Automation (ICRA), pp. 6386–6393, 2022.
  * [6] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, (New York, NY, USA), p. 41–48, Association for Computing Machinery, 2009.
  * [7] S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone, “Curriculum learning for reinforcement learning domains: A framework and survey,” J. Mach. Learn. Res., vol. 21, pp. 181:1–181:50, 2020.
  * [8] P. Shiri, V. Lai, T. Zepel, D. Griffin, J. Reifman, S. Clark, S. Grunert, L. P. Yunker, S. Steiner, H. Situ, F. Yang, P. L. Prieto, and J. E. Hein, “Automated solubility screening platform using computer vision,” iScience, vol. 24, p. 102176, 2021.
  * [9] G. Pizzuto, J. de Berardinis, L. Longley, H. Fakhruldeen, and A. I. Cooper, “Solis: Autonomous solubility screening using deep neural networks,” IEEE International Joint Conference on Neural Networks, 2022.
  * [10] B. Burger, P. M. Maffettone, V. V. Gusev, C. M. Aitchison, Y. Bai, X. Wang, X. Li, B. M. Alston, B. Li, R. Clowes, N. Rankin, B. Harris, R. S. Sprick, and A. I. Cooper, “A mobile robotic chemist,” Nature, vol. 583, pp. 237–241, Jul 2020.
  * [11] H. Fleischer, L. A. Kroos, S. Joshi, T. Roddelkopf, R. Stoll, and K. Thurow, “Dual-arm robotic compound-oriented measurement system: Integration of a positive pressure solid phase extraction unit,” in 2021 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), pp. 1–6, 2021.
  * [12] H. Fleischer, R. R. Drews, J. Janson, B. R. C. Patlolla, X. Chu, M. Klos, and K. Thurow, “Application of a dual-arm robot in complex sample preparation and measurement processes,” Journal of Laboratory Automation, vol. 21, no. 5, pp. 671–681, 2016.  PMID: 27000132.
  * [13] S. Joshi, X. Chu, H. Fleischer, T. Roddelkopf, M. Klos, and K. Thurow, “Analysis of measurement process design for a dual-arm robot using graphical user interface,” in 2019 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), pp. 1–6, 2019.
  * [14] X. Zhang and P. Ding, “Hole-peg assembly strategy based on deep reinforcement learning,” in Cognitive Systems and Signal Processing (F. Sun, H. Liu, and B. Fang, eds.), (Singapore), pp. 15–32, Springer Singapore, 2021.
  * [15] A. Yuji Yasutomi, H. Mori, and T. Ogata, “A peg-in-hole task strategy for holes in concrete,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2205–2211, 2021.
  * [16] T. Inoue, G. De Magistris, A. Munawar, T. Yokoya, and R. Tachibana, “Deep reinforcement learning for high precision assembly tasks,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 819–825, 2017.
  * [17] G. Pizzuto and M. Mistry, “Physics-penalised regularisation for learning dynamics models with contact,” in Proceedings of the 3rd Conference on Learning for Dynamics and Control, vol. 144 of Proceedings of Machine Learning Research, pp. 611–622, PMLR, 07 – 08 June 2021.
  * [18] M. Liu, M. Zhu, and W. Zhang, “Goal-conditioned reinforcement learning: Problems and solutions,” CoRR, vol. abs/2201.08299, 2022.
  * [19] V. Pecharsky and P. Zavalij, In Fundamentals of Powder Diffraction and Structural Characterization of Materials.  01 2003.
  * [20] R. Bellman, Dynamic Programming.  Princeton, NJ, USA: Princeton University Press, 1 ed., 1957.
  * [21] M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming.  USA: John Wiley and Sons, Inc., 1st ed., 1994.
  * [22] A. Kuznetsov, P. Shvechikov, A. Grishin, and D. P. Vetrov, “Controlling overestimation bias with truncated mixture of continuous distributional quantile critics,” ArXiv, vol. abs/2005.04269, 2020.
  * [23] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in ICML, 2018.
  * [24] S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” ArXiv, vol. abs/1802.09477, 2018.
  * [25] W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos, “Distributional reinforcement learning with quantile regression,” in Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI Press, 2018.
  * [26] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, “Hindsight experience replay,” in Advances in Neural Information Processing Systems (I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), vol. 30, Curran Associates, Inc., 2017.
  * [27] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” Journal of Machine Learning Research, vol. 22, no. 268, pp. 1–8, 2021.
  * [28] A. Raffin, “Rl baselines3 zoo.” https://github.com/DLR-RM/rl-baselines3-zoo, 2020.
  * [29] Q. Gallouédec, N. Cazin, E. Dellandréa, and L. Chen, “Multi-goal reinforcement learning environments for simulated franka emika panda robot,” CoRR, vol. abs/2106.13687, 2021.