# Learning to Manipulate Object Collections
Using Grounded State Representations

Matthew Wilson
University of Utah
United States
<EMAIL_ADDRESS>&Tucker Hermans
University of Utah & NVIDIA
United States
<EMAIL_ADDRESS>
###### Abstract

We propose a method for sim-to-real robot learning which exploits simulator
state information in a way that scales to many objects. We first train a pair
of encoder networks to capture multi-object state information in a latent
space. One of these encoders is a CNN, which enables our system to operate on
RGB images in the real world; the other is a graph neural network (GNN) state
encoder, which directly consumes a set of raw object poses and enables more
accurate reward calculation and value estimation. Once trained, we use these
encoders in a reinforcement learning algorithm to train image-based policies
that can manipulate many objects. We evaluate our method on the task of
pushing a collection of objects to desired tabletop regions. Compared to
methods which rely only on images or use fixed-length state encodings, our
method achieves higher success rates, performs well in the real world without
fine tuning, and generalizes to different numbers and types of objects not
seen during training. Video results: bit.ly/2khSKUs.

> Keywords: sim-to-real, reinforcement learning, manipulation

##  1 Introduction

Humans regularly manipulate object groups as collections. When scooping up a
bunch of grapes or sweeping a pile of coins into their hands, humans can track
and manipulate the objects without needing to know, for example, how many
nickels or dimes are present. Current robot systems lack such capabilities and
most recent work has focused on picking or pushing objects one at a time [1,
2, 3, 4, 5, 6]. Learning to simultaneously manipulate many objects, on the
other hand, is a greater challenge, as it requires a robot to track and reason
about the many possible configurations and physical interactions between
objects. To overcome this challenge, we use raw object poses from a simulator
to learn a latent space that captures multi-object state information; we then
use this learned representation to train an image-based policy that reasons
about and manipulates variably-sized object collections as a whole.

Learning policies directly from RGB images stands as a popular approach for
solving manipulation tasks, but as an approach it comes with several
challenges. Namely, it is not obvious how to form task-relevant features or
generate meaningful reward functions directly from high-dimensional sensory
inputs. Since labeled data for manipulation requires significant time and
effort to collect, researchers often leverage indirect self-supervised signals
for training (e.g., [7, 8, 9]). Since such signals, by definition, do not
directly correspond to task-relevant state information (such as poses and
velocities of relevant objects), they can be prone to failure. For example, an
autoencoder loss [7, 8] will incentivize a network to waste latent space
capacity modeling more visually salient large objects and background in a
scene, sometimes ignoring more fine-grained information relevant to the task.

Figure 1: Cartoon diagram of our approach. We first independently train two
encoder networks, one convolutional neural network (CNN) and one graph neural
network (GNN) using a multi-object state and dynamics loss function. Then,
during our RL phase, we embed the observation:
$o\xrightarrow{\text{CNN}}\phi_{o}$, state:
$s\xrightarrow{\text{GNN}}\phi_{s}$, and goal:
$g\xrightarrow{\text{GNN}}\phi_{g}$, and we use the embeddings in an
asymmetric actor critic framework [10] to train a multi-object policy $\pi$.

Simulation-based training is a promising route to sidestep these difficulties.
Researchers have shown that by exploiting raw state information to calculate
rewards [10, 11, 12, 13, 14, 15, 16, 17, 18], construct grounded latent
representations [10, 11, 19], and improve value function estimation [10, 11],
they can greatly improve the stability and speed of policy training. However,
all existing work rely on _fixed-length_ vector inputs, limiting their
effective use to settings with a known number of individual objects. Ours is
the first work we are aware of in this space capable of handling a variable
number of multiple objects.

Our primary contribution is an approach for exploiting simulator state
information in a way that scales to multiple and variable object settings. In
our approach, we train a pair of encoder networks to capture the state and
dynamics information of a variable number of objects; we then use them to aid
in learning multi-object manipulation policies. Our pair of encoders consists
of one convolutional neural network (CNN) trained with RGB image inputs on
sets of object pose targets and one graph neural network (GNN) [20] trained
with sets of object poses as both its inputs and targets (i.e., it learns to
autoencode the state). Since the CNN operates directly on RGB images, it gives
us the ability to learn a policy that can be deployed in the real world. Since
the GNN is doubly grounded in simulator state information (via input and
output), it provides a more accurate and stable representation, so we use it
both for calculating rewards and feeding to the value network in our actor-
critic RL algorithm. (The idea of providing different inputs to the policy and
value networks in this way is known as an asymmetric actor critic [10].) Fig.
1 provides an overview of our network structure and full approach.

To evaluate our approach, we construct a multi-object manipulation task (Fig.
2) which is challenging to learn from RGB images alone and ill-suited for
prior fixed-state representation learning approaches. We consider a collection
of 1-20 homogeneously shaped objects that start in arbitrary configurations on
a tabletop, which the robot must push into desired goal configurations. We
train on a simulated version of this task but we learn a policy that can be
deployed in the real-world without any fine tuning. We conduct extensive
comparison and ablation studies of our trained policy in both simulation and
on the physical robot. We find that our approach achieves higher success rates
than an autoencoder or a fixed-state training approach, while also
generalizing well to configurations not seen during training.

##  2 Related Work

We draw on and extend two main lines of robot learning research: self-
supervised representation learning and sim-to-real learning.

##### Self-Supervised Representation Learning:

A common pattern in robot learning methods is to pre-train a representation
using a self-supervised signal and then use that representation for downstream
task planning or task learning [7, 8, 9, 17, 21, 22, 23, 24, 25, 26, 27, 28].
The pattern usually goes as follows: researchers use an exploration policy
(often scripted [8, 9, 21, 22, 23]) to collect a dataset of real world
interactions, consisting of either single frames [7, 8], state action
transitions [9, 21, 25], or long sequences of frames and actions [22, 23].
They then use this dataset to train a model via a self-supervised signal, such
as an autoencoder loss [7, 8], GAN loss [17, 28], or some related pre-text
task [9, 21, 25, 26, 29, 30]. Once they learn a good representation, they
often use it in either a model-based/planning framework [9, 21, 22, 23, 25,
27, 30] or in a model-free RL framework [7, 8] to learn a task policy. In this
work, we follow a model-free RL approach. Compared to prior work, we learn
representations which are _grounded_ , both in their inputs and outputs.
Unlike most prior work, where representations are trained on image-based
predictions, we directly train our representations to capture the state of the
objects (through Eq. 2). We also directly consume ground truth information via
a GNN [20]. Our experiments suggest that these components speed up and
stabilize policy training and improve the quality and generalizability of
trained models.

##### Sim-to-Real Learning:

Sim-to-real methods—those trained primarily or exclusively in physics
simulators and targeted to work in the real world—are starting to yield
impressive results on physical robotics tasks [10, 11, 12, 13, 14, 15, 16, 17,
18, 28, 31, 32]. Simulation training offers several advantages over real-world
training, including increased speed and parallelizability of data collection,
safety of exploration, and full observability and control over environmental
factors. However, there is often a large gap between the crude simulator
phenomena and their real world analogs which causes naively-trained models to
fail in the real world. Researchers have developed several methods to help
enable policies to transfer across this gap, including using GANs to map from
simulation visuals to the real world [28] or vice versa [17], and
incorporating learned models from the real world into the simulation loop
[14]. One simple and effective approach we use, known as domain randomization
[32], is to randomize the physical and visual properties of an environment
during training. This variability makes trained policies more robust and less
prone to overfitting to simulator characteristics [10, 12, 15, 31, 32]. In
this work, we demonstrate a sim-to-real approach for speeding up training in
multi-object settings. Our approach is symbiotic with existing methods, scales
to a variable number of objects, and is extensible to more complex tasks
through its flexible state encoding.

##### Graph Neural Networks in RL:

GNNs are becoming popular in control and decision making after their strong
performance in natural language processing [33] and visual-spatiability tasks
[34]. DeepMind researchers use GNNs in RL to play the Star Craft II video game
[35, 36]. Ajay et al. [37] use a GNN for robotic manipulation to create a
hybrid analytic and data-driven simulator for task planning; however, they
rely on precise object knowledge and motion capture in the real world. Our
work is the first we know of to use GNNs to learn state representations for
robotic manipulation tasks.

##  3 Preliminaries

##### Goal-conditioned RL:

We use the standard multi-goal or goal-conditioned reinforcement learning
formalism, as in Schaul et al. [38]. We describe an environment by states
$s\in\mathcal{S}$, goals $g\in\mathcal{G}$, actions $a\in\mathcal{A}$, a
distribution for sampling initial state-goal pairs $p(s_{0},g)$, and a goal-
conditioned reward function
$r:\mathcal{S}\times\mathcal{A}\times\mathcal{G}\rightarrow\mathbb{R}$. To
account for partial observability, we denote images as observations
$o\in\mathcal{O}$, and images of goals as $g^{o}\in\mathcal{O}$. The objective
of the agent is to learn a goal-conditioned policy $a_{t}=\pi(o_{t},g)$ which
maximizes the expected discounted return
$\mathbb{E}[R_{t}]=\mathbb{E}[\sum_{i=t}^{\infty}\gamma^{i-t}r_{i}]$.

Figure 2: Top: Simulation and real world environments. Bottom: An example of a
state-action transition in simulation showing a trained policy pushing towards
a goal area, which is marked by a red square.

##### Task Overview:

The task we evaluate on requires an agent to push a collection of 1-20
homogeneous objects to desired areas on a tabletop. The agent’s state is
factored into a set of 2D coordinates of object centers
$\\{\mathbf{x}_{1},\mathbf{x}_{2},...\\}$. Similar to prior work [9], we use
pushing actions which are parameterized by 4 continuous values: a 2D starting
pose $(x,y,\theta)$ and a pushing distance $d$. These values are constrained
to the reachable table workspace of our physical Baxter robot [39]. For data
collection we use a simple scripted policy, which randomly samples an object
to push for a random distance in a random direction.

##### Graph Neural Networks:

We use a simplified version of the graph neural network framework introduced
by Battaglia et al. [20]. We define graphs by a 3-tuple $G=(V,E,\mathbf{u})$,
where $V=\\{\mathbf{v}_{i}\\}$ is the set of nodes with each $\mathbf{v}_{i}$
a vector representing the 2D pose of an object;
$E=\\{(\mathbf{v}_{r},\mathbf{v}_{s})\\}$ is an edge between a pair of
receiver, $\mathbf{v}_{r}$, and sender, $\mathbf{v}_{s}$, nodes; and
$\mathbf{u}$ is a global attribute vector which we compute to aggregrate the
full graph information. GNNs have been used in a variety of domains [34, 37]
and they vary widely in how they operate on graph structures [20]. However,
they all use some variation of two key functions: _updates_ and _aggregations_
[20]. Updates run computations on individual nodes or edges (e.g., linear
layer forward pass on each node). Aggregrations perform reductions across
graph structures (e.g., elementwise summation over the set of all nodes). See
Battaglia et al. [20] for a more in-depth introduction to GNNs.

##  4 Learning to Manipulate Object Collections

To train an image-based policy capable of tracking and reasoning about the
many possible configurations and interactions in multi-object tasks, we
develop a method to exploit variably-sized state information from a physics
simulator.

Our approach, illustrated in Fig. 1, consists of two phases: a supervised
representation learning phase and a reinforcement learning (RL) policy
training phase. The interesting details of our apporach lay mostly in how we
train our representations to capture variably-sized information and how we
incorporate them in the RL phase to aid in policy training. Subsection 4.1
covers the supervised learning phase and our solutions to the technical
challenges of consuming and predicting variably-sized object state information
(Fig. 3). Subsection 4.2 covers the RL phase and how we calculate rewards and
incorporate our learned representations into a full policy-training algorithm.

###  4.1 Learning Grounded State Representations

We seek to learn representations which capture the variably-sized and order-
invariant _set_ of objects in the environment state $s$. Formally, we seek to
learn two encoders $\mathbf{E}_{s}(s)$ and $\mathbf{E}_{o}(o)$ that map from
their respective modalities to latent vector spaces $\phi_{s}$ and $\phi_{o}$
which each represent the relevant information of the variably-sized state.
This objective introduces challenges both on encoding the inputs and decoding
the outputs of the neural networks. We discuss these challenges and our
solutions below.

##### The Input Side - Encoding Raw State:

Figure 3: CNN and GNN architectures and losses for self-supervised
representation learning

If our state, $s$, consisted of a single object pose $\mathbf{x}$, we could
simply learn our state encoder $\mathbf{E}_{s}$ via a multi-layer perceptron,
$\phi=\textbf{MLP}(\mathbf{x})$. To scale this approach to many objects, we
could increase the input size of the MLP and set unused values to zero when
there are fewer than the maximum number of objects present. Unfortunately, in
practice, the weak inductive bias of this approach makes it poorly suited for
moderately large numbers of objects. With a set of $n$ objects, there are $n!$
ways that these objects can be arranged:
$(\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{n})$,
$(\mathbf{x}_{n},\mathbf{x}_{n-1},...,\mathbf{x}_{1})$, etc. Each of these
will appear unique to the MLP, requiring an exponential order of training
examples for the network to learn in the worst case [20]. To scale to many
objects, we instead use a graph neural network (GNN) that operates on set
structures in an order-invariant way.

The specific graph network model we use is a single encoder block from the
transformer model [33]. One of the transformer’s primary mechanisms, the
multi-head dot-product attention (MHDPA) can be interpretted as implementing a
special parallelized version of updates and aggregations over a fully
connected graph [20]. (Others have demonstrated the use of MHDPA on
reinforcement learning tasks like Star Craft II [35, 36].) We convert our
state into a fully connected graph $G$ associating a vertex, $\mathbf{v}_{i}$,
with each object centroid, $\mathbf{x}_{i}$. We use the MHDPA operation to
compute an updated graph of the same shape, $G^{\prime}=\textbf{MHDPA}(G)$. To
aggregate information from the nodes $\mathbf{v}^{\prime}_{i}$ of
$G^{\prime}$, we compute a gated activation [40] on each node
$\mathbf{v}^{\prime}_{i}$ followed by an elementwise summation over the set of
all nodes. This yields a single vector that summarizes the full graph
information:
$\mathbf{u}=\sum_{i=1}^{n}\text{tanh}(W_{1}\mathbf{v}^{\prime}_{i})\odot\sigma(W_{2}\mathbf{v}^{\prime}_{i})$,
where $\odot$ denotes element-wise multiplication. (A simpler aggregation
without the gated activation:
$\mathbf{u}=\sum_{i=1}^{n}\mathbf{v}^{\prime}_{i}$ also works, just not as
well.) We then linearly transform $\mathbf{u}$ to achieve the desired vector
size of our latent space: $\phi=W_{3}\mathbf{u}$.

##### The Output Side - Learning State Representations:

We face similar concerns in constructing an appropriate loss function for
learning to predict a variable number of outputs. For the single object or
fixed-length state setting, we could use mean-squared error regression from
the latent space $\phi$ to that object pose $\mathbf{x}$: $\frac{1}{2}\lVert
g(\phi)-\mathbf{x}\rVert^{2}_{2}$ [10, 11, 19]. This approach, however, is
prone to failure in the multi-object setting. For our image-based encoder
network, $\mathbf{E}_{o}$, it would be ambiguous which labels
($\mathbf{x}_{1},\mathbf{x}_{2},...$) belong to which of several visually
indentical objects in the state; in this case, the network has no way of
knowing which object predictions it should place in which of its arbitrarily
assigned prediction slots ($\hat{\mathbf{x}}_{1},\hat{\mathbf{x}}_{2},...$).

Figure 4: Left: MDN learning curves for various networks trained with either
$\mathcal{L}_{\text{FULL}}$ or $\mathcal{L}_{\text{STATE}}$. Right:
Visualizations of the trained MDNs; orange crosses represent ground truth
locations. All networks capture the state of clustered objects well, but the
MLP fails to precisely localize the individual objects. Incorporating MHDPA
into the CNN leads to fewer false positives and greater precision. And the GNN
produces the tightest bounds and places the greatest amount of probability
mass correctly on the objects and not other points.

To sidestep this and other similar issues, instead of producing individual
predictions, we construct a mixture density network (MDN) [41].
$p_{\text{MDN}}(\mathbf{p}|\phi)$ encodes the probability of a 2D point
$\mathbf{p}$ on the table corresponding to one of the object centers
$\mathbf{x}_{i}$ conditioned on our latent representation $\phi$:

$p_{\text{MDN}}(\mathbf{p}|\phi)=\sum\limits_{k=1}^{K}\alpha(\phi)_{k}~{}\mathcal{N}\Big{(}\mathbf{p}\Big{|}\mu(\phi)_{k},~{}\Sigma(\phi)_{k}\Big{)}$
(1)

The MDN is a Gaussian mixture model, with weights $\alpha_{k}$, means
$\mu_{k}$, and variances $\Sigma_{k}$, computed as learned linear
transformations of the latent space. (See Fig. 4 for visualizations of MDN
predictions.) We train $p_{\text{MDN}}$ using a maximum likelihood loss which
optimizes the network to assign high probability to locations of ground truth
object centers ($\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{n}$): 111We
note that our MDN loss is specific to tasks where object types can be ignored.
One could instead use an object-type-sensitive loss or perhaps one which
assigns probability density to space contained within objects.

$\mathcal{L}_{\text{STATE}}=-\frac{1}{n}\sum\limits_{i=1}^{n}\text{log}~{}p_{\text{MDN}}(\mathbf{p}=\mathbf{x}_{i}|\phi)$
(2)

While $\mathcal{L}_{\text{STATE}}$ enables our network to learn a latent
representation that can accurately capture object position information, it
does not account for dynamics information. Some state encodings may be close
according to $\mathcal{L}_{\text{STATE}}$, but difficult to traverse between,
based on the actions available to an agent. To account for state
traversability and actionability, we train our latent space using forward and
inverse dynamics functions, as in prior work (e.g. [9, 42]):

$\mathcal{L}_{\text{DYN}}=\mathcal{L}_{\text{FWD}}+\mathcal{L}_{\text{INV}}=\frac{1}{2}\left\lVert
f_{\text{FWD}}(\phi_{t},a_{t})-\phi_{t+1}\right\rVert^{2}_{2}+\frac{1}{2}\left\lVert
f_{\text{INV}}(\phi_{t},\phi_{t+1})-a_{t}\right\rVert^{2}_{2}$ (3)

Here $f_{\text{FWD}}$ is a small two-layer MLP trained to predict the forward
dynamics in the latent space; while $f_{\text{INV}}$ is a small two-layer MLP
trained to predict the inverse dynamics—the necessary action to move between
two consecutive latent-space states. We use $\mathcal{L}_{\text{FULL}}$ to
train our encoders:

$\mathcal{L}_{\text{FULL}}=\mathcal{L}_{\text{STATE}}+\mathcal{L}_{\text{DYN}}$
(4)

We hypothesized that the addition of $\mathcal{L}_{\text{DYN}}$ would help
regularize the latent space and condition it for use in deciding actions,
leading to better performance in the real world. Our results seem to weakly
support this hypothesis, with an $\mathcal{L}_{\text{STATE}}$ model performing
slightly (but not significantly) better in simulation and an
$\mathcal{L}_{\text{FULL}}$ model generalizing slightly (but not
significantly) better to the real world. Figure 4 summarizes the training
results for our encoder networks.

##### Architecture Summary:

To summarize, we learn two separate encoders:
$\mathbf{E}_{\text{GNN}}:s\rightarrow\phi_{s}$ which maps from state, and
$\mathbf{E}_{\text{CNN}}:o\rightarrow\phi_{o}$ which maps from image
observations. Each encoder learns separate forward and inverse dynamics
functions as well as separate MDN parameters.222We first tried learning
$\phi_{s}$ and setting it as a regression target for our CNN encoder, but this
failed to produce useful estimates. We also tried sharing the output dynamics
and density weights, but this performed worse. Fig. 3 shows the full
architecture. We incorporate an MHDPA operation into the CNN as in prior work
[35].

As illustrated in Fig. 3, we treat each of the ($1\times 1\times C$ sized)
slices of the last ($W\times H\times C$ sized) CNN activation as elements of a
set (of cardinality $N^{W\times H}$), and compute the same MHDPA and
aggregation operations we use in the GNN. Incorporating GNN mechanisms into
CNN architectures this way has been shown to avoid some shortcomings of
vanilla CNN models in tasks that require tracking object counts and reasoning
about relative positions of objects [34], and we also find it helps.

###  4.2 RL Training

During the RL phase, we use our grounded state representations to train an
image-based policy via an off-policy actor critic algorithm (Soft Actor Critic
(SAC) [43] in this case). We use the doubly-grounded state-based
representation $\phi_{s}$ to both compute rewards and train the critic
efficiently. We apply domain randomization and use the image-based
representation $\phi_{o}$ to train the actor. Our full approach, which we call
Multi-Object Asymmetric Actor Critic (MAAC), is described in Algorithm 1.

Algorithm 1 MAAC: Learning manipulation tasks using a Multi-Object Asymmetric
Actor Critic

1:Encoders $\mathbf{E}_{\text{CNN}}(o_{t})$, $\mathbf{E}_{\text{GNN}}(s_{t})$,
replay buffer $R$, SAC [43] or other off-policy RL algorithm

2:Collect dataset $\mathcal{D}$ in simulation using scripted policy, while
applying domain randomization to images

3:Train encoders $\mathbf{E}_{\text{CNN}}$ and $\mathbf{E}_{\text{GNN}}$ on
$\mathcal{D}$ by optimizing (4)

4:for $\text{simulation episode~{}}e=1,\ldots,M$ do

5: Sample and embed initial state
$\phi_{s_{0}}=\mathbf{E}_{\text{GNN}}(s_{0})$, observation
$\phi_{o_{0}}=\mathbf{E}_{\text{CNN}}(o_{0})$, and goal
$\phi_{g}=\mathbf{E}_{\text{GNN}}(g)$

6: for $t=0,\ldots,T$ do

7: Execute action $a_{t}=\pi(\phi_{o_{t}},\phi_{g})$, and obtain new state
$s_{t+1}$ and observation $o_{t+1}$

8: Embed new state and observation
$\phi_{s_{t+1}}=\mathbf{E}_{\text{GNN}}(s_{t+1})$,
$\phi_{o_{t+1}}=\mathbf{E}_{\text{CNN}}(o_{t+1})$

9: Compute reward $r_{t}=r(\phi_{s_{t}},\phi_{g})$, and store transition
$(\phi_{s_{t}},\phi_{o_{t}},\phi_{g},a_{t},r_{t},\phi_{s_{t+1}},\phi_{o_{t+1}})$
in $R$

10: end for

11: Generate virtual goals ($g^{\prime}_{1},g^{\prime}_{2},...$) and rewards
($r^{\prime}_{1},r^{\prime}_{2},...$) for each step $t$ and store in $R$ (w/
HER [44])

12: Optimize actor ($\pi(\phi_{o_{t}},\phi_{g})$) using SAC with $o_{t}$
embeddings

13: Optimize critic
($V(\phi_{s_{t}},\phi_{g}),Q_{1}(\phi_{s_{t}},\phi_{g},a),Q_{2}(\phi_{s_{t}},\phi_{g},a)$)
using SAC with $s_{t}$ embeddings

14:end for

##### Algorithm 1 Overview:

First, before the RL phase, we run a scripted policy to collect a dataset
$\mathcal{D}$ of transitions of states and observations
($s_{t},o_{t},a_{t},s_{t+1},o_{t+1}$) (line 1). We use $\mathcal{D}$ to train
our encoders via supervised learning (line 2). Then, for each RL episode, we
first embed the goal and initial observation and state (line 4). At each step,
we pass the observation embedding $\phi_{o_{t}}$ through the policy network to
get an action (line 6). We also use the state embedding $\phi_{s_{t}}$ (along
with $\phi_{g}$) to compute a reward (line 8), and later to feed to the value
network. We parameterize the policy network
$\pi(\mathbf{\phi_{o_{t}}},\phi_{g})$ and the value networks
$\left(V(\mathbf{\phi_{s_{t}}},\phi_{g}),Q_{1}(\mathbf{\phi_{s_{t}}},\phi_{g},a),Q_{2}(\mathbf{\phi_{s_{t}}},\phi_{g},a)\right)$
by MLPs with two hidden layers with the learned embeddings as input; we
optimize them using standard SAC losses.

##### Specifying a Reward:

To specify a reward for multi-object tasks, we face similar considerations as
when embedding a variably sized state (Section 4.1). Our approach is to use
the GNN-based latent space vector to specify a distance between the current
state and goal state. To do this, we embed the current and goal states
$\phi_{s_{t}}=\mathbf{E}_{\text{GNN}}(s_{t})$ and
$\phi_{g}=\mathbf{E}_{\text{GNN}}(g)$. Then we specify a goal success
condition $f_{g}(s_{t})$ by computing the cosine distance between these
embeddings; when the cosine distance falls below a threshold $\epsilon$, the
goal is considered met. This can be described by the following equation:
$f_{g}(s_{t})=\texttt{cos\\_dist}(\phi_{s_{t}},\phi_{g})<\epsilon$. For every
step $t$ that the agent has not reached the goal, we provide a $-1$ reward.
Once it reaches the goal we provide a $+1$ reward. 333 We originally tried
using a denser reward based on changes in cosine distance to the goal, but
found it did not work as well; we think likely because it penalized incorrect
movement too much, hindering exploration [44]. We terminate episodes with a
final $-1$ reward if the goal condition has not been met after $T$ steps
(where $T\approx 50$).

##### Tricks to Speed Up Training:

We keep the encoder networks frozen in this phase and cache the latent
embeddings, increasing the maximum batch size and size of replay buffer we can
keep in memory and reducing the computation required to train the SAC heads.
To incentivize object movement, we add an extra reward of $+0.1$ when the
agent moves any of the objects more than $\delta$ cm (where $\delta\approx
2.0$). To improve exploration, we usually sample actions from our learned
policy $a_{t}=\pi(\phi_{o_{t}},\phi_{g})$ (line 6), but with probability $p$,
we sample actions from our scripted policy. We set $p$ to 1.0 early in
training and linearly anneal it over time. To help learn from our sparse
reward, we use Hindsight Experience Replay (HER) [44] to augment the replay
buffer (line 10). Also, our actual implementation is a parallelized version of
Algorithm 1, where we intermix SAC optimization cycles with experience
collection. 444We additionally tried directly learning a CNN policy
$\pi(O,\phi_{g})$ without pre-training a latent representation from images,
but found this to train much slower during RL as we could not use the cached
embeddings. See the supplementary material for these implementation details.

##  5 Experimental Results

In our experiments we study the following questions:

  1. 1.

How does the full method compare with alternative training formulations in
simulation? (Sec. 5.1)

  2. 2.

What is the effect of domain randomization and using $\phi_{s}$ in policy
training? (Sec. 5.2)

  3. 3.

How do the learned policies compare and generalize in real-world experiments?
(Sec. 5.3)

We train our policies purely in simulation and evaluate them in simulation and
the real world. We use two methods for sampling states; uniform: where all
objects are uniform randomly distributed across the table; and cluster: where
objects are sampled from 25cm x 25cm square areas on the table. For initial
state sampling $p(s_{0})$, we use uniform and cluster each 50% of the time.
For goal sampling, we always use the cluster approach. We train with 10 cubes,
but evaluate on various objects in the real world. We compare our full method
to several methods and ablations, including an MLP and autoencoder, an image-
based goal ($\pi(\phi_{s},\phi_{g^{o}})$ vs. $\pi(\phi_{s},\phi_{g})$), a CNN
that does not incorporate the MHDPA mechanism, and models trained with only
$\mathcal{L}_{\text{STATE}}$ or $\mathcal{L}_{\text{DYN}}$ rather than
$\mathcal{L}_{\text{FULL}}$.

Unless otherwise specified, we always use domain randomization and an
asymmetric actor critic (AAC) approach—using $\phi_{s}$ for both computing
rewards and value functions. $\phi_{s}$ and $\phi_{o}$ are always vectors of
size 128. We train all models on $\mathcal{L}_{\text{FULL}}$ (or some ablation
of it), besides the autoencoder approach, which uses a reconstruction loss.
555 We found that a naive autoencoder approach failed to train on this task
when applying domain randomization. We had to apply a semi-novel technique,
based on prior canonicalization work [17], to get it to work at all. Recall
that during RL training, we determine goal success as
$f_{g}(s_{t})=\texttt{cos\\_dist}(\phi_{s_{t}},\phi_{g})<\epsilon$. If
$\epsilon$ is too high, the task is trivially easy, and if it is too low, the
agent will never reach the goal. To ensure $\epsilon$ corresponds to
approximately equal state similarities across approaches, we collect a set of
transitions, compute a list of cosine distances for each model, and apply a
scaling based on the relative values that the models produce.

Figure 5:  Left: Sim training. Our full method reaches a higher success rate
in a fewer number of training iterations compared to an MLP approach, an
autoencoder approach, and several ablations. Right: Evaluation results. We
evaluate all trained models on a slight variation of the task that does not
rely on thresholds in a learned embedding space. We find that the full method
produces nearly the best performance, only being slightly beaten out by a
model trained on $\mathcal{L}_{\text{STATE}}$. We test models using both a
clustered and uniform initial state distribution of objects, and find that
other methods perform worse, especially with objects initially uniformly
distributed.

###  5.1 Simulation Eval Performance

To evaluate our training models in simulation, we run 100 trials of each
method and count the success rate of when they are able to push the cubes into
the desired regions. We cannot rely on the learned $\phi$ space to fairly
compare model performance, as some models could be more lenient of state
similarities. Instead we iterate through the positions of all the objects and
check if they fall within the 25cm x 25cm goal area. (See more detail in the
supplementary material.) The uniform distribution (which is sparse across the
table) better represents true multi-object handling, whereas a cluster of
objects is easier and more like a single-object setting. The results shown in
Fig. 5 illustrate that the MDN and full method best handle this more
challenging setting.

Figure 6: Sequence of states and pushing actions chosen by a policy trained
with our full approach leading to successful execution. The blue and red
square overlays represent the initial state and goal areas sampled with the
cluster state distribution. Contours represent the MDN output density
estimates of object locations.

###  5.2 Effects of Domain Randomization and Using $\phi_{s}$

Figure 7: Effects of domain randomization and asymmetric actor critic.

We found that applying domain randomization (DomRand) made the $\phi_{o}$
estimates noisy and highly variable, even when the underlying state did not
change. This ultimately led to our use of $\phi_{s}$ in an asymmetric actor
critic approach (AAC) to stabilize training signals. To study these effects,
we ran our full approach with different combinations of AAC and DomRand. When
applying DomRand, we find the use of AAC makes the difference between the
policy achieving high success rates and completely failing to learn. AAC seems
to help primarily by stabilizing learning signals.

###  5.3 Real World Evaluation Performance

We use a similar approach as we did in simulation for evaluating our policies
in the real world, but we always use cluster as our initial state
distribution. To study which loss function generalizes best, we compare the
full method, $\mathcal{L}_{\text{DYN}}$, $\mathcal{L}_{\text{STATE}}$, and the
autoencoder as these all learn different image models. Along with the 10 cube
task we train on, we test on tasks with 1 cube; 20 cubes; 10 spoons, knives,
and forks (Silverware); and 5 crumpled paper balls. Fig. 6 shows a successful
sequence of states and pushing actions taken by the robot using the trained
policy. Fig. 8 shows quantitative results. We see that all variants of our
approach significantly outperform the autoencoder, and
$\mathcal{L}_{\text{FULL}}$ slightly outperforms the ablations in these
trials. See the associated video for more experiment visualizations.

Figure 8: Left: Silverware task. Right: Real world evaluation results. We
evaluate on 15 different start states and goal locations for the 10 Cube
training task, and on 5 for the other generalization tasks. We terminate after
successes or 15 push actions. We hold start states and goals constant across
different methods. The full method produces the greatest performance and
generalizability, especially outperforming the autoencoder approach.

##  6 Discussion

We present an approach for sim-to-real robot manipulation learning that
exploits variable and multi-object simulator state. We do this by learning
grounded state representations and using them to train an RL policy that
outperforms alternative approaches in both simulated and real-world
experiments.

##### Limitations and Future Work:

We use a coarse goal specification of 25cm x 25cm areas. It would be
interesting to try to achieve more precise goal states, perhaps by gradually
annealing the threshold $\epsilon$ during training. Our scripted policy is
specific to the pushing task we consider, and the MDN loss
($\mathcal{L}_{\text{STATE}}$) is specific to tasks where objects can be
considered all of the same type. Exploration-based policies (e.g., [42]) and
set-based loss functions (e.g., [45]) could perhaps be used instead. We tried
uniform initial states in the real world, but found our policy often got
stuck. We believe incorporating memory (e.g., with recurrent neural networks)
may lead to more adaptive and robust policies.

#### Acknowledgments

Matthew Wilson was supported in part by the National Science Foundation under
Grant #1657596, as well as the Undergraduate Research Opportunities Program at
the University of Utah.

## References

  * Tella et al. [1982] R. Tella, J. Birk, and R. Kelley.  General purpose hands for bin-picking robots.  _IEEE Transactions on Systems, Man and Cybernetics_ , 12(6):828–837, 1982.
  * Mason and Salisbury Jr [1985] M. Mason and J. Salisbury Jr.  _Robot hands and the mechanics of manipulation_.  The MIT Press, Cambridge, MA, 1985.
  * Berenson and Srinivasa [2008] D. Berenson and S. Srinivasa.  Grasp Synthesis in Cluttered Environments for Dexterous Hands.  In _Humanoids_ , 2008.
  * Cosgun et al. [2011] A. Cosgun, T. Hermans, V. Emeli, and M. Stilman.  Push Planning for Object Placement on Cluttered Table Surfaces.  In _IROS_ , 2011.
  * Hermans et al. [2012] T. Hermans, J. M. Rehg, and A. Bobick.  Guided Pushing for Object Singulation.  In _IROS_ , 2012.
  * Mahler et al. [2019] J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg.  Learning ambidextrous robot grasping policies.  _Science Robotics_ , 4(26), 2019.
  * Finn et al. [2016] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel.  Deep spatial autoencoders for visuomotor learning.  In _ICRA_ , 2016.
  * Nair et al. [2018] A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine.  Visual reinforcement learning with imagined goals.  In _NeurIPS_ , 2018.
  * Agrawal et al. [2016] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine.  Learning to Poke by Poking: Experiential Learning of Intuitive Physics.  In _NeurIPS_ , 2016.
  * Pinto et al. [2017] L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel.  Asymmetric actor critic for image-based robot learning.  _arXiv preprint arXiv:1710.06542_ , 2017.
  * OpenAI et al. [2018] OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba.  Learning dexterous in-hand manipulation.  _CoRR_ , 2018.
  * Peng et al. [2018] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel.  Sim-to-real transfer of robotic control with dynamics randomization.  In _ICRA_ , 2018.
  * Xie et al. [2019] Z. Xie, P. Clary, J. Dao, P. Morais, J. Hurst, and M. van de Panne.  Iterative reinforcement learning based design of dynamic locomotion skills for cassie.  _arXiv preprint arXiv:1903.09537_ , 2019.
  * Hwangbo et al. [2019] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter.  Learning agile and dynamic motor skills for legged robots.  _Science Robotics_ , 2019.
  * Tan et al. [2018] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke.  Sim-to-real: Learning agile locomotion for quadruped robots.  _arXiv preprint arXiv:1804.10332_ , 2018.
  * Matas et al. [2018] J. Matas, S. James, and A. J. Davison.  Sim-to-real reinforcement learning for deformable object manipulation.  In _CoRL_ , 2018.
  * James et al. [2018] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis.  Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks.  _arXiv preprint arXiv:1812.07252_ , 2018.
  * Chebotar et al. [2019] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. D. Ratliff, and D. Fox.  Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience.  In _ICRA_ , 2019.
  * Zhang et al. [2016] F. Zhang, J. Leitner, M. Milford, and P. Corke.  Modular deep q networks for sim-to-real transfer of visuo-motor policies.  _arXiv preprint arXiv:1610.06781_ , 2016.
  * Battaglia et al. [2018] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al.  Relational inductive biases, deep learning, and graph networks.  _arXiv preprint arXiv:1806.01261_ , 2018.
  * Nair et al. [2017] A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbeel, J. Malik, and S. Levine.  Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation.  In _ICRA_ , 2017.
  * Ebert et al. [2017] F. Ebert, C. Finn, A. X. Lee, and S. Levine.  Self-supervised visual planning with temporal skip connections.  In _CoRL_ , 2017.
  * Ebert et al. [2018] F. Ebert, S. Dasari, A. X. Lee, S. Levine, and C. Finn.  Robustness via retrying: Closed-loop robotic manipulation with self-supervised learning.  In _CoRL_ , 2018.
  * Pinto and Gupta [2016] L. Pinto and A. Gupta.  Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours.  In _ICRA_ , 2016.
  * Zhang et al. [2018] M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. J. Johnson, and S. Levine.  Solar: Deep structured latent representations for model-based reinforcement learning.  _arXiv preprint arXiv:1808.09105_ , 2018.
  * Lee et al. [2019] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg.  Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks.  In _ICRA_ , 2019.
  * Sutanto et al. [2019] G. Sutanto, N. D. Ratliff, B. Sundaralingam, Y. Chebotar, Z. Su, A. Handa, and D. Fox.  Learning Latent Space Dynamics for Tactile Servoing.  In _ICRA_ , 2019.
  * Bousmalis et al. [2018] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, et al.  Using simulation and domain adaptation to improve efficiency of deep robotic grasping.  In _ICRA_ , 2018.
  * Byravan and Fox [2017] A. Byravan and D. Fox.  Se3-nets: Learning rigid body motion using deep neural networks.  In _ICRA_ , 2017.
  * Byravan et al. [2018] A. Byravan, F. Leeb, F. Meier, and D. Fox.  Se3-pose-nets: Structured deep dynamics models for visuomotor planning and control.  In _ICRA_ , 2018.
  * Sadeghi and Levine [2016] F. Sadeghi and S. Levine.  Cad2rl: Real single-image flight without a single real image.  _arXiv preprint arXiv:1611.04201_ , 2016.
  * Tobin et al. [2017] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel.  Domain randomization for transferring deep neural networks from simulation to the real world.  In _IROS_ , 2017.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention Is All You Need.  In _NeurIPS_ , 2017.
  * Santoro et al. [2017] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap.  A simple neural network module for relational reasoning.  In _NeurIPS_ , 2017.
  * Zambaldi et al. [2018] V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. Reichert, T. Lillicrap, E. Lockhart, et al.  Relational deep reinforcement learning.  _arXiv preprint arXiv:1806.01830_ , 2018.
  * Zambaldi et al. [2019] V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. Reichert, T. Lillicrap, E. Lockhart, M. Shanahan, V. Langston, R. Pascanu, M. Botvinick, O. Vinyals, and P. Battaglia.  Deep reinforcement learning with relational inductive biases.  In _ICLR_ , 2019.
  * Ajay et al. [2019] A. Ajay, M. Bauza, J. Wu, N. Fazeli, J. B. Tenenbaum, A. Rodriguez, and L. P. Kaelbling.  Combining physical simulators and object-based networks for control.  _arXiv preprint arXiv:1904.06580_ , 2019.
  * Schaul et al. [2015] T. Schaul, D. Horgan, K. Gregor, and D. Silver.  Universal Value Function Approximators.  In _ICML_ , 2015.
  * Fitzgerald [2013] C. Fitzgerald.  Developing Baxter.  In _2013 IEEE Conference on Technologies for Practical Robot Applications (TePRA)_ , 2013.
  * Oord et al. [2016] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ , 2016.
  * Bishop [1994] C. M. Bishop.  Mixture density networks.  Technical report, Citeseer, 1994.
  * Pathak et al. [2017] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.  Curiosity-driven Exploration by Self-supervised Prediction.  In _CVPR Workshops_ , 2017.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _ICML_ , 2018.
  * Andrychowicz et al. [2017] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. P. Abbeel, and W. Zaremba.  Hindsight experience replay.  In _NeurIPS_ , 2017.
  * Zhang et al. [2019] Y. Zhang, J. Hare, and A. Prügel-Bennett.  Deep set prediction networks.  _arXiv preprint arXiv:1906.06565_ , 2019.
  * Ziebart et al. [2008] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey.  Maximum entropy inverse reinforcement learning.  In _Association for the Advancement of Artificial Intelligence_ , 2008\.

##  Appendix A Hardware and Software

##### Hardware:

Figure 9: Close-up of compliant pushing end effectors.

All models for the full method were trained on a single desktop computer
(Intel i7-8700k CPU @ 3.70GHZ x 12) with an NVIDIA GTX 1080 Ti GPU. Some of
the comparison models were also trained on a shared NVIDIA DGX machine. We use
the Baxter robot from the late great RethinkRobotics [39], with custom
3D-printed plastic end effectors and foam inserts for compliant pushing of
objects. We use the ASUS Xtion Pro RGB-D camera to capture images during
inference. We only use the RGB channels, and we downscale the $640\times
480\times 3$ image the camera produces to size $84\times 84\times 3$ for use
in our CNN. We use the same render image size in simulation and downscale the
same way.

##### Software:

We use the Mujoco simulator with its built-in renderer. During training in
simulation, for simplicity, we do not articulate the robot arm to push the
objects. Instead, we articulate a disembodied paddle using simple PD control
(seen in Fig. 2). We also place invisible walls around the table to prevent
objects from falling off. To disincentivize policies from learning strategies
that would push objects off the table in the real world, we give -1 reward
each time one of the objects contacts the walls. In the real world, we use the
Move-It! manipulation software, with an RRT Connect planner to plan pushing
actions for the 7-dof Baxter arms. We choose to either use the right or left
arm based on where the pushing action is on the table. During our evaluation,
we run an autonomous loop where we: (1) collect an image from the ASUS camera,
(2) feed that image to the policy, (3) sample and execute an action, and (4)
repeat until 15 pushes have been executed or the goal has been reached. During
training, we always sample actions according to the Gaussian parameterized by
the SAC policy network. At inference, we continue to sample, but we reduce the
standard deviation by 50%. We find this strikes a good balance between having
too much noise, where the policy often executes suboptimal pushes, and having
too little noise, where the policy gets stuck in certain states when slightly
confused about the location of the objects.

We use TensorFlow as our deep learning framework. On top of that, we use
TensorFlow Probability 666https://www.tensorflow.org/probability to implement
our Mixture Density Network. And we use the Sonnet
777https://github.com/deepmind/sonnet and graph_nets
888https://github.com/deepmind/graph_nets frameworks developed by DeepMind to
implement our CNNs and GNNs, including by using their implementation of multi-
head dot product attention (MHDPA).

##  Appendix B Implementation Details

##### Dataset for Training the Encoders:

We collect 1 million transitions in our initial dataset for training the
model. This takes about 24 hours of wall time on the desktop machine. We did
not experiment with using less data, but we suspect 1 million is a bit
overkill; fewer examples may have worked just as well. We also apply data
augmentation by adding random image contrast (tf.image.random_contrast) and
noise (tf.random.normal) to the image as well as the embeddings before
applying the MDN and dynamics heads.

##### Autoencoder Details:

Our Autoencoder encodes its input into a latent space $\phi$ and then outputs
a set of logits of the same size as the observation image. To better control
for differences in the model, we use the same asymmetric actor critic apporach
we do with the full method. We use both a CNN and GNN that map to latent
spaces $\phi_{o}$ and $\phi_{s}$, repsectively. We then independently pass
both these latent spaces through a series of upconvolutions to output a 3D
array ($W\times H\times C$) of logits that parameterize independent Bernoulli
distributions which are trained to maximize the log probability of the true
pixel values. The difference between this and a full Variational Autoencoder
is that we do not enforce a prior distribution onto the latent space and we do
not sample from the latent space. We tried enforcing this prior and sampling,
but found that doing this did not allow the model to capture the object
positions, at least during initial development. We use the same architecture
for the GNN and CNN as we do in our full method. We find that the GNN-based
representation provides a similar performance boost as what we see in the full
method.

Figure 10: Domain randomized (1st & 3rd columns) and canonical (2nd & 4th)
observations. Domain randomized images are used for the inputs of the
Autoencoder, whereas a standard canonicalized version of the image is always
used for the output prediction. This canonicalized version is always from an
overhead view with no background distractors and only majenta objects.

Initially, we found that applying domain randomization led this Autoencoder
approach to fail. All of the capacity of the Autoencoder was wasted modeling
the randomized textures and varying camera characteristics and it was not able
to localize the objects. To solve this issue, we use the canonicalization
approach from James et al. [17]. The idea is to collect another image
observation at each time step that is in a “canonical" format which is not
randomized and thus the only source of difference is the object positions (see
Fig. 10).

##### MLP Details:

Our MLP consumes the set of $n$ object coordinates as an array of size
$n\times 2$ (which gets flattened) and produces a latent representation
$\phi$. We design it to have roughly the same number of parameters as the GNN,
as shown in Table 3. It consists of 3 hidden layers of size 256.

##  Appendix C Hyperparameters

We use the same hyperparameters, except for $\epsilon$, to train all of the
models during the supervised and reinforcement learning phases. We found the
GNN and MLP were robust to different learning rates, so we use a value which
works well for the CNNs, including the Autoencoder network. We find using
fairly large batch sizes was helpful to learn while applying noise via domain
randomization and data augmentation. We use tanh (instead of ReLU) activations
because prior sim-to-real work suggests that since tanh is a smoother
activation, it leads to better domain transfer performance [14].

We list the supervised learning hyperparameters in Table 1, the reinforcement
learning hyperparameters in Table 2, and the model parameter counts in Table
3. The asterisks (*) represent hyperparameters from the supervised phase that
stay the same in the RL phase.

Table 1: Supervised representation learning hyperparameters to train all models Hyperparameter | Value
---|---
optimizer* | Adam
learning rate | $3\times 10^{-4}$
batch size | $512$
number of iterations | $75000$
activation* | tanh
conv filters* | $64$
$\phi$ size* | $128$
MDN $K$* | $25$
transformer vector size* | $128$
transformer num heads* | $8$
Table 2: RL hyperparameters to train all models Hyperparameter | Value
---|---
learning rate | $1\times 10^{-3}$
batch size | $1024$
replay buffer size | $1\times 10^{6}$
$N$ (max episode steps before termination) | $50$
number of parallel environments | $8$
number of iterations | $10000$
$\phi$ noise | $0.1$
HER [44] k | $8$
SAC [43] entropy bonus ($\alpha$) | $0.1$
polyak averaging ($\rho$) | $0.995$
Table 3: Number of parameters in different network models (during supervised learning) Model | Number of parameters
---|---
GNN for Full | 232197
CNN for Full | 353157
CNN Autoencoder and Decoder | 937975
MLP (3 hidden layers of size 256) | 253445
Table 4: $\epsilon$ values Model |  $\epsilon$ Value
---|---
Full | 0.005
Autoencoder | 0.2
Only $\mathcal{L}_{\text{STATE}}$ | 0.04
Only $\mathcal{L}_{\text{DYN}}$ | 0.004
MLP | 0.005

##### Tuning $\epsilon$ Across Methods:

To ensure that all methods face a similarly difficult training task, we have
to make sure the $\epsilon$ we use correspond to similar state similarities.
We do this empirically by collecting a set of $4096$ transition pairs
($s_{t},s_{t+1}$) using our scripted policy. We filter out all transitions
where the objects have not moved at all. Then we compute the cosine distances
between all remaining pairs for each of the models. Finally, we compute
relative scalings between the values produced by each model. Through the
manual tuning of $\epsilon$ for various models, we found these relative
scalings correlated well with similar task difficulties and training
performances. The resultant $\epsilon$ values are shown in Table 4.

##  Appendix D Extra Background

###  D.1 Soft (Asymmetric) Actor Critic:

To train a goal-conditioned policy, we use a slightly modified version of the
Soft Actor Critic (SAC) algorithm [43]. SAC is a Deep Deterministic Policy
Gradient (DDPG) style algorithm, which optimizes a stochastic policy in an
off-policy way. It is based on the maximum entropy reinforcement learning
framework [46] and has been efficient and robust for learning real-world
robotics tasks. As opposed to vanilla SAC, we feed the value networks
$(V,Q_{1},Q_{2})$ a representation encoded from the underlying state, $s$,
while we feed the policy network $\pi$ a representation encoding the
observation (RGB image), $o$, in an asymmetric actor critic style approach
[10]. We have found that it leads to stable training and helped policies from
plateuaing too early in training, compared to other model-free RL algorithms
we tried.

###  D.2 Hindsight Experience Replay:

We also use Hindsight Experience Replay (HER) [44]. HER is a method for
augmenting replay buffer data by relabeling goals to aid training in sparse
reward settings. After sampling a goal $g$ and collecting an episode of
experience, HER stores the original transitions
$(s_{t},a_{t},r_{t},s_{t+1},g)$ in the replay buffer, as usual. Additionally,
for each step $t$, it samples a set of virtual goals
$(g^{\prime}_{1},g^{\prime}_{2},...)$ that come from future states of the
episode (which it has guaranteed to have reached), and it uses these to create
several relabeled transitions
$(s_{t},a_{t},r^{\prime}_{t},s_{t+1},g_{i}^{\prime})$ that it also stores in
the replay buffer. The denser rewards of these relabeled transitions is used
to bootstrap learning and make the policy more capable of reaching original
goals.


# Assessing the Performance of 1D-Convolution Neural Networks to Predict
Concentration of Mixture Components from Raman Spectra

Dexter Antonio
Department of Chemical Engineering
University of California, Davis
Davis, CA
<EMAIL_ADDRESS>
Hannah O’Toole
Department of Biomedical Engineering
University of California, Davis
Davis, CA
<EMAIL_ADDRESS>
Randy Carney
Department of Biomedical Engineering
University of California, Davis
Davis, CA
<EMAIL_ADDRESS>
Ambarish Kulkarni
Department of Chemical Engineering
University of California, Davis
Davis, CA
<EMAIL_ADDRESS>Ahmet Palazoglu
Department of Chemical Engineering
University of California, Davis
Davis, CA
<EMAIL_ADDRESS>GitHub

###### Abstract

An emerging application of Raman spectroscopy is monitoring the state of
chemical reactors during biologic drug production. Raman shift intensities
scale linearly with the concentrations of chemical species and thus can be
used to analytically determine real-time concentrations using non-destructive
light irradiation in a label-free manner. Chemometric algorithms are used to
interpret Raman spectra produced from complex mixtures of bioreactor contents
as a reaction evolves. Finding the optimal algorithm for a specific bioreactor
environment is challenging due to the lack of freely available Raman mixture
datasets. The RaMix Python package addresses this challenge by enabling the
generation of synthetic Raman mixture datasets with controllable noise levels
to assess the utility of different chemometric algorithm types for real-time
monitoring applications. To demonstrate the capabilities of this package and
compare the performance of different chemometric algorithms, 48 datasets of
simulated spectra were generated using the RaMix Python package. The four
tested algorithms include partial least squares regression (PLS), a simple
neural network, a simple convolutional neural network (simple CNN), and a 1D
convolutional neural network with a ResNet architecture (ResNet). The
performance of the PLS and simple CNN model was found to be comparable, with
the PLS algorithm slightly outperforming the other models on 83% of the data
sets. The simple CNN model outperforms the other models on large, high noise
datasets, demonstrating the superior capability of convolutional neural
networks compared to PLS in analyzing noisy spectra. These results demonstrate
the promise of CNNs to automatically extract concentration information from
unprocessed, noisy spectra, allowing for better process control of industrial
drug production. Code for this project is available at
github.com/DexterAntonio/RaMix.

_Keywords_ Raman Spectroscopy $\cdot$ Machine Learning $\cdot$ Mixtures
$\cdot$ Bioprocess Control $\cdot$ Signal Processing

##  1 Introduction

Industrial fermentation processes are used in the pharmaceutical industry to
produce biologic drugs, many of which have therapeutic effects unachievable
with traditional small molecule pharmaceuticals [1, 2]. Despite these unique
therapeutic benefits, biologic drugs are challenging to characterize and
manufacture [3]. Specifically, minute differences in process conditions can
subtly alter a protein drug’s structure, such as its glycosylation pattern,
and reduce the effectiveness of the drug [3, 4]. To ensure that the critical
quality attributes (CQAs) of pharmaceuticals are met, the biopharmaceutical
industry has adopted a quality-by-design (QbD) strategy, which achieves high
product quality by controlling critical process parameters (CPPs) during the
production process [5]. By maintaining CPPs with process control techniques
exacting drug specifications can be met. Complementing the QbD are process
analytic tools (PAT) that provide accurate measurements of the production
systems state, allowing for more advanced process control and further
increasing product quality [6, 7].

Real time process analytical tools which minimize or eliminate the lag between
sample collection and measurement are needed to detect production anomalies
sufficiently early to implement corrective actions [5]. Some system attributes
such as dissolved oxygen, pH, and temperature can easily be measured with
embedded sensors [8], whereas others such as product concentrations, and
glucose levels cannot [4]. Traditional methods for biochemical composition
measurements are liquid chromatography (LC), mass spectrometry (MS), and
combination methods (LCMS) [9]. Although these techniques are well developed
and highly accurate, they require sample collection, extensive sample
preparation unique to the analyte class, and trained experts to interpret the
results. These drawbacks combined with the high cost of their use (columns,
buffers, labor), limit their integration into large-scale production systems
[9]. One emerging class of techniques for monitoring the chemical composition
of fermentation processes is Raman spectroscopy [6, 9]. Unlike LCMS, Raman
spectroscopy can be configured to measure the chemical composition in situ,
without the need for sample preparation [9, 10]. This permits real-time,
continuous measurements made a distance, i.e., using non-destructive light
irradiation.

A hurdle to the widespread implementation of Raman spectroscopy for process
monitoring is the post-collection analysis required to extract chemical
concentrations from Raman signals [11, 12]. Addressing this issue, is the key
focus on the Ramix package and this paper. Traditional methods to extract
concentration information from Raman spectra require pre-processing procedures
(baseline correction, smoothing, and normalization) followed by peak fitting,
through a semi-manual curve fitting procedure performed by a trained
scientist. Advances in baseline correction algorithms and chemometric
algorithms allow for the automation of post-collection parameter extraction,
permitting online data analysis [5, 11, 12]. Two major classes of chemometric
algorithms for Raman spectroscopy analysis are partial least squares (PLS)
[13] and convolutional neural network (CNN) methods [14].

To assess the performance of different machine learning (ML) algorithms to
predict mixture concentrations from Raman spectra, a standard dataset is
needed. As of January 2023, no freely available dataset of Raman spectra of
mixtures was available. Naturally, the performance of these algorithms depends
on the level of pre-processing and noise in the dataset as well as the dataset
size, meaning that variations in the noise levels and baseline can affect the
overall model performance.

This lack of a standard dataset and unquantified effect of the baseline on the
model performance motivated the creation of RaMix, a Python package capable of
producing datasets of mixture spectra with various degrees of noise. By
examining how the model performance varies with different levels of
augmentations, the strengths and weaknesses of different model classes can be
assessed, without the need for massive experimental undertakings. Most
importantly, this package only requires a single, individual component
spectrum for each mixture compound to generate the synthetic spectra. Thus,
the RaMix package is able to produce synthetic spectra that can be used to
assess different chemometric algorithms. The purpose of the RaMix package is
help address this lack of a standard dataset by providing simulated spectra
that can be used to understand chemometric algorithms performance.

To demonstrate the use of RaMix and demonstrate how it can be useful for
assessing the performance of chemometric models, the RaMix package was
utilized to generate 48 distinct datasets from mixtures consisting of the
chemical signatures of five compounds, namely ethanol, glucose, water,
glycerol, and lactic acid. Cellular attenuation of the Raman signal was not
included in this simulation. The ML algorithms assessed were partial least
squares (PLS), a simple neural network (simpleNN), a simple convolutional
neural network (simpleCNN), and a 1D-Convolutional Neural network with a
ResNet architecture (ResNet).

The major contributions of this work are the development of RaMix, an
integrated software package that allows for the generation of datasets of
synthetic Raman spectra and the assessment of the effect of noise on
chemometric algorithm performance with the RaMix package. We first introduce
the structure of the RaMix package, before demonstrating this package’s
usefulness in assessing the effect of noise on chemometric model performance.
Finally, we present an overview of the implications and use cases of this
package and the future work needed to further quantify the effect of noise on
chemometric methods.

##  2 Experimental

A Raman spectrometer probes a solution and generates a Raman spectrum, which
contains information about the concentration and identity of each species in
the solution based on their inelastic scattering of incident light [15]. Due
to their underlying differences in rotational and vibrational Raman active
modes, each molecule has a unique Raman spectrum, which consists of several
peaks corresponding to each mode. These peaks can be modeled as Lorentzian
(Equation 1) in shape and are typically sharp with minimal overlap [15]. The
position of the peak in the spectra relates to the energy of the vibrational
or rotational mode, and this position can be altered by changes in the
chemical environment or bonding.

$L(x_{i})=\frac{a_{0}}{1+\frac{x_{i}-p_{0}}{w/2}^{2}}$ (1)

In Equation 1, $L(x_{i})$ is the intensity of the signal at wavenumber
$x_{i}$, $a_{0}$ represents the maximum amplitude, $p_{0}$ is the peak
position and $w$ is the full width at the half maximums. For spontaneous Raman
scattering, a spectra of mixtures components are approximately the sum of the
individual component spectra, weighted by their concentration. Deviation from
this linear relation occurs when non-covalent interactions between the solute
molecules alter the chemical bond strength of the solutes, changing the
individual component spectra through peak broadening and minor peak shifts.
Variation in laser power and instrument noise can also lead to small peak
shifts [15]. Another significant source of noise in Raman spectra is
fluorescence, which can be strong enough to overwhelm the Raman signal and
limits the usefulness of conventional Raman spectroscopy [9].

Quantum chemical calculation methods exist for simulating Raman spectra [16]
but are computationally expensive. An alternative simulation technique
involves using the Lorentzian fitting parameters to recreate the spectral line
shapes. This technique reduces the computational expense of simulating spectra
while still producing convincing synthetic spectra. Furthermore, by simulating
spectra in this manner, a direct understanding of the effect of peak noise on
prediction accuracy can be gained. Thus, simulating spectra using Lorentzian
line shapes was the technique chosen for the RaMix package. An experimental
study that utilized Raman spectroscopy for real-time monitoring of ethanol
production with baker’s yeast (_saccharomyces cerevisiae_) was chosen as a
model for the simulated spectra [10]. In the experimental study, ethanol,
glucose, water, glycerol, and lactic acid concentrations were monitored in
real-time using a Raman spectrometer embedded in the bioreactor. The data
analysis involved using a multi-linear regression model and accurate
concentration data from HPLC analysis to relate Raman spectra to sample
composition. The analysis was complicated by Raman signal attenuation from
high cellular density, thus additional modeling was performed in attempt to
compensate for this cell density-related Raman signal attenuation. This study
validated the use of Raman spectroscopy to monitor the aforementioned chemical
components in a real bioreactor. Therefore in the simulation study the same
five chemicals were used (Figure 1).

Figure 1: Comparison between the experimental spectra from the Biorad spectra
database (red) and the simulated individual component spectra generated from
the spectral skeletons (blue). The simulated spectra are generated through the
summation of Lorentzian peaks, with parameters manually tuned to ensure
resemblance between the experimental and simulated spectra

The RaMix package requires that the parameters for all the Lorentzian peaks
fit to the experimental spectra. These fitting parameters are referred to as
“spectral skeletons”. The Bio-Rad Raman spectral database contains freely
available images of Raman spectra and was utilized to locate spectra of the
five chosen components [17]. Lorentzian peaks were then fitted to these
spectra. The spectrum skeletons were visualized by using them to generate
Lorentzian peaks and compared to the Biorad spectra in Figure 1.

Noise was added to the individual peak parameters and a random baseline was
added to some of the simulated spectra. Noisy spectra skeletons were generated
to assess the sensitivity of the different chemometric algorithms to variation
in peak location, shape, and height. The noise was added to the spectrum
skeleton via a function which input a spectrum skeleton and noise parameters
and outputted a noisy spectrum skeleton, with random numbers added and
multiplied by each peak parameter (Figure 2).

Figure 2: Overview of the noisy spectrum skeleton generator function. The
function takes as input a spectrum skeleton and noise parameters and outputs a
“noisy spectrum skeleton with noise applied to each peak parameter”. Random
variables sampled from the log-normal distribution were multiplied by the
amplitude and the width parameter to generate the noisy spectrum skeleton,
where a random variable sampled from the Gaussian distribution was added to
the wavenumber position. Preliminary experiments indicated that the noise in
the peak location had less of an effect on the prediction compared to the
noise in the peak width and amplitude. Thus, in this study, a singular “noise
level” was used where $s_{1}=noise\ level$ and $s2=s3=noise\ level/10$. By
constraining the distribution parameters to a single metric, the effect of
noise on spectra prediction could more easily be explored. The noise levels
selected were $0$, $0.5$, $1.0$, $1.5$, and $2.0$ to span the somewhat
arbitrary qualitative range of “no-noise” to “high noise”.

To assess the performance of the ML algorithms on datasets with different
levels of noise, 48 separate datasets were generated. A diagram demonstrating
the generation of a single dataset is shown in Figure 3. The formula for the
mixture spectra $S_{mixture}$ is defined in equation 2

Figure 3: An overview of the random mixture spectra dataset generation.
Individual component spectra skeletons are transformed into noisy spectra
skeletons. These spectra skeletons are then used to generate simulated
individual component spectra, which are randomly scaled and combined to form
simulated mixture spectra. This generation process is repeated $m$ times, for
each datapoint in a simulated mixture dataset. The first stage in the creation
of the mixture spectrum dataset involved adding noise to each spectrum
skeleton to generate noisy spectrum skeletons. After creating the noisy
spectrum skeletons, each skeleton was used to generate pure component spectra
vectors. The pure component spectra vectors were combined in random linear
combinations to generate mixture spectra.

$S_{Mixture}=c_{1}S_{Ethanol}+c_{2}S_{Glucose}+c_{3}S_{Water}+c_{4}S_{Glycerol}+c_{5}S_{LacticAcid}$
(2)

In equation 2 $c_{i}$ is a random number between $0$ and $1$ and $S_{i}$ is a
$1x1800$ spectrum row vector. The weights applied to each spectrum were saved
along with the resulting mixture spectra. This process was repeated $m$ times,
where $m$ is the size of the datasets ranging from $10$ to $10,000$ for this
simulation experiment. The range $\left[0,1\right]$ is not representative of
the expected concentration of each species in a fermentation process, but this
simplification was made to make the data easier to interpret. Furthermore, the
Bio-R ad Raman spectral database contained spectra with normalized peaks,
making the intensity of each spectrum arbitrary. After generating the spectra,
additional noise was added to some of the data in the form of a noisy random
baseline. This baseline was generated by feeding a noisy random sine vector
into a baseline correction algorithm, which tried to fit a polynomial to the
sine function [18]. This process generated a random baseline that resembled
the fluorescence baselines typically found in Raman spectroscopy, which
results from the fluorescent glass composing the bioreactor. Additional random
noise was also added to the spectra. The final spectra, baseline, and species
concentrations were saved in data matrices, which allowed for the subsequent
data analysis to take place. Each dataset was split into a training and
testing set with 80% of the data going to the training set. The training set
was used to train the model and estimate the model parameters. By refraining
from testing each model on the test set until after the hyperparameter tuning,
a more accurate gauge of each model’s performance was obtained. Four different
algorithms were used to predict concentrations from the mixture spectrum:
Partial Least Squares (PLS), a simple Artificial Neural Network (simpleNN), a
simple Convolution Neural Networks (simpleCNN) and a 1D implementation of
ResNet-34 (ResNet). The Python package scikit-learn PLS algorithm was used for
the PLS regression [19]. Python’s Tensorflow 2.0 was used to build neural
networks and convolution neural networks [20].

As noted, the 48 datasets were each split into training and test sets. The
training set was utilized for fitting and the test set was utilized for model
validation. The models were trained on each datasets and their performance was
assessed. The spectral datasets were not standardized, however the y matrices
were standardized using Equation 3.

$y_{norm}=\frac{y_{data}-mean(y_{data})}{std(y_{data})}$ (3)

All three models were fitted to the training data set. The NNs and CNNs were
trained with a maximum of 200 epochs with early stopping implemented with a
0.75% training validation split. This early stopping criterion reduced the
training data available to the CNN models by 25% but reduced overfitting by
stopping the models before overfitting could occur. After the four models were
trained on the training data their predictive performance was assessed on the
test sets. The three related metrics that were used to assess the model’s
performance were explained variance, root mean squared error ($RMSE$), and
coefficient of variation ($R^{2}$).

##  3 Results and Discussion

The purpose of the RaMix package is to provide simulated spectra that can be
used to assess the ability of various chemometric algorithms to predict the
concentration of chemical species from simulated spectra with varying degrees
of noise. To demonstrate this capability, four separate chemometric algorithms
were assessed on 48 distinct datasets generated with the RaMix package. The
noise level, which is a single metric that encapsulates the noise added to the
peak parameters in each spectrum, varied between $0.0$ (no noise) and $2.0$
(significant noise). Example simulated spectra with and without baselines for
the noise level $1.5$ are shown in Figure 4.

Figure 4: Example simulated Raman spectra with noise levels of $1.5$ with and
without baselines with varying species concentrations. The spectra of mixture
spectra share approximately common peaks but vary in overall shape, intensity
and exact peak location.

The assessment of four chemometric algorithms was performed using the
performance metrics, $RMSE$, explained variance, and the coefficient of
determination, $R^{2}$, for each model on each dataset. The $R^{2}$ value was
chosen to select the best performing model and the result of that comparison
is shown in Figure 5.

Figure 5: Comparison of model performance on 48 datasets. The model with the
highest coefficient of determination, $R^{2}$, for a given dataset when
trained and tested on partitions of the dataset is indicated by colored
blocks.

Figure 5 shows that the PLS algorithm performed the best for most datasets
(40/48) yet was outperformed by the simple CNN model on the higher noise
datasets with $10,000$ data points. The strong performance of the PLS
algorithm is due to its ability to generate reasonable basis vectors from a
small dataset and the inherent assumption about a linear relationship between
spectral intensity and species concentration.

By contrast, CNNs require “large” datasets to fully train the model, make no
assumptions about the underlying functional relationship between the input and
output vectors, and incorporate the spatial relationship between data points
into the model. The additional complexity of the CNN models worsens their
performance on small, low-noise datasets without baselines, yet this
flexibility allows it to outperform the PLS model on the larger, high noise
datasets. The CNN’s convolutional layers also allow for partial baseline
removal with trained convolutional layers, explaining the algorithm’s better
performance on the large dataset, lower noise spectra with baselines.

Another useful comparison is the difference in individual model performance on
each dataset. Figure 6 shows this performance by displaying the $R^{2}$ value
for each dataset, rounded to two decimal places.

Figure 6: Individual model performance ($R^{2}$ value) for each dataset
rounded to two decimal places. Dark blue squares indicate higher performance
and white squares indicate lower performance. In situations where the model
did not converge, the R2 value is less than or equal to zero and is indicated
by the $\leq 0$ annotation.

Figure 6 shows the performance for the different models on various datasets.
The simpleNN is a poor-performing algorithm, achieving an $R^{2}$ value below
$0.80$ on all but one dataset. The ResNet model does not perform well on the
datasets with less than $1,000$ data points, including the low noise datasets.
The ResNet architecture was selected because previous studies that used CNNs
for Raman spectra classification tasks achieved high performance with ResNet
architectures. The poor performance can be explained by noting that the ResNet
architecture has many more parameters to train than the simple CNN. Figure 9
also reveals the similar performance between the PLS and simpleCNN algorithm
on datasets with $1,000$ and $10,000$ data points in the low noise categories
($0.0$ and $0.1$), with each model achieving comparably high performance
($R^{2}>0.98$). The PLS algorithm is insensitive to the dataset size above
$100$, yet the CNN algorithm continues to improve with increasing dataset size
indicating that the performance of the CNN algorithm has more potential for
improvement. To further compare the PLS and simpleCNN models, their
performance on two datasets was examined in detail. The datasets selected for
this analysis were the largest ($10,000$ data points) datasets with a noise
level of $0.5$, one having a baseline and the other without an added baseline.
A calibration plot for the two best-performing models on these datasets (PLS
and simpleCNN) is shown in Figure 7.

Figure 7: Comparison of the PLS, SimpleNN, simpleCNN resnet algorithm
performance on a $10,000$ point dataset with a noise level of $0.5$ with and
without a baseline.

Figure 7 indicates that, for the PLS algorithm, the predictions are evenly
clustered around the $x=y$ line, demonstrating that model is not biased. In
contrast, the predictions for glycerol in the simpleCNN with the baseline is
mostly under the $x=y$ line indicating a chronic underprediction of the
glycerol concentration. Surprisingly, neither model is dramatically affected
by the addition of a baseline, with its presence only leading to a slightly
more dispersed clustering around the $x=y$ line. The predicted versus actual
plots also verify the good performance of both models in predicting
concentration data from the simulated spectra and show the potential of CNNs
to act as a substitute for the PLS algorithm.

One of the main shortcomings of this simulation experiment is that the linear
way in which the data is generated does not account for spectra changes that
result from interactions among the solutes. Another shortcoming is that the
cellular attenuation of the Raman signal, an important confounder in real-time
process monitoring, is not incorporated into this model [10]. The performance
of the PLS algorithm was bolstered by the simple manner with which the data
was generated and by ignoring cellular attenuation. By adding these sources of
nonlinear behavior into the data generation step, the CNN performance would
likely surpass that of the PLS algorithm. Nevertheless, the RaMix model
provides an important first step in generating datasets useful for comparing
the performance of chemometric algorithms and can easily be extended to other
systems.

##  4 Conclusions

The RaMix package allows for the generation of synthetic Raman spectra, which
can be used to assess the performance of chemometric algorithms to predict
concentrations from simulated spectra. In this study, $48$ datasets were
generated with the package, and the predictive ability of CNNs and PLS
algorithms was assessed. CNNs were shown to have comparable performance to the
standard PLS algorithm and outperformed the PLS algorithm in certain high-
noise situations. The lack of large public Raman spectroscopy datasets makes
validating that the conclusions drawn from the synthetic data with
experimental data challenging. Nevertheless, the release of this package is an
important first step in the generation of standard datasets for chemometric
model assessment and will hopefully catalyze the public release of the large
spectra datasets needed for chemometric model refinement. Synthetic data also
provides the ability to probe the effects of distinct types of noise on
chemometric model performance and can complement the conclusions found from
the analysis of real experimental spectra. In addition, the use of
convolutional neural networks for multi-component mixture concentration
predictions has not previously been reported, and thus this study shows a new
application of CNNs to address a pressing issue in biotechnology.
Specifically, these results demonstrate that the use of CNNs for the
prediction of mixture concentrations holds considerable promise for the
automated real-time monitoring of bioreactors.

##  5 Code Availability

Code for this project is available at github.com/DexterAntonio/RaMix under an
MIT license.

## References

  * [1] Patrick Chames, Marc Van Regenmortel, Etienne Weiss, and Daniel Baty.  Therapeutic antibodies: successes, limitations and hopes for the future: Therapeutic antibodies: an update.  157(2):220–233.
  * [2] Henrique Neves and Hang Fai Kwok.  Recent advances in the field of anti-cancer immunotherapy.  3:280–288.
  * [3] Arnold G. Vulto and Orlando A. Jaquez.  The process defines the product: what really matters in biosimilar design and production?  56:iv14–iv29.
  * [4] Peiqing Zhang, Susanto Woen, Tianhua Wang, Brian Liau, Sophie Zhao, Chen Chen, Yuansheng Yang, Zhiwei Song, Mark R. Wormald, Chuanfei Yu, and Pauline M. Rudd.  Challenges of glycosylation analysis and control: an integrated approach to producing optimal and consistent therapeutic drugs.  21(5):740–765.
  * [5] André Guerra, Moritz Stosch, and Jarka Glassey.  Toward biotherapeutic product real-time quality monitoring.  39(3):289–305.
  * [6] Karen A. Esmonde-White, Maryann Cuellar, Carsten Uerpmann, Bruno Lenain, and Ian R. Lewis.  Raman spectroscopy as a process analytical technology for pharmaceutical manufacturing and bioprocessing.  409(3):637–649.
  * [7] Moo Sun Hong, Kristen A. Severson, Mo Jiang, Amos E. Lu, J. Christopher Love, and Richard D. Braatz.  Challenges and opportunities in biopharmaceutical manufacturing control.  110:106–114.
  * [8] Baowei Wang, Zhiwen Wang, Tao Chen, and Xueming Zhao.  Development of novel bioreactor control systems based on smart sensors and actuators.  8:7.
  * [9] Jens A. Iversen, Rolf W. Berg, and Birgitte K. Ahring.  Quantitative monitoring of yeast fermentation using raman spectroscopy.  406(20):4911–4919.
  * [10] Shuxia Guo, Thomas Bocklitz, and Jürgen Popp.  Optimization of raman-spectrum baseline correction in biological application.  141(8):2396–2404.
  * [11] Shuxia Guo, Petra Rösch, Jürgen Popp, and Thomas Bocklitz.  Modified PCA and PLS: Towards a better classification in raman spectroscopy–based biological applications.  34(4):1–10.
  * [12] Jinchao Liu, Margarita Osadchy, Lorna Ashton, Michael Foster, Christopher J. Solomon, and Stuart J. Gibson.  Deep convolutional neural networks for raman spectrum recognition: A unified solution.
  * [13] Cross decomposition — scikit-learn 0.24.2 documentation.
  * [14] Yasir Aslam and Santhi N.  A review of deep learning approaches for image analysis.  In 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT), pages 709–714. IEEE.
  * [15] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [16] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  Tensorflow: Large-scale machine learning on heterogeneous distributed systems, 2015.
  * [17] Kevin Buckley and Alan G. Ryder.  Applications of raman spectroscopy in biopharmaceutical manufacturing: A short review.  71(6):1085–1116.  Publisher: SAGE Publications Ltd STM.
  * [18] Michael S.  Lineshapes in IR and raman spectroscopy: A primer.  30(11):42–46.  Publisher: MJH Life Sciences.
  * [19] Shixuan He, Wei Zhang, Lijuan Liu, Yu Huang, Jiming He, Wanyi Xie, Peng Wu, and Chunlei Du.  Baseline correction for raman spectra using an improved asymmetric least squares method.  6(12):4402–4407.  Publisher: The Royal Society of Chemistry.
  * [20] Zhi-Min Zhang, Shan Chen, and Yi-Zeng Liang.  Baseline correction using adaptive iteratively reweighted penalized least squares.  135(5):1138–1146.  Publisher: The Royal Society of Chemistry.