§ INTRODUCTION

Decision makers acting in real-world problems often have to take into account multiple objectives. When maximizing one objective comes at the cost of another, the objectives are in conflict, and the decision maker must find a compromise between them. For example, maximizing the electricity output of a hydroelectric power plant comes at the expense of increased flooding risks downstream as well as irrigation deficits <cit.>. In the medical field, radiotherapy should generally maximize the destruction of cancer cells while minimizing the damage to the healthy surrounding tissue<cit.>. The optimal trade-offs might differ on a case-by-case basis, and they can be difficult to identify without a picture of all possible options.

Unfortunately, research work on optimizing a sequential decision problem often focuses on maximizing a single, scalar objective. For the aforementioned real-world problems, this means somehow combining the different objectives into a single target metric. A possible option is for the decision maker to ask an expert to manually engineer a metric so to induce a specific behavior after optimization. However, this approach has several drawbacks. First, such work can be expensive and error-prone, as it often requires domain expertise and extensive tuning before the optimization process finally produces a behavior that satisfies the decision maker. By focusing on a single solution at a time, this process also fails to inform the decision maker of all the trade-offs that could be possible. Finally, the decision maker has no way to convey directly their actual preferences, instead having to rely on the expertise of the designer to construct a suitable metric. These issues can severely restrict the influence that the decision maker has on the process, thus possibly missing their preferred solutions.

In contrast we can directly learn the best compromises by using an explicitly multi-objective approach. Assuming that improving an objective is always preferable, we can build a set of all optimal trade-offs called a Pareto front. Once learned, the Pareto front stays fixed, since it does not depend on the preferences of the decision maker. The decision maker can then use the Pareto front to review all available policies, and use this knowledge to select their preferred one <cit.>. This greatly simplifies the task of the decision maker, as the consequences of any given choice are clear and explainable in advance. Additionally, if the decision maker changes their mind at a later time, they can simply select a different policy without the need to re-tune an optimization metric and perform the learning process again.

In this work we propose a novel method, Pareto Conditioned Networks (PCN), that is able to efficiently learn the policies that belong to the Pareto front. conditions a single neural network on the desired compromise, so that it outputs the policy predicted to achieve it. Our method is sample-efficient, as it feeds the experience used for different compromises into the same network, thus allowing to share experience across policies. This also means it does not need to learn a policy independently for each trade-off found in the Pareto front, which is an approach taken by several works in multi-objective reinforcement learning (MORL) <cit.>. Moreover, we make minimal assumptions concerning the utility function — i.e., the range of possible preferences of the decision maker — as opposed to the often-used assumption of linear scalarization of the objectives in the MORL literature <cit.>. Finally, our method is scalable with respect to the number of objectives, as opposed to many of the current state-of-the-art MORL methods, who often limit themselves to 2- or 3-objective problems <cit.>.

§ BACKGROUND

§.§ Multi-Objective Reinforcement Learning

In reinforcement learning (RL), an agent learns to optimise its behaviour by interacting with the environment. In this paper, we deal with decision problems with multiple objectives, and model this as a multi-objective Markov decision process (MOMDP). A MOMDP is a tuple, $\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \gamma, \vec{\mathcal{R}}, n \rangle$, where $\mathcal{S}, \mathcal{A}$ are the state and action spaces respectively, $\mathcal{T} \colon \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \left[ 0, 1 \right]$ is a probabilistic transition function, $\gamma$ is a discount factor determining the importance of future rewards and $\vec{\mathcal{R}} \colon \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}^n$ is an $n$-dimensional vector-valued immediate reward function, with $n$ being the number of objectives of the problem. In single-objective RL, $n=1$ while in multi-objective reinforcement learning (MORL), $n>1$.

When $n=1$, the goal is to find the policy $\pi^*$ that maximizes the expected sum of discounted rewards:

\begin{equation}
\pi^*= \arg\max_\pi \mathbb{E} \left [ \sum_{t=0}^h \gamma^t r_t |\ \pi, s_0 \right ]
\end{equation}

In contrast, when $n > 1$, this sum can lead to returns for which, without any additional information, there is no clear winner (e.g., we cannot decide which return is optimal between $(0, 10)$, $(5, 5)$ or $(10, 0)$). A solution with values for all objectives lower than another is said to be dominated. A solution for which it is impossible to improve one of the objectives without hampering another (e.g., any of the solutions in the previous example) is said to be Pareto-efficient. The set of all Pareto-efficient solutions — i.e., the set of all best possible compromises — is called the Pareto front.

We can represent the decision maker preferences with a utility function $u : \mathbb{R}^n \rightarrow \mathbb{R}$, which converts the multi-objective returns back to a single scalar signal. Unfortunately, in most settings, the utility function is not known in advance: it either varies in each individual instance of the problem, or it cannot be easily formalized by the decision maker. Thus, in this work we assume that $u$ is unknown but monotonically increasing, and we focus on learning the full set of non-dominated solutions represented by the Pareto front. In addition, we must learn multiple separate policies that can reach all of the non-dominated solutions, since without knowing $u$ we must assume that any solution could be optimal. After learning, the decision maker can select their preferred solution and execute the corresponding policy.

We now define these concepts more formally. A policy $\pi$ is said to Pareto-dominate another policy $\pi'$ if its expected return $\mathbf{V}^{\pi}$, also called V-value, is higher or equal across all objectives than $\mathbf{V}^{\pi'}$, and there exist at least an objective where $\mathbf{V}^{\pi}$ is better than $\mathbf{V}^{\pi'}$:

\begin{equation}
\mathbf{V}^\pi \succ_P \mathbf{V}^{\pi'} \Longleftrightarrow (\forall i: \mathbf{V}^\pi_i \ge \mathbf{V}^{\pi'}_i) \land (\exists i: \mathbf{V}^\pi_i > \mathbf{V}^{\pi'}_i)
\end{equation}

where $\succ_P$ is the Pareto-dominance operator.

Our goal is then to find the set of policies that are not dominated by any other policy:

\begin{equation}
\Pi^* = \{\pi \in \Pi\ |\ \nexists \pi' \in \Pi : \mathbf{V}^{\pi'} \succ_P \mathbf{V}^{\pi}\}
\end{equation}

where $\Pi$ is the set of all possible policies. $\Pi^*$ then maps to the Pareto front $\mathcal{F} = \{\mathbf{V}^{\pi}\ |\ \pi \in \Pi^* \}$. In general, we call any set of V-values mapped from a set of policies a solution set, and any solution set composed only of non-dominated V-values a coverage set. Thus, the Pareto front is the optimal coverage set of the problem.

Learning the full set of Pareto-efficient policies $\Pi^*$ requires that the policies $\pi^* \in \Pi^*$ are deterministic stationary policies <cit.>. This is useful in settings where stochastic policies are not desired, such as the management of a hydroelectric power plant. In that scenario, the decision maker does not want to be presented with a policy that has a probability of completely draining the water reservoir even if that policy is optimal, as it would have catastrophic consequences for nearby towns <cit.>.

§.§ Multi-Objective Metrics

Comparing the learned coverage sets of different algorithms is a non-trivial task, as one algorithm's output might dominate the other in some part of the objective-space, but be dominated in another. Intuitively, one would generally prefer the algorithm that obtains better returns for a wider range of utility functions. In this work we use several metrics to evaluate an algorithm's performance.

The most widely used metric in the literature is called the hypervolume <cit.>. This metric evaluates the learned coverage set by computing its volume w.r.t. a fixed specified reference point. The reference point is taken as a lower bound on the achievable returns so that the volumes are always positive. Thus, the hypervolume envelops all possible V-values that are dominated by that coverage set, with more dominating coverage sets having a larger hypervolume. This metric is by definition the highest for the Pareto front, as no other possible solution can increase its volume (since they are all dominated). While the hypervolume metric is widely used and does give a measure of the coverage of a solution set, it can be difficult to interpret. The benefit of a certain increase or decrease in hypervolume is not readily apparent to the end user, and does not necessarily correlate to significant changes in expected utility. When working in high-dimensional objective-spaces, adding or removing a single point can lead to wildly different hypervolume values, especially if the point lies close to an extremum of the space.

To combat these limitations, we additionally evaluate our work with a different metric called the $\varepsilon$-indicator $I_\varepsilon$ <cit.>. $I_\varepsilon$ measures how close a coverage set is to the Pareto front $\mathcal{F}$. The $\varepsilon$-indicator of a coverage set $\hat{\Pi}$ is computed such that, for every $\mathbf{V}^{\pi}$ of the Pareto front, there exist a V-value in the coverage set that is at most $\epsilon$ smaller than $\mathbf{V}^{\pi}$:

\begin{equation}
\label{eq:epsilon-metric}
I_{\varepsilon} =
\inf_{\varepsilon\in\mathbb{R}}
\{
\forall \mathbf{V}^\pi\!{\in}\ \mathcal{F}, ~
\exists \mathbf{V}^{\pi'}\!{\in}\ \hat{\Pi} :\
|| V^\pi - V_o^{\pi'} ||_{\infty} \le \varepsilon
\}
\end{equation}

From a user's perspective, $I_\varepsilon$ has the intuitive meaning of showing that the proposed coverage set is at most $\varepsilon$ worse than any V-value of the Pareto front. The main disadvantage of this metric is that to compute it we need the true Pareto front, so it can only be used in test problems with a known structure.

While the $\varepsilon$-indicator has nice theoretical properties guaranteeing performance in the worst case <cit.> — i.e., the maximum utility loss (MUL) with respect to any possible utility function — it does not give any information about the expected utility loss (EUL). This makes it a highly pessimistic metric; the $\varepsilon$-indicator will still report bad performance even if nearly the whole Pareto front is learned exactly, as long as a single point is not correctly modeled.

While measuring the expected utility loss precisely requires knowing in advance the distribution of possible utility functions, we propose a variation on the $I_{\varepsilon}$ metric that aims to approximate an upper bound to this value, by making an additional assumption.

In particular, we propose computing $I_{\varepsilon-mean}$, that assumes that each point in the Pareto front is equally likely to be selected as the best choice by a randomly sampled utility function. We compute $I_{\varepsilon-mean}$ by taking the mean of the computed $\varepsilon$ values, which are computed in the same way as the original $I_\varepsilon$ metric. Because the maximal utility loss of incorrectly preferring another vector in the coverage set over a given vector in the Pareto front is $\varepsilon\sqrt{n}L$ <cit.>, where $L$ is the Lipschitz constant that described the level of continuity of the utility function, the $I_{\varepsilon-mean}$ metric describes an upper bound on the EUL, given that each vector in the Pareto set is (approximately) equally likely to be the vector that maximises the user's utility.

Figure <ref> shows a visual representation of the hypervolume and $\varepsilon$ metrics in 2 dimensions.

[legend pos=south west, xmin=-1, xmax=5, ymin=-1, ymax=5,ytick=0,1,2,3,4,5]
[only marks,mark=x,mark size=3] coordinates (-0.5,0);
[only marks,mark=*] coordinates (1,4)(2,2)(4,1);
[only marks,mark=o] coordinates (0.5,3)(0.75,2.3)(2.3,1)(3.3,0.7);

[cyan,opacity=0.2] (-0.5,0) – (-0.5,3) – (0.5,3) – (0.5,2.3) – (0.75,2.3) –
(0.75,1) – (2.3,1) – (2.3,0.7) – (3.3,0.7) – (3.3,0) – cycle;

[dashed, -|] (0.5, 3) – node[right]$\varepsilon_1$ (0.5,4);

[dashed, -|] (2.3,1) – node[right]$\varepsilon_2$ (2.3,2);
[dashed, -|] (3.3,0.7) – node[below]$\varepsilon_3$ (4,0.7);
[opacity=0.2,line width=2,orange, dotted] (-0.5,0) – (-0.5,4) – (1,4) – (1,2) –
(2,2) – (2,1) – (4,1) – (4,0) – (-0.5,0);

Pareto front and coverage set in 2-objective environment. The Pareto front is the set containing the best compromises that can be achieved (black dots). We want to learn a coverage set that is as close as possible to the true Pareto front (white dots). The hypervolume metric, in light blue, measures the volume of all dominated solutions w.r.t. some reference point (cross). The $\varepsilon$ metrics first compute the maximum distance between each point in the Pareto front and its closest point in the coverage set ($\varepsilon_i$). We can then take their maximum value to compute the $I_\varepsilon$ metric, or their mean value to obtain the $I_{\varepsilon-mean}$ metric of the coverage set.
The picture contains a set of points in a 2D space. The points belonging to the true Pareto front are black, while the other ones are drawn in black outlines. A light blue shade denotes the hypervolume of the coverage set.

§.§ Reward Conditioned Policies

Our work is inspired by the Reward Conditioned Policies algorithm proposed by <cit.>.
Using neural networks as function approximators in RL comes with many challenges. One of them is that the target (e.g., the optimal action of the policy) is not known in advance — as opposed to classical supervised learning where the ground-truth target is provided. As the behavior of the agent improves over time, the action used as target can change, often leading to hard-to-tune and brittle learners <cit.>.

Instead of trying to continuously improve the policy by learning actions that should lead to the highest cumulative reward, Reward Conditioned Policies flips the problem, by learning actions that should lead to any desired cumulative reward (be it high or low). In this way, all past trajectories can be reused for supervision, since their returns are known, as well as the actions needed to reach said returns. We can thus train a policy that, conditioned on a desired return, provides the optimal action to reach said return. By leveraging the generalization properties of neural networks, we can accumulate incrementally better experience by conditioning on increasingly higher reward-goals.

In our work, we condition our policy on a multi-objective return, such that we can execute policies to reach diverse points on the Pareto front. We propose a training regimen focused on increasing the current solution set uniformly across the whole objective-space to avoiding catastrophic forgetting.

§ RELATED WORK

Current  MORL methods can be broadly divided into two main categories: single-policy and multi-policy algorithms <cit.>. In the first case, one tries to learn a single, optimal policy for a given set of preferences, i.e., for a known utility function. In the second case, the utility function is unknown (or uncertain) and the goal is to learn a set of policies that cover all possible utility functions, i.e., a coverage set.

Most of the recent work on MORL falls into the second category and assumes an unknown, but linear utility function. However, this restricts the solution set that can be learnt, as linear scalarization assumes the Pareto front to be convex. In the linear setting, the goal is to train an agent such that the optimal policy can be recovered for any preference weights. <cit.> propose Optimistic Linear Support (OLS), a generic method that iteratively selects different sets of weights and calls a single-objective learner as subroutine to find the corresponding optimal policy. <cit.> extend this method for Deep RL. Another approach, taken by <cit.>, is to directly optimise on the coverage set without single-objective subroutine, by modifying the Bellman equation. Contrary to PCN, these methods are unable to discover any V-values on the concave regions of the Pareto front <cit.>.

Using conditioned networks has been explored in MORL, but again restricted to the linear scalarization setting. In <cit.>, Fitted Q-Iteration (FQI) is extended to use a modified Q-network conditioned on preference weights instead of target returns. Similarly, <cit.> use such a conditioned Q-network to extend Deep Q-Networks (DQN). Moreover, a similar network is used in <cit.>, in combination with a multi-objective Bellman operator.

Our work also can be related to imitation learning, as it also uses supervised learning to learn a policy <cit.>. However, imitation learning requires expert trajectories to train on, while PCN generates its own set of trajectories.

When the utility function can be any monotonically-increasing function, <cit.> adapt tabular Q-learning to directly learn the Pareto front. However, it is limited to discrete low-dimensional state-spaces.

Finally, <cit.> learn to reach the Pareto front using a modified policy gradient search, and <cit.> do this by modifying evolution strategies. Both algorithms make the same assumptions on the utility functions as PCN and are used as baselines in our experimental section (Section <ref>).

§ PARETO CONDITIONED NETWORKS

In this Section we introduce our main contribution, the Pareto Conditioned Networks (PCN) algorithm. The key idea behind our approach is to use supervised learning techniques to improve the policy instead of resorting to temporal-difference learning. As explained in Section <ref>, this eliminates the moving-target problem, resulting in stable learning.

PCN uses a single neural network that takes a tuple $\langle s, \hat{h}, \mathbf{\hat{R}}  \rangle$ as input. They represent, for state $s$, the return $\mathbf{\hat{R}}$ that PCN should reach at the end of the episode, i.e. the desired return of the decision maker. The desired horizon $\hat{h}$, that says how many timesteps should be executed before reaching $\mathbf{\hat{R}}$. At execution time, both $\hat{h}$ and $\mathbf{\hat{R}}$ are chosen by the decision maker at the start of the episode.

PCN's neural network has a separate output for each action $a_i \in \mathcal{A}$. Each output represents the confidence the network has that, by taking the corresponding action, the desired return will be reached in the desired number of timesteps. We can draw an analogy with a classification problem where the network should learn to classify $(s, \hat{h}, \mathbf{\hat{R}})$ to its corresponding label $a_i$.

Similarly as with classification, PCN requires a labeled dataset with training examples to learn a mapping from input to label. Contrary to classification, however, the data present in the dataset is not fixed. PCN collects data from the trajectories experienced while exploring the environment (see Section <ref>). Thus, the dataset improves over time, as we collect better and better trajectories. In particular, since the ability of PCN to reach Pareto-dominating solutions depends on the data on which its network is trained, we want to keep only relevant experiences. We do so by limiting the size of the dataset and pruning it in such a way so to keep only tuples with $\mathbf{\hat{R}}$'s from different parts of the objective-space (see Section <ref>).

We collect new data for several episodes, after which we re-train the network with a number of batch updates from the new dataset (see Section <ref>). This improves the policies induced by the network, which in turn allows to gather better data for the next training batch.

§.§ Building the dataset

As mentioned before, each datapoint in PCN's dataset is comprised of an input $\langle s, \hat{h}, \mathbf{\hat{R}} \rangle$ and an output $a$. These are computed from observed transitions in the environment. As the dataset is empty at first, we execute a random policy on the environment for the first few episodes in order to collect a variety of trajectories.

After each episode is completed we store its trajectory. Then, for each timestep $t$ of the trajectory, we know how many timesteps are left until the end is reached, i.e., the episode's horizon $h_t=T-t$. We can also compute the cumulative reward obtained from timestep $t$ onward, i.e., $\mathbf{R}_t = \Sigma_{i=t}^{T}{\gamma^i \mathbf{r}_i}$. Since for this trajectory executing action $a_t$ in state $s_t$ resulted in return $\mathbf{R}_t$ in $h_t$ timesteps, we add a datapoint with input $\langle s, \hat{h}, \mathbf{\hat{R}}\rangle = \langle s_t, h_t, \mathbf{R}_t \rangle$ and output $a=a_t$ to the dataset. In other words, when the observed return corresponds to the desired return in that state, then $a_t$ is the optimal action to take. Figure <ref> shows how a full trajectory is decomposed into individual datapoints.

[scale=0.65,every node/.style=scale=0.75]
[] at (0,1.3) (t0) $t_0 = T$;
[] at (0,0)  (s0) $s_0$;
[] at (0,-1.3)  (r0) $R_0$;
at (2,0) (a0) $a_0$;
[draw=red,diamond,line width=1,fill=red,fill opacity=0.05] at (2,0) (a0box) $a_{T-1}$;

[blue,line width=1,fill=blue,fill opacity=0.05] (t0.north west) rectangle (r0.south east -| t0.north east);
[->,line width=1] (s0-|t0.east) – (a0box);

[] at (4.1,1.3) (t1) $T - 1$;
[] at (4.1,1.3) (t1box) $t_0 = T$;
[] at (4.1,0)  (s1) $s_1$;
[] at (4.1,-1.3)  (r1) $R_1$;
at (6.1,0) (a1) $a_1$;
[draw=red,diamond,line width=1,fill=red,fill opacity=0.05] at (6.1,0) (a1box) $a_{T-1}$;

[blue,line width=1,fill=blue,fill opacity=0.05] (t1box.north west) rectangle (r1.south east -| t1box.north east);

[->,line width=1] (a0box) – (s1-|t1box.west);
[->,line width=1] (s1-|t1box.east) – (a1box);

[] at (8.1,1.3)  (tn) ...;
[] at (8.1,0)  (sn) ...;
[] at (8.1,0)  (snbox) $t_0 = T a$;
[] at (8.1,-1.3) (rn) ...;

[->,line width=1] (a1box) – (snbox.west);

[draw=red,diamond,line width=1,fill=red,fill opacity=0.05,text opacity=1] at (10.1,0) (at) $a_{T-1}$;

[->,line width=1] (snbox.east) – (at);

[draw,circle,line width=1] at (12,0)  (st) $s_T$;

[->,line width=1] (at) – (st);

[draw,circle,line width=1] at (a0.west|-0,-1.3) (p0) +;
[draw,circle,line width=1] at (a1.west|-0,-1.3) (p1) +;

[line width=1] (a0box.east) to [out=-45,in=0] node[right] $r_0$ (p0.east);
[line width=1] (a1box.east) to [out=-45,in=0] node[right] $r_1$ (p1.east);
[->,line width=1] (at.east) to [out=-45,in=0] node[right=5pt] $r_{T-1}$(snbox.east |- rn);

[->,line width=1] (p0) – (t0.east |- r0);
[->,line width=1] (t1box.west |- r1) – (p0);
[->,line width=1] (p1) – (t1box.east |- r1);
[->,line width=1] (snbox.west |- rn) – (p1);

[decoration=calligraphic brace,amplitude=8pt,raise=10pt,line width=1,decorate]
(t0.north west) – node[above=20pt]datapoint $0$(a0box.east|-t0.north);
[decoration=calligraphic brace,amplitude=8pt,raise=10pt,line width=1,decorate]
(t1box.north west) – node[above=20pt]datapoint $1$(a1box.east|-t1box.north);
[pen colour=white,decoration=calligraphic brace,amplitude=8pt,raise=10pt,line width=1,decorate]
(tn.north west|-t0.north) – node[above=20pt]datapoint $T-1$(at.east|-t0.north);

(at.west |- 0,3) rectangle (14,0);
[decoration=calligraphic brace,amplitude=8pt,raise=10pt,line width=1,decorate]
(tn.west|-t0.north) – (at.east|-t0.north);

(r0.south) rectangle (at.west |- 0,3);
[decoration=calligraphic brace,amplitude=8pt,raise=10pt,line width=1,dashed,decorate]
(tn.west|-t0.north) – (at.east|-t0.north);

(legend) at (t0.west |- 0,-2);

[matrix of nodes,draw,below right,column sep=10pt] at (legend)
[shape=rectangle,line width=2,draw=blue,fill=blue,fill opacity=0.05,label=right:input] ;
[shape=diamond,inner sep=2.5pt,line width=2,draw=red,fill=red,fill
opacity=0.05,label=right:classplabel] ;

Conversion from a trajectory to labeled datapoints. For each timestep, we extract a single datapoint. The input (blue) is composed of the state at that timestep, and total return and number of timesteps until the end of the episode. The label (red) is the action taken at that timestep.
A series of boxes and diamonds connected by arrows, showing the progression of state, returns, horizons and actions for each timestep along a single trajectory.

§.§ Training the Network

PCN's network architecture uses a separate embedding for the state and another one for the desired return and horizon. The desired return and horizon are concatenated together and multiplied by a scaling factor to normalize their values. They then pass through a single fully connected layer followed by a sigmoid function. Similarly, the state-embedding also ends with a fully connected layer and sigmoid activation. Both layers have the same number of output nodes (we use 64 for all our experiments), which are combined together using the Hadamard product. The Hadamard product has been shown to make a more effective use of conditioning variables <cit.>, and the sigmoid used on both outputs ensure that both embeddings are equally important. Finally, the resulting output passes through a multilayer perceptron that has a separate output node for each action, and a single hidden layer of 64 nodes with a ReLU activation function.

PCN trains the network as a classification problem, where each class represents a different action. Transitions $x= \langle s_t, h_t, \mathbf{R}_t \rangle, y=a_t$ are sampled from the dataset, and the ground-truth output $y$ is compared with the predicted output $\hat{y}=\pi(s_t, h_t, R_t)$. The predictor (i.e., the policy) is then updated using the cross-entropy loss function:

\begin{equation}
H = -\sum_{a \in \mathcal{A}}{y_a \log \pi(a|s_t,h_t,\mathbf{R}_t)}
\end{equation}

where $y_a = 1$ if $a = a_t$ and $y_a = 0$ otherwise.

The network is re-trained periodically, but only after a set number of episodes (which is a hyperparameter that depends on the problem), to ensure that enough new experience has been collected and that the underlying dataset has been improved sufficiently.

§.§ Policy Exploration

As our dataset is composed of transitions collected from training experience, we can see that the quality of our dataset crucially depends on the quality of the executed trajectories. It is unrealistic to expect PCN to reliably produce trajectories with high-valued desired returns when it has only been trained on datapoints originating from random trajectories. Rather, we can expect PCN to produce trajectories with returns in the range of the ones from the current training data. Therefore, if we obtain trajectories reaching high returns, PCN will be able to confidently return high-return policies.

PCN leverages the fact that, due to the generalization capabilities of neural networks, the policies obtained from the network will still be reliable even if the desired return is marginally higher than what is present in the training data. In fact, they will perform similar actions to those in the training data, but lead to a higher return. Thus, we incrementally condition the network on better and better returns, in order to obtain trajectories that extend the boundaries of PCN's current coverage set.

More precisely, we randomly select a non-dominated return $\mathbf{R}_{nd}$ and its corresponding horizon $\hat{h}$ from the dataset. By randomly picking a non-dominated return from the entire coverage set we ensure equal chance of improvement to each part of the objective space. However, using $\mathbf{R}_{nd}$ exactly would induce the network to only replicate the already observed sampled trajectory so, as a second step, we choose a single objective $o$ to improve upon. We then increase the desired value for that objective to obtain a new target return. PCN determines the magnitude of the increase by computing the standard deviation $\sigma_o$ for the selected objective, using all non-dominated returns from the trajectories in the dataset. The magnitude is then sampled from the uniform distribution $U(0, \sigma_o)$ and added to $R_{nd,o}$ to form our desired return $\mathbf{\hat{R}}$. By restricting the improvement to at most $\sigma_o$, $\mathbf{\hat{R}}$ stays in the range of possible achievable returns and, by only modifying one objective at a time, the changes to the network's input compared to the training data are kept at a minimum.

With $\mathbf{\hat{R}}$ and $\hat{h}$ selected, PCN can condition its network and act during the training episode. At the start of the episode, $\langle s_0, \hat{h}, \mathbf{\hat{R}}\rangle$ results in executing $a_0$ and observing $\mathbf{r}_0, s_1$. PCN then updates the desired return and horizon such that they stay consistent throughout the episode: $\mathbf{\hat{R}} \longleftarrow \mathbf{\hat{R}} - \mathbf{r}_0$ and $\hat{h} \longleftarrow \max(\hat{h}-1, 1)$. We ensure that $\hat{h}$ is at least 1 to avoid impossible desired horizons. can then choose an action for $s_1$. This process is repeated until PCN encounters a terminal state. The trajectory is then added to the dataset, using the conversion to datapoints explained in Section <ref>.

To increase the range of observations in the environment during training, PCN samples actions from a categorical distribution with each action's confidence score corresponding to its probability of being sampled. This is done  by using a softmax function on the network's output. Please note that at execution time — i.e., after the training process — we use a deterministic policy by systematically selecting the action with the highest confidence. This is because, as mentioned in Section <ref>, only deterministic policies are optimal when learning the complete set of Pareto-efficient policies.

bluenode/.style=shape=circle, draw=blue, fill=blue, inner sep=2pt, outer sep=4pt,
rednode/.style=shape=circle, draw=red, fill=red, inner sep=2pt, outer sep=4pt
[legend pos=south west, xmin=0, xmax=5, ymin=0, ymax=5,xtick=1,2,3,4,5,ytick=1,2,3,4,5]

[only marks,mark=o] coordinates (1,1)(1.6,2.8)(2.5,3)(2.9,2);

[only marks,mark=o] coordinates (2.7,2.9);

[only marks,mark=*,orange] coordinates (2,4)(3,3)(4,1);

[only marks,mark=*,black] coordinates (1.9,3.5);
[dashed,blue] (1.6,2.8) – (1.6,0);
[dashed,blue] (2,4) – (2,0);
[dashed,blue] (2.5,3) – (0,3);
[dashed,blue] (2,4) – (0,4);
[blue,line width=3pt] (1.6,0) – (2,0);
[blue,line width=3pt] (0,3) – (0,4);
[red,line width=0.7pt,-latex] (1.9,3.5) – (1.985,3.94);

[draw,below left=1cm] at (current bounding box.north east)
[rednode,label=right:$I_{l2}$] ;

[bluenode,label=right:$I_{cd}$] ;

The $I_{ds}$ metric for the solid black dot combines its negative L2-norm distance (red arrow) to its closest non-dominated neighbor (orange), and its crowding distance as the sum, for each dimension, of the max distance between a point's upper and lower neighbour (blue).
A 2D plot showing a current solution set, with both dominated and non-dominated points. An arrow denotes the L2-norm distance from a point to its closest non-dominated point. A set of blue dashed lines denote the crowding distance, as distances of that same points to its closest neighbors in both dimensions.

§.§ Updating the Dataset

As it collects new experience, PCN needs to use it to train its network without forgetting about previous, relevant experience. Unfortunately, measuring relevance of data in our setting is non-trivial. We mainly care about the non-dominated solutions, since those are the ones that compose our coverage set, so ideally we would only keep their associated trajectories in our dataset. However, focusing solely on the current coverage set can lead to performance degradation if it is composed of too few V-values. This is because, during exploration (see Section <ref>), we will keep collecting very similar trajectories to the ones we already have. In turn, this will reinforce PCN's strategy to focus only on these few policies, disrupting the learning process. Thus, throughout training we keep a dataset of $N$ trajectories, favoring trajectories that span different parts of the objective-space while removing highly clustered solutions. To do this we employ an additional metric, the crowding distance, that assigns a lower score to points with close neighbors. We then combine both the distance to the coverage set and the crowding distance in a single metric, which we use to prune less relevant points from the dataset.

We measure our preference for non-dominated V-values by computing for each solution its negative L2-norm distance, $I_{l2}$, to its closest non-dominated solution in the dataset. Non-dominated V-values thus score the highest with $I_{l2} = 0$.

\begin{equation}
I_{l2, i} = -\min \norm{p_{i}-p_{j}}_2, p_{j} \in \hat{\Pi}
\end{equation}

where $i$ is the index of the $i$-th solution in the dataset and $p_j$ is a non-dominated point in the current coverage set $\hat{\Pi}$.

We measure the level of clustering of a V-value, $I_{cd}$, using the crowding distance <cit.>. It assigns a score for each solution based on the distance between its neighbours in each dimension. Thus, V-values with close neighbors will have lower scores, while more isolated points will have higher scores. Algorithm <ref> shows how the crowding distance is computed in practice.

Crowding Distance

[1]
$p$ points
The crowding distance $I_{cd}$
$o \gets 0$ to $n$
$s_i \gets \argsort p_{.,o}$
$j \gets 0$ to $\texttt{len}(p)$
$n_u, n_d \gets s_{i_{j+1}}, s_{i_{j-1}}$
$c_{s_{i_j}, o} = p_{n_u} - p_{n_d}$
$I_{cd} \gets \sum_{o}{c_{.,o}}$

We then combine $I_{l2}$ with $I_{cd}$ in a single score metric, which we call dominating score, $I_{ds}$. We define it as:

\begin{equation}
I_{ds,i} =
\begin{cases}
I_{l2,i} & \texttt{if } I_{cd,i} > 0.2 \\
2(I_{l2,i}-c) & \texttt{if } I_{cd,i} \leq 0.2
\end{cases}
\end{equation}

Note that since $I_{l2}$ corresponds to a negative distance, if a point is crowded — i.e., $I_{cd,i} \leq 0.2$ – we double the distance penalty. In addition, we add an additional small penalty $c$ to crowded points to prune, to detect and prune duplicate points on the coverage set. Figure <ref> shows an example on how to compute the L2-norm and crowding distances for a given solution set.

3c||hypervolume     3c|$I_\varepsilon$ indicator

|=|= = =||= = =|
(ours)     MONES     RA     (ours)     MONES     RA

DST     $\mathbf{22845.40 \pm 19.20^*}$     $17384.83 \pm 6521.10$     $22437.40 \pm 49.20$     $\mathbf{0.039 \pm 0.087^*}$     $0.687 \pm 0.222$     $0.667 \pm 0.000$

Minecart     $\mathbf{197.56 \pm 0.70^*}$     $123.81 \pm 23.03$     $123.92 \pm 0.25$     $\mathbf{0.271 \pm 0.087^*}$     $1.596 \pm 0.889$     $1.000 \pm 0.000$

Crossroad     $\mathbf{539.53 \pm 6.27^*}$     $429.09 \pm 27.47$     $466.02 \pm 31.23$     $\mathbf{0.247 \pm 0.172^*}$     $0.660 \pm 0.200$     $0.408 \pm 0.039$

Mean and standard deviation of hypervolume and $\varepsilon$-indicator across all 5 runs for all algorithms. For hypervolume, higher is better. For $I_\varepsilon$, lower is better. Best results are highlighted in bold and with an asterisk.

§ EXPERIMENTS

Most state-of-the-art algorithms in MORL assume that the utility function can be expressed as a linear scalarization. This makes them unsuitable as relevant baselines, as their setting is different from ours. Thus, for all experiments, we compare PCN with 2 baselines that share our same assumptions on the utility function and that learn a set of policies that estimates the whole Pareto front. The first baseline, Multi-Objective Natural Evolution Strategies (MONES) <cit.> uses a parametrized policy. It learns a distribution over the policy parameters such that sampling from this distribution produces a Pareto-efficient policy. Different samples produce different Pareto-efficient policies, each leading to a different non-dominated return. The advantage of this method is that it can produce an infinite number of different policies. However, the main drawback is that we do not know the return of the sampled policy without executing it first.

The second baseline used is the Radial Algorithm (RA) <cit.>. RA trains a fixed number of independent policies. Each policy is trained using a policy gradient algorithm, where the gradients w.r.t. the different objectives are weighted together. Using different weights on these gradients produces distinct policies, each aiming for different regions of the objective-space. The main disadvantage of this approach is that every new policy is learnt independently, disregarding potentially useful experience encountered by other policies.

In contrast, our approach makes efficient usage of encountered experience as it learns a single network that, when conditioned on a desired return, produces a policy with predictable behavior.

When not mentioned otherwise, all results are averaged over 5 runs.

The results for all experiments are summarized in Table <ref>. Figures <ref>-<ref> show, for each environment, the coverage set found by each algorithm, with dominated solutions filtered out.

We experimentally validate our method on three multi-objective benchmarks:

* Deep-Sea-Treasure <cit.>, a well-known 2-objective grid-world problem (see Section <ref>),

* Minecart <cit.>, a 3-objective problem with a continuous state-space (see Section <ref>),

* Crossroad, a novel 2-objective traffic environment, with high-dimensional pixel-like states (see Section <ref>).

In addition, in Section <ref> we propose an additional novel high-objective environment, Walkroom, where we test our algorithms with up to 9 objectives.

§.§ Deep-Sea-Treasure

Deep-Sea-Treasure (DST) is a well known environment in multi-objective literature <cit.>, where the agent controls a submarine in search for treasure hidden in the depth of the ocean. The agent must balance a trade-off between fuel consumption and treasure value. Navigating consumes fuel, but deeper treasures are worth more than shallow ones.

DST is a fairly small environment, which allows us to compute Pareto front analytically. It is composed of 10 different points, which form a concave front.

Figure <ref> shows the discovered points of the Pareto front by PCN and our two baselines. Only PCN is able to fully recover the Pareto front. RA only discovers the extrema, and is unable to find any of the points in the concave part of the front. While MONES performs slightly better, the number and value of points it discovered was highly variable depending on the run, explaining the high variance seen in Table <ref>.

[legend pos=north east]
[only marks,mark=+] table [x=o0, y=o1, col sep=comma] data/dst/udrl.csv;
[only marks,mark=o,blue] table [x=o0, y=o1, col sep=comma] data/dst/mones.csv;
[only marks,mark=x,red] table [x=o0, y=o1, col sep=comma] data/dst/ra.csv;

Best coverage sets for each algorithm in the Deep Sea Treasure environment. PCN is the only algorithm that recovered the full Pareto front.
A 2D plot showing dots for each solution found by the algorithms in the Deep Sea Treasure environment.

§.§ Minecart

Minecart is a complex environment with a continuous state-space <cit.>. Starting at a base station, the agent controls a cart with the goal to extract ores from mines scattered in the environment, and sell them back at its base.

The agent can execute 6 possible actions: it can accelerate, decelerate and rotate the cart to the left or right. It can mine ores, which will only be effective if it is located in a mine. It can also simply do nothing. There are 2 types of ores located in the different mines, where each type represents a separate objective. The cart has a limited capacity, so it cannot be filled indefinitely. Finally, actions consume fuel, making the Minecart problem a 3-objective problem.

Because the cart has a limited capacity to store ores, the agent must decide the ratio of each ore present in the cart. This is why, as can be seen in Figure <ref>, the vast majority of policies discovered by PCN are laid out in a straight line: the sum of $R_0$ and $R_1$ equals $1.5$ (which is the cart capacity). A few policies only partially fill the cart, saving a bit of fuel in the process. In comparison, the coverage sets discovered by both baselines only contain a small subset of the possible ore ratios, and they systematically consume more fuel than our method. Finally, Table <ref> reports that PCN achieves a low variance in hypervolume across the different runs, which shows that PCN is consistent in finding these diverse and efficient policies.

§.§ Crossroad

We evaluate our proposed method on a novel traffic environment, Crossroad, developed using the SUMO framework <cit.>. In this environment, the agent controls the traffic lights at a busy intersection between two bidirectional roads, an horizontal one with two lanes and a vertical one with a single lane. The agent can choose between 2 actions, either switching the lights or not. We consider 2 objectives: the first is the traffic flow, computed as the number of cars that leave the intersection. The second objective is the car waiting time, i.e., the number of timesteps a car has to wait before exiting the crossroad. Thus, favoring traffic on the horizontal road will favor the first objective, while alternating often will favor the second.

Figure <ref> shows the coverage sets found by PCN and the two baselines. The points discovered by our method dominate the ones found by the baselines across nearly the whole objective-space. There is a single exception is the rightmost extremum, which corresponds to the policy that never switches the light, only allowing cars from the major road to cross the intersection. This policy is pretty simple, as it consists of always performing the same action, which explains why all methods find it.

[legend pos=north east,view=2070,zmin=-70,zmax=0]
[col sep=comma]./data/minecart/mones-nd.csv
[col sep=comma]./data/minecart/udrl.csv
[col sep=comma]./data/minecart/ra-nd.csv

3[only marks,mark=+,mark options=line width=.6] table [x=o0, y=o1, z=o2, col sep=comma] ;
3[only marks,mark=o,blue,mark options=line width=.6] table [x=o0, y=o1, z=o2, col sep=comma] ;
3[only marks,mark=x,red,mark options=line width=.5,scale=1.2] table [x=o0, y=o1, z=o2, col sep=comma] ;

3[only marks,mark=+,dashed,mark options=xscale=1.5, xslant=.37,yslant=-.3, dash
pattern=on 1pt off .5pt, opacity=0.6] table [x=o0, y=o1, z expr=-70.0, col sep=comma] ;
3[only marks,mark=o,blue,dashed,mark options=xscale=1.5, xslant=.37,yslant=-.3, dash
pattern=on 1pt off 1pt, opacity=0.6] table [x=o0, y=o1, z expr=-70.0, col sep=comma] ;
3[only marks,mark=x,red,dashed,mark options=xscale=1.5, xslant=.37,yslant=-.3, dash
pattern=on 1pt off .5pt,opacity=0.6] table [x=o0, y=o1, z expr=-70, col sep=comma] ;

error bars/z dir=minus,
error bars/z explicit,
error bars/error mark=none,
error bars/error bar style=
dashed, dash pattern=on 1pt off 2pt on .3pt off 2pt,
] table [col sep=comma, x=o0, y=o1, z=o2, z error expr=o2+70] ;
error bars/z dir=minus,
error bars/z explicit,
error bars/error mark=none,
error bars/error bar style=
dashed, dash pattern=on 1pt off 2pt, dash phase=0pt,
] table [col sep=comma, x=o0, y=o1, z=o2, z error expr=o2+70] ;
error bars/z dir=minus,
error bars/z explicit,
error bars/error mark=none,
error bars/error bar style=
dashed, dash pattern=on .5pt off 1pt, dash phase=0pt,
] table [col sep=comma, x=o0, y=o1, z=o2, z error expr=o2+70] ;

Best coverage sets for each algorithm in the Minecart environment. The straight shape of PCN's coverage set is due to the cart weight limit. PCN's coverage set fully dominates the ones from the baselines.
A 3D plot showing dots for each solution found by the algorithms in the Minecart environment.

§ SCALING UP THE OBJECTIVE-SPACE

Our experimental section shows that our proposed method significantly outperforms the baselines in several settings, with discrete, continuous and high-dimensional state-spaces. It also empirically shows that the coverage sets discovered by contain more non-dominated solutions than our selected baselines. Nevertheless, the vast majority of benchmarks used in MORL, including the ones used in our experiments, are limited to 2, sometimes 3 different objectives <cit.>.

To get a better understanding of the performance of w.r.t. the number of objectives, we devise a synthetic environment, Walkroom, that can be instanced with an arbitrary number of objectives. Walkroom takes inspiration from Deep Sea Treasure, and is modeled as an $n$-dimensional grid-world in which the agent can move in every cardinal direction. Thus, the dimensions correspond to the number of objectives of the environment. The action space increases linearly with the number of objectives ($|\mathcal{A}| = 2n$). At each timestep, the agent receives a $-1$ reward for the objective corresponding to its moving dimension, and a $0$ reward for all other objectives. There are no other rewards. Similarly to Deep Sea Treasure, Walkroom has a set of goal states positioned along an uneven border, so that reaching each goal state results in a different Pareto-efficient solution. The optimal policies are thus to go directly towards any of the border-positions, at which point the episode ends.

[legend pos=south west]
[only marks,mark=+] table [x=o0, y=o1, col sep=comma] data/sumo/udrl.csv;
[only marks,mark=o,blue] table [x=o0, y=o1, col sep=comma] data/sumo/mones-nd.csv;
[only marks,mark=x,red] table [x=o0, y=o1, col sep=comma] data/sumo/ra-nd.csv;

Best coverage sets for each algorithm in the Crossroad environment. PCN fully dominates the baselines but for the naive policy that never switches the traffic lights (bottom-right point).
A 2D plot showing dots for each solution found by the algorithms in the Crossroad environment.

We evaluate each method in a set of randomly generated Walkroom environments, from 2 to 9 dimensions. We perform 20 runs for every algorithm, on every version of the environment ($n=2,\dots,9$). Note that we do not plot the discovered coverage sets since this is not possible for $n > 3$. Instead, we show boxplots of the hypervolume and $\varepsilon$ metrics computed on all runs.  Since the Pareto fronts can be computed analytically when the environments are generated, the $\varepsilon$ metrics give us an accurate representation of the coverage of each solution set found by each learning algorithm.

Results are summarized in Figure <ref>. RA performs poorly compared to the other algorithms across all metrics. This is because the number of policies that RA requires to cover the objective-space increases exponentially with the number of objectives. Thus, training time must be split between more and more policies, which reduces their individual performance. For the same reason, we were unfortunately unable to compare in RA in environments with more than 5 objectives, due to its exponential computational costs.

For $n \leq 4$, MONES achieves similar or better scores than in the $I_{\varepsilon}$ metric. This is likely due to missing some points from the Pareto front, which impact this metric significantly. However, we can see from the hypervolume and $I_{\varepsilon-mean}$ metrics that generally performs better than MONES on the rest of the Pareto front. Surprisingly, for $n > 4$ MONES performs significantly worse than . This might be because the parameter distributions of MONES are not able to explore efficiently in very large dimensional spaces.

Box plots of normalized hypervolume, $I_{\varepsilon}$ and $I_{\varepsilon-mean}$ in the Walkroom environments, from 2 to 9 objectives. We normalized the hypervolumes as their true values scale exponentially with the number of objectives (for $n=9$, in the order of $10^{9}$). For hypervolume, higher is better. For $\varepsilon$-indicator metrics, lower is better.
Box plot of various metrics obtained in the Walkroom environments.

§ CONCLUSION

We have presented a novel algorithm, Pareto Conditioned Networks, which is able to efficiently and effectively learn coverage sets in multi-objective sequential problems. PCN uses a single neural network to generalize experience across all possible multi-objective returns, learning coverage sets even in concave Pareto fronts.

We evaluated the empirical performance of PCN in several environments against state-of-the-art benchmarks. PCN was able to consistently obtain higher returns than the baselines throughout the whole objective-space, demonstrating its ability to exhaustively discover optimal coverage sets. In addition, PCN demonstrated its ability to learn the Pareto front even when dealing with a large number of objectives.

While PCN can work in continuous state-spaces, its network architecture is currently limited to discrete action-spaces. However, the main ideas behind PCN do not change for continuous action-spaces and PCN can be extended to continuous actions by changing the classification problem to a regression one. We leave this for future work.

The authors would like to acknowledge FWO (Fonds Wetenschappelijk Onderzoek) for their support through the SB grant of Eugenio Bargiacchi (#1SA2820N).
This research was additionally supported by funding from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.

We would also like to thank Diederik M. Roijers for helpful feedback.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Abels et al(2019)]
authorpersonAxel Abels,
personDiederik Marijn Roijers, personTom Lenaerts,
personAnn Nowé, and personDenis Steckelmacher.
Dynamic Weights in Multi-Objective Deep
Reinforcement Learning. In booktitleProceedings of the 36th
International Conference on Machine Learning
(seriesProceedings of Machine Learning Research,
Vol. volume97). publisherPMLR,
addressLong Beach, California, USA, pages11–20.

[Barrett and Narayanan(2008)]
authorpersonLeon Barrett and personSrini
Narayanan. year2008.
Learning all optimal policies with multiple
criteria. In booktitleProceedings of the 25th international
conference on Machine learning. ACM, pages41–47.

[Castelletti et al(2012)]
authorpersonAndrea Castelletti,
personFrancesca Pianosi, and personMarcello
Restelli. year2012.
Tree-based fitted Q-iteration for multi-objective
Markov decision problems. In booktitleThe 2012
International Joint Conference on Neural Networks (IJCNN). IEEE,

[Castelletti et al(2013)]
authorpersonA Castelletti, personFrancesca
Pianosi, and personMarcello Restelli.
A multiobjective reinforcement learning approach to
water resources systems operation: Pareto frontier approximation in a single
journalWater Resources Research
volume49, number6 (year2013),

[Das and Dennis(1997)]
authorpersonIndraneel Das and personJohn E
Dennis. year1997.
A closer look at drawbacks of minimizing weighted
sums of objectives for Pareto set generation in multicriteria optimization
journalStructural optimization
volume14, number1 (year1997),

[Deb et al(2000)]
authorpersonKalyanmoy Deb, personSamir
Agrawal, personAmrit Pratap, and personTanaka
Meyarivan. year2000.
A fast elitist non-dominated sorting genetic
algorithm for multi-objective optimization: NSGA-II. In
booktitleInternational conference on parallel problem
solving from nature. Springer, pages849–858.

[Fu et al(2019)]
authorpersonJustin Fu, personAviral Kumar,
personMatthew Soh, and personSergey Levine.
Diagnosing bottlenecks in deep q-learning
algorithms. In booktitleInternational Conference on Machine
Learning. PMLR, pages2021–2030.

[Hayes et al(2021)]
authorpersonConor F Hayes, personRoxana
Rădulescu, personEugenio Bargiacchi,
personJohan Källström, personMatthew
Macfarlane, personMathieu Reymond, personTimothy
Verstraeten, personLuisa M Zintgraf, personRichard
Dazeley, personFredrik Heintz, et al
A practical guide to multi-objective reinforcement
learning and planning.
journalarXiv preprint arXiv:2103.09568

[Hiraoka et al(2009)]
authorpersonKazuyuki Hiraoka, personManabu
Yoshida, and personTaketoshi Mishima.
Parallel reinforcement learning for weighted
multi-criteria model with adaptive margin.
journalCognitive neurodynamics
volume3, number1 (year2009),

[Jalalimanesh et al(2017)]
authorpersonAmmar Jalalimanesh,
personHamidreza Shahabi Haghighi, personAbbas Ahmadi,
personHossein Hejazian, and personMadjid Soltani.
Multi-objective optimization of radiotherapy:
distributed Q-learning and agent-based simulation.
journalJournal of Experimental & theoretical
artificial intelligence volume29, number5
(year2017), pages1071–1086.

[Kumar et al(2019)]
authorpersonAviral Kumar, personXue Bin
Peng, and personSergey Levine.
Reward-conditioned policies.
journalarXiv preprint arXiv:1912.13465

[Lopez et al(2018)]
authorpersonPablo Alvarez Lopez,
personMichael Behrisch, personLaura Bieker-Walz,
personJakob Erdmann, personYun-Pang
Flötteröd, personRobert Hilbrich,
personLeonhard Lücken, personJohannes Rummel,
personPeter Wagner, and personEvamarie
Wießner. year2018.
Microscopic Traffic Simulation using SUMO, In
booktitleThe 21st IEEE International Conference on Intelligent
Transportation Systems.
journalIEEE Intelligent Transportation Systems
Conference (ITSC).

[Mnih et al(2015)]
authorpersonVolodymyr Mnih, personKoray
Kavukcuoglu, personDavid Silver, personAndrei A
Rusu, personJoel Veness, personMarc G Bellemare,
personAlex Graves, personMartin Riedmiller,
personAndreas K Fidjeland, personGeorg Ostrovski,
et al year2015.
Human-level control through deep reinforcement
journalNature volume518,
number7540 (year2015), pages529.

[Mossalam et al(2016)]
authorpersonHossam Mossalam, personYannis M.
Assael, personDiederik M. Roijers, and personShimon
Whiteson. year2016.
Multi-Objective Deep Reinforcement Learning.
journalCoRR  volumeabs/1610.02707

[Osa et al(2018)]
authorpersonTakayuki Osa, personJoni
Pajarinen, personGerhard Neumann, personJ Andrew
Bagnell, personPieter Abbeel, and personJan
Peters. year2018.
An algorithmic perspective on imitation learning.
journalarXiv preprint arXiv:1811.06711

[Parisi et al(2017)]
authorpersonSimone Parisi, personMatteo
Pirotta, and personJan Peters.
Manifold-based multi-objective policy search with
sample reuse.
journalNeurocomputing  volume263
(year2017), pages3–14.

[Parisi et al(2014)]
authorpersonSimone Parisi, personMatteo
Pirotta, personNicola Smacchia, personLuca Bascetta,
and personMarcello Restelli. year2014.
Policy gradient approaches for multi-objective
sequential decision making. In booktitle2014 International
Joint Conference on Neural Networks (IJCNN). IEEE,

[Perez et al(2018)]
authorpersonEthan Perez, personFlorian
Strub, personHarm De Vries, personVincent Dumoulin,
and personAaron Courville. year2018.
Film: Visual reasoning with a general conditioning
layer. In booktitleProceedings of the AAAI Conference on
Artificial Intelligence, Vol. volume32.

[Roijers et al(2013)]
authorpersonDiederik M Roijers, personPeter
Vamplew, personShimon Whiteson, and personRichard
Dazeley. year2013.
A survey of multi-objective sequential
journalJournal of Artificial Intelligence Research
volume48 (year2013), pages67–113.

[Roijers et al(2015)]
authorpersonDiederik Marijn Roijers,
personShimon Whiteson, and personFrans A Oliehoek.
Computing convex coverage sets for faster
multi-objective coordination.
journalJournal of Artificial Intelligence Research
volume52 (year2015), pages399–443.

authorpersonJuergen Schmidhuber.
Reinforcement Learning Upside Down: Don't Predict
Rewards–Just Map Them to Actions.
journalarXiv preprint arXiv:1912.02875

[Sun et al(2018)]
authorpersonWen Sun, personJ Andrew
Bagnell, and personByron Boots.
Truncated horizon policy search: Combining
reinforcement learning & imitation learning.
journalarXiv preprint arXiv:1805.11240

[Vamplew et al(2011)]
authorpersonPeter Vamplew, personRichard
Dazeley, personAdam Berry, personRustam Issabekov,
and personEvan Dekker. year2011.
Empirical evaluation methods for multiobjective
reinforcement learning algorithms.
journalMachine learning volume84,
number1-2 (year2011), pages51–80.

[Van Moffaert and Nowé(2014)]
authorpersonKristof Van Moffaert and
personAnn Nowé. year2014.
Multi-objective reinforcement learning using sets
of pareto dominating policies.
journalThe Journal of Machine Learning Research
volume15, number1 (year2014),

[Yang et al(2019)]
authorpersonRunzhe Yang, personXingyuan
Sun, and personKarthik Narasimhan.
A Generalized Algorithm for Multi-Objective
Reinforcement Learning and Policy Adaptation.
In booktitleAdvances in Neural Information
Processing Systems 32. publisherCurran Associates, Inc.,

[Zintgraf et al(2015)]
authorpersonLuisa M Zintgraf, personTimon V
Kanters, personDiederik M Roijers, personFrans
Oliehoek, and personPhilipp Beau.
Quality assessment of MORL algorithms: A
utility-based approach. In booktitleBenelearn 2015:
proceedings of the 24th annual machine learning conference of Belgium and the

[Zitzler and Thiele(1999)]
authorpersonEckart Zitzler and
personLothar Thiele. year1999.
Multiobjective evolutionary algorithms: a
comparative case study and the strength Pareto approach.
journalIEEE transactions on Evolutionary
Computation volume3, number4
(year1999), pages257–271.

[Zitzler et al(2003)]
authorpersonEckart Zitzler, personLothar
Thiele, personMarco Laumanns, personCarlos M
Fonseca, and personViviane Grunert Da Fonseca.
Performance assessment of multiobjective
optimizers: An analysis and review.
journalIEEE Transactions on evolutionary
computation volume7, number2
(year2003), pages117–132.

§ EXPERIMENTAL DETAILS

fill opacity=.5,
coordinates (0,9)(1,9)(1,8)(2,8)(2,7)(3,7)(3,6)(6,6)(6,3)(8,3)(8,1)(9,1)(9,0)(0,0);
[] at (axis cs: 0.5,9.5) 1;
[] at (axis cs: 1.5,8.5) 2;
[] at (axis cs: 2.5,7.5) 3;
[] at (axis cs: 3.5,6.5) 5;
[] at (axis cs: 4.5,6.5) 8;
[] at (axis cs: 5.5,6.5) 16;
[] at (axis cs: 6.5,3.5) 24;
[] at (axis cs: 7.5,3.5) 50;
[] at (axis cs: 8.5,1.5) 74;
[] at (axis cs: 9.5,0.5) 124;
[rounded corners,black,fill=black] (axis cs:0.1,10.4) rectangle (axis cs:0.8,10.6) ;
[black,fill=black] (axis cs:0.55,10.5) rectangle (axis cs:0.6,10.7) ;

The Deep Sea Treasure environment. The agent starts on the top-left corner and tries to reach any of the treasures. Farther treasures are worth more.

The Minecart environment. The base is located at the top-left corner, while the 5 mines are spread around the environment.

§.§ Deep Sea Treasure

Deep Sea Treasure is a grid-world environment, where the submarine moves on a $10 \times 11$ grid, resulting in $110$ different states. The state number is one-hot encoded, resulting in a 110-sized vector. At every timestep, the agent can move up, left, right or down.

§.§ Minecart

In Minecart, the agent moves a cart in a continuous 2-dimensional space. The state is represented by a 6-dimensional continuous vector. It encodes the $x$- and $y$-positions of the cart, as well as the cart's speed and orientation. The last 2 dimensions correspond to the amount of each ore type present in the cart, with the total capacity of the cart limited to $1.5$. The agent can execute 6 possible actions: it can accelerate, decelerate and rotate the cart to the left or right. It can mine ores, which will only be effective if it is located in a mine (see Figure <ref> for the layout of the mines used in this environment). Finally, it can also simply do nothing.

§.§ Crossroad

Hyperparameters for the Split, Deep-sea-treasure and Minecart environments.

Split     Deep-Sea-Treasure     Minecart

lr     $0.001$     $0.001$     $0.0003$

$\gamma$     $1.00$     $0.95$     $1.00$

timesteps     $6,000$     $1,000,000$     $20,000,000$

neurons (actor)     $(26,20,2)$     $(132,50,4)$     $(6,20,20,6)$

non-linearity     Tanh     Tanh     Tanh

clip-grad-norm     None     50     50

value-coef     $0.5$     $0.5$     $0.5$

entropy-coef     $0.01$     $0.1$     $0.1$

update every     1     10     200

neurons (critic)     $(26,50,121)$     $(132,50,50,121)$     $(6,20,20,20,121)$

c     11     11     11

$V_{\text{MIN}}$     $(-1, -1)$     $(0, -20)$     $(0, -4)$

$V_{\text{MAX}}$     $(10, 10)$     $(100, 0)$     $(1.5, 0)$

MOAC     2|c|A2C

value-coef     $0.5$     $0.5$     $0.5$

entropy-coef     $0.01$     $0.1$     $0.1$

update every     1     10     200

neurons (critic)     $(26,50,1)$     $(132,50,50,1)$     $(26,20,20,1)$