2-Level Reinforcement Learning for Ships on Inland Waterways

a]Martin WaltzCorrespondingAuthor
a]Niklas Paulig
a,b]Ostap Okhrin

[a]organization=Technische Universität Dresden, Chair of Econometrics and Statistics, esp. in the Transport Sector,
[b]organization=Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), city=Dresden/Leipzig, country=Germany

[CorrespondingAuthor]Corresponding author

This paper proposes a realistic modularized framework for controlling autonomous surface vehicles (ASVs) on inland waterways (IWs) based on deep reinforcement learning (DRL). The framework comprises two levels: a high-level local path planning (LPP) unit and a low-level path following (PF) unit, each consisting of a DRL agent. The LPP agent is responsible for planning a path under consideration of nearby vessels, traffic rules, and the geometry of the waterway. We thereby transfer a recently proposed spatial-temporal recurrent neural network architecture to continuous action spaces. The LPP agent improves operational safety in comparison to a state-of-the-art artificial potential field method by increasing the minimum distance to other vessels by 65% on average. The PF agent performs low-level actuator control while accounting for shallow water influences and the environmental forces winds, waves, and currents. Compared with a proportional-integral-derivative (PID) controller, the PF agent yields only 61% of the mean cross-track error while significantly reducing control effort in terms of the required absolute rudder angle. Lastly, both agents are jointly validated in simulation, employing the lower Elbe in northern Germany as an example case and using real automatic identification system (AIS) trajectories to model the behavior of other ships.

deep reinforcement learning path planning path following autonomous surface vehicle inland waterway

§ INTRODUCTION

Inland waterway (IW) transport is widely regarded as an energy-efficient and low-emitting mode of transportation in terms of greenhouse gases compared to road or rail transportation. Additionally, IW transport offers a significant freight capacity, making it an integral part of a sustainable transport system [92, 17]. Traditionally, the navigation and control of inland vessels is performed by human operators, and recent research began to investigate the use of autonomous surface vehicles (ASVs) for such inland operations [30, 116]. For a comprehensive understanding of ASV systems, we refer to [66], [26], and [78].

One crucial factor affecting the economic potential of shipping companies is the costs associated with crew members and personnel [2]. These costs can be significantly reduced since ASVs require little to no on-board personnel [114]. Furthermore, although IW operations already exhibit relatively low accident rates compared to other transportation modes [41], the human factor can still pose a significant threat to operational safety. For instance, according to [6], human failures were responsible for 58% and 19% of accidents in Austria and Serbia, respectively, from the early 2000s to 2017. The lack of centralized data collection at the European level makes it challenging to conduct a more detailed analysis of accident causes in inland navigation [6]. However, as emphasized by [30], IW operations can be particularly challenging for human operators due to waterway geometry and potential high traffic densities, further reinforcing the potential benefits of employing ASVs in inland operations.

There are different approaches to categorizing the level of autonomy of autonomous ships [90]. In this paper, we adopt the definition from <cit.>, which defines an ASV as "a vessel capable of making decisions and operating independently, without human guidance, navigation, and control". An ASV has to continuously generate a path which it can subsequently follow. The literature distinguishes between global (GPP) and local path planning, where GPP addresses the static problem of defining a plan for the entire voyage ignoring kinematic and dynamic constraints, while LPP is an ongoing process based on real-time information to generate a feasible local path [100, 114]. Path following describes the task of following a pre-determined (local) path without considering the visitation time of a particular waypoint [26]. As stated in the beginning, in this paper, we focus on LPP and PF. We refer to the controlled ASV as the own ship and to its surrounding ships as target ships.

In recent years, the maritime literature has started to embrace the advancements in artificial intelligence [77, 137]. One notable approach is deep reinforcement learning (DRL, 71), which combines reinforcement learning (RL, 104, 105) with deep neural networks [32]. DRL has showcased remarkable success across challenging application domains [117, 43, 7, 101, 63, 48], including maritime control tasks [39, 36]. In RL, an agent acquires knowledge by engaging in iterative interactions with an environment, where it learns to make decisions through trial-and-error. For instance, in the context of LPP, the agent represents the own ship and is required to adapt its heading based on the presence of nearby target ships and the geometric characteristics of the waterway. By defining an appropriate feedback mechanism, known as a reward, the agent receives penalties for causing collisions or running aground.

In contrast to many traditional control methods, DRL does not require prior knowledge or a complete model of the environment. Instead, this essential information is inherently embedded in the experiences the agent accumulates through training. Moreover, due to the use of deep neural networks as powerful function approximators, DRL can generalize across environmental situations, facilitating adaptation to previously unencountered scenarios [51].

There have been recent attempts to perform maritime path planning based on DRL. The first major study on this topic was [15], in which a concise DRL algorithm for static obstacle avoidance was proposed. [39] designs a DRL agent using Proximal Policy Optimization [96] while relying on advanced collision risk assessment theory. [127] apply the Deep Deterministic Policy Gradient (DDPG, 61) with a modified experience replay mechanism to tackle a path planning and collision avoidance (COLAV) task. A similar proposal has been made in [131], where the authors build on the popular Deep $Q$-Network (DQN) of [74] to construct a COLAV algorithm. Finally, [119] outline a spatial-temporal recurrent network architecture for the DQN in maritime LPP. The architecture effectively handles an arbitrary number of target ships and is robust to partial observability. Further contributions for DRL-based approaches to maritime path planning include [16], [60], [73], and [33].

However, these studies primarily focus on path planning for open waters and do not consider the unique challenges associated with IW operations. These challenges include navigating narrow waterways, accounting for water depth effects on marine craft dynamics, and dealing with a potentially large number of target ships [12]. To the best of our knowledge, only [116] have explored the application of DRL to LPP on IWs, albeit limited to static obstacles, without addressing the dynamic COLAV problem. Our study fills this gap by explicitly considering dynamically moving target ships during the path planning phase.

Moreover, recent research started to investigate the suitability of DRL for path following of ASVs. An important contribution in this realm is [125], where the DDPG algorithm was used to successfully control a small-scale vessel to follow a linear path in a real-world experiment. [135] outline a smoothly convergent modification of the DQN algorithm to perform PF on polygonal and sinusoidal paths. Another relevant related work is [70], in which the DDPG algorithm was used for curved PF of three different vessel classes: a mariner, a container, and a tanker. [31] combine the DDPG algorithm with an adaptive sliding mode control strategy [13] to let an ASV follow straight and zig-zag-like paths. Further contributions to DRL-based path following of ASVs include [124], [134], and [87].

However, many existing studies overlook the impact of environmental disturbances such as winds, waters, and waves, which are critical factors in complex maritime operations [24, 26]. A notable exception is [86], in which the angle of attack of the currents acting on the marine craft is explicitly considered in the state space of the RL agent. Using the DQN modification proposed in [118], [86] showed strong path following performances on different paths under varying influence of currents. Their work serves as a motivation for our PF module. Additionally, we consider the impact of waves and winds on the vessel, further increasing the comprehensiveness and robustness of the control system.

In summary, our study makes the following contributions:

* We introduce a two-level architecture for ASV control on IWs based on DRL. Our framework comprises separate agents dedicated to the LPP and PF tasks. The architecture takes into account environmental disturbances such as winds, waves, and currents, while also adhering to existing traffic rules and considering the geometry of the waterway.

* To accommodate continuous action spaces, we transfer the spatial-temporal recurrent network architecture from [119] to actor-critic frameworks, and employ it for the LPP agent.

* To assess the effectiveness of our approach, we conduct extensive testing on both agents in a variety of challenging scenarios that are representative of the most difficult practical occurrences. In particular, we focus on advanced overtaking maneuvers and situations involving strong environmental forces. Furthermore, we validate the entire architecture by using real trajectories obtained from the automatic identification system (AIS).

* We numerically compare the performance of the two agents with strong baselines in the form of an artificial potential field (APF) [62, 122] method for the LPP task and a PID controller [83] for the PF task. The performance metrics used for the comparison build on [47].

* To ensure reproducibility and facilitate further research, we have made the source code for this paper publicly accessible in [120]. In addition, we have open-sourced our trajectory extraction pipeline from AIS data in [85]. By providing these resources, we aim to encourage the inclusion of real AIS data in the validation process of ASV control systems.

This paper is structured as follows: Section <ref> provides additional maritime background information about traffic rules, sensor systems, and classical path following and planning algorithms. The newly proposed architecture is visualized and described in Section <ref>, while Section <ref> details the relevant theory used in this work. Sections <ref> and <ref> contain detailed descriptions of the LPP and PF modules, respectively. The results and validation scenarios are shown in Section <ref>, and Section <ref> concludes the paper.

§ BACKGROUND AND RELATED WORK

§.§ Traffic rules

While path planning is an extensively studied problem in robotics [99], maritime path planning presents unique challenges due to the incorporation of traffic rules specific to each waterway. The International Regulations for Preventing Collisions at Sea (COLREGs, 44) form the basis of these rules, governing the required behavior of ships during encounters at sea. However, one significant limitation of these regulations is their lack of precise metrics. Formulated in the 1970s, when autonomous systems did not exist, the rules rely on verbal descriptions based on principles of good seamanship [103, 39]. For example, COLAV actions should "be large enough to be readily apparent to another vessel" (Rule 8 of 44), and vessels should "take early and substantial action to keep well clear" (Rule 16).

Additionally, national regulations exist that further define the traffic rules for specific waters. As a use case, we focus in this paper on the lower part of the Elbe river in northern Germany, where the relevant regulation is the Seeschifffahrtsstraßen-Ordnung of the [11]. Of particular importance to us are two specific rules within this regulation, which are also representative of inland waterway regulations of other countries. First, overtaking should be conducted on the portside of the vessel being overtaken. Second, the vessel being overtaken should facilitate the overtaking maneuver as much as possible. For the convenience of the reader, the exact wording of the regulation in German together with its translation into English is provided in <ref>.

§.§ ASV sensors

LPP relies on navigational information on the vessel's current state, including position, velocity, accelerations, and environmental forces such as current and wind speed. These quantities stem from various sensors such as radar, LIDAR, sonar, visual sensors, infrared sensors, inertial measurement units, or the global positioning system [66]. However, not every seagoing vessel is necessarily equipped with each of these sensors, and the sensor data might be noisy or erroneous and requires advanced state estimation techniques [58, 75]. Moreover, data about target ships is received via AIS, mandatory equipment since the end of 2004 for all cargo ships of certain sizes and all passenger ships [45]. In addition to the sensor-related challenges, practical aspects such as environmental disturbances, non-cooperative target ships, the interaction between ASVs and human-operated ships, and cybersecurity threats must be considered before deploying an ASV in real-world scenarios [1].

§.§ Path following algorithms

The PF task describes the derivation of low-level control commands from a given local path. First, the controller, sometimes called autopilot, needs to derive a guidance law from the received set of waypoints, which is typically a vector-field guidance (VFG, 79), a line-of-sight guidance [27], a pure pursuit guidance [82], or a constant bearing guidance law [26]. The functionality of the VFG mechanism, which is used throughout the paper, will be detailed in Section <ref>. Moreover, we refer the reader to [9] for additional information on guidance methods.

Once the directional awareness is established through the guidance law, the next step is to convert it into a low-level actuator command that minimizes spatial angular and deviation from the desired path and the guidance signal. These actuator commands are typically a propeller or rudder adjustment to control the vessel's speed and course, respectively. In this paper, we focus on using rudder adjustments as the primary actuator command. Steering changes are preferred over speed adjustments due to fuel considerations and better visual and radar observability of the course changes by other ships [123].

Various control algorithms can be employed to translate the guidance signal into an actuator command. Arguably, one of the most popular techniques is the classic proportional-integral-derivative (PID) controller [83], which will also serve as a benchmark for our PF module. Further methodologies used in the literature include $H_\infty$ control [19], linear quadratic Gaussian control [98], model predictive control [5], sliding motion control [64], backstepping control [132], $\mathcal{L}_1$ adaptive control [10], or dynamic surface control [121], to mention a few. For a comprehensive comparison of path following control methods for ASVs, we refer to the recent survey of [126].

§.§ Path planning algorithms

[109] provide historical perspectives on the development of maritime path planning algorithms, while recent reviews are available in [113], [80], and [130]. The most important methodological approaches in the field include evolutionary algorithms [108], velocity obstacle methods [56], model predictive control [49], APF methods [68], rapidly-exploring random trees [133], dynamic-window approaches [97], fast marching methods [65], and particle swarm optimization [18]. These approaches focus primarily on LPP tasks, although some studies consider GPP or perform overlapping tasks.

Moreover, these works are primarily concerned with ASVs operating on the ocean. In contrast to the extensive literature on classical algorithms for ASVs in open waters, there is a notable scarcity of work done on ASVs operating in IWs. [133] tackle this gap by employing an anisotropic fast marching algorithm for ASVs, with a particular emphasis on navigating through restricted areas near bridges. [14] compare the performance of the heuristic search algorithm A* [38] and its various extensions for GPP in Dutch IWs. Similar to the concepts introduced in [67], [30] develop a planning algorithm for inland rivers that incorporates the safety potential field theory to account for static and dynamic obstacles and waterway bank walls. Finally, [12] detail a modification of the RRT algorithm to perform path planning on the Yangtze River Channel in Zhenjiang.

§ PROPOSED ARCHITECTURE

Figure <ref> visualizes the proposed architecture for ASV control on inland waterways. The vessel controller comprises three key components: a GPP module, an LPP module, and a PF module. Two DRL agents are involved: a high-level agent at the core of the LPP module (in yellow) and a low-level agent for the PF task (in red). The computational flow of the architecture is as follows: The global path generated by the GPP module (in green) serves as input for the LPP module, which in turn produces a local path. The PF module processes this local path and generates an actuator control command. Note that we leave the algorithm or heuristic employed in the GPP module unspecified, although A* or its extensions are reasonable choices [14, 103].

The proposed architecture for an ASV based on DRL, with the visualization being inspired by [14].

Describing the procedure in more detail, the LPP module is activated every $t_{\rm replan}$ seconds to generate a new local plan based on existing information. In our simulation, we set $t_{\rm replan} = \unit[30]{s}$, although this parameter can be freely chosen depending on the geometrical difficulty of the waterway segment or the traffic density. To optimize computation time, the local path is replanned only when target ships are near the own ship. Otherwise, a simple linear local path can be constructed to return to the global path. In this case, it suffices to select a waypoint of the global path and linearly connect it to the own ship's position. Figure <ref> (a) illustrates this procedure, while Figure <ref> (b) visualizes the functionality of the framework in an overtaking scenario. In the latter case, the own ship recognizes the target ship in front and plans a safe local path accordingly.

[Linear local path]
[Overtaking maneuver]
The simplification of the local path planning procedure if no target ships are present (left), and an overtaking maneuver (right).

When the LPP unit is activated, the DRL agent within it processes information from two sources: a cross-track and course error signal, indicating deviation from the global path, and navigational data from the sensor suite. The sensor data includes the positions and velocities of target ships as well as information about the waterway's geometry. Based on this information, the DRL agent generates a heading change command for the own ship, and the positions and velocities of all vessels are updated according to the vessel's model detailed in Section <ref>. The own ship's position is stored as a waypoint of the local path, and the error signals to the global path are updated. This process is repeated until a sufficient number of waypoints has been generated. We emphasize that the LPP procedure always occurs in simulation, even when deployed on a real ship, as path planning does not involve any actuator control.

For simplicity, control commands for the target ships are not incorporated during these planning iterations. Therefore, the LPP agent assumes that all target ships move linearly and maintain a constant course and speed, regardless of their actual behavior. In practice, this simplification can be replaced with a more sophisticated intent estimation or trajectory prediction unit [42, 69], which can be seamlessly integrated into the architecture.

Finally, the PF module receives the local path and calculates the cross-track and course error using the VFG approach. Combined with velocity information and observations of environmental forces, an actuator command is generated to navigate the vessel along the desired path.

§ THEORY

§.§ Vessel dynamics

In this study, we employ the 3-degree-of-freedom Maneuvering Modeling Group (MMG) model introduced by [129] to simulate the dynamic behaviour of a 1:5 scale replica of the KVLCC2 tanker, a vessel commonly used in maritime operations. The principal particulars of the tanker can be found in [86]. Two coordinate systems are considered in our research, which are illustrated in Figure <ref>. The first system, denoted as $\{n\}$, follows the North-East-Down convention. It describes the navigational status of the vessel using the vector $\eta = (x_n, y_n, \psi)^\top$. The variables $x_n$ and $y_n$ represent the north and east positions of the vessel, respectively, relative to the origin point $o_n$. The variable $\psi$ is the heading angle of the ship and represents the angle between the $x_n$-axis and the $x_b$-axis of the second coordinate system, denoted as $\{b\}$. The $\{b\}$ reference frame is a body-fixed frame that is centered at the midship position of the vessel. Within this coordinate system, the longitudinal axis is denoted as $x_b$, and the transversal axis is denoted as $y_b$. The velocity of the vessel is described by the vector $\nu = (u, v, \Tilde{r})^\top$, consisting of the surge velocity $u$, sway velocity $v$, and yaw rate $\Tilde{r}$. Following [26], the total speed of the vessel is $U = \sqrt{u^2 + v^2}$ and the course angle is $\chi = \psi + \arctan(v/u)$.

Coordinate systems considered in this work; see [129].

Unlike the majority of prior studies in the field of ASVs [39, 25, 131], our research incorporates environmental forces, namely winds, waves, and currents, into our simulations. These forces play a critical role in real-life marine operations. Consequently, the vessel's movement is governed by the following set of equations:

\begin{equation}\label{eq:MMG_model}
\begin{aligned}
(m + m_{x_b}) \dot{u} - (m + m_{y_b}) v \Tilde{r} - x_G m \Tilde{r}^2 &= X = X_H + X_R + X_P + X_{WI} + X_{WA}, \\
(m + m_{y_b}) \dot{v} + (m + m_{x_b})u\Tilde{r} + x_G m \dot{\Tilde{r}} &= Y = Y_H + Y_R + Y_{WI} + Y_{WA}, \\
(I_{zG} + x_{G}^{2} m + J_z) \dot{\Tilde{r}} + x_G m (\dot{v} + u\Tilde{r}) &= N_m = N_H + N_R + N_{WI} + N_{WA},
\end{aligned}
\end{equation}

where $X$ is the surge force, $Y$ denotes the lateral force, and $N_m$ signifies the yaw moment around the midship. The first order derivative of a variable $x$ with respect to time is denoted $\dot{x}$. The variables $m$, $m_{x_b}$, and $m_{y_b}$ represent the mass of the ASV, the added masses in the $x_b$ and $y_b$ directions, respectively. Additionally, $x_G$ is the longitudinal coordinate of the center of gravity in $\{b\}$, $I_{zG}$ represents the moment of inertia, and $J_z$ denotes the added moment of inertia. The set of equations (<ref>) encompasses five distinct force components: hull (H), rudder (R), propeller (P), wind (WI), and wave (WA). It is important to note that the propeller component only affects the surge force $X$. The equations for the hull, rudder, and propeller components are derived and explained in detail in [129], which also provides relevant hydrodynamic derivatives and further parameter values. Importantly, the rudder components $X_R$, $Y_R$, and $N_R$ depend on the rudder angle $\delta$, which is used to control the ship.

To compute the wind forces depending on the wind speed $V_{wi}$ and wind angle $\beta_{wi}$, we use the wind coefficient approximation for symmetrical ships described in <cit.>. Regarding wave forces and moments, we follow [107] and [94] to compute the respective quantities depending on the wave amplitude $\zeta_{wa}$, wave angle $\beta_{wa}$, wave period $T_{wa}$, and wave length $\lambda_{wa}$. The most significant environmental force for vessels operating on IWs are currents, represented by the current speed $V_c$ and the current angle $\beta_c$ in the global frame $\{n\}$. Following <cit.>, these are not considered via an additional force or moment term in (<ref>) but by directly specifying the velocity vector of the vessel relative to the currents. Consequently, the total speed of the ship in the presence of currents becomes $\sqrt{(u-u_c)^2 + (v-v_c)^2}$, where $u_c$ and $v_c$ are the longitudinal and lateral components of the currents in $\{b\}$, respectively.

Furthermore, another crucial yet often neglected environmental characteristic for vessels operating on IWs is the influence of the water depth $H$ on the vessel dynamics. To account for this circumstance, we follow [107] and consider the effects of shallow waters on wake fraction, thrust deduction, and flow-straightening coefficient as proposed by [3], while the hydrodynamic derivatives in shallow waters are approximated following [53] and [4].

In our simulation, we discretize the dynamics in (<ref>) using a step size of 5 seconds and use the ballistic method of [111] to update the velocity and the position vector of the vessel. Throughout the paper, we refer to a variable $x$ at time $t$ as $x_t$.

§.§ Vector-field guidance

Both our LPP and PF units use vector-field guidance (VFG) signals to ensure accurate path tracking. The concept of VFG originated from unmanned aerial vehicle control [79] and is visually represented in Figure <ref>. The objective is to establish a vector field that guides the vessel back to its intended path, with the steepness of the field determined by a proportional gain parameter, denoted as $k > 0$. When given two consecutive waypoints, $P_k$ and $P_{k+1}$, the angle from $P_k$ to $P_{k+1}$ in the global frame $\{n\}$ is denoted as $\chi_{P_k}$. The vessel's cross-track and along-track errors are denoted as $y_e$ and $x_e$ respectively. Computation details for these quantities can be found in <cit.>.

Visualization of the induced vector field; inspired by [86].

Based on this information, the VFG method defines the desired course of the vessel as $\chi_d = \chi_{P_k} - \chi^\infty \frac{2}{\pi} \arctan(k \cdot y_e)$, where $\chi^\infty \in \left(0, \frac{\pi}{2}\right]$. In this study, we set $\chi^\infty = \frac{\pi}{2}$, effectively reducing the guidance mechanism to a proportional line-of-sight guidance law [28]. The formulation introduces a course error, $\chi_e = \chi_d - \chi$, which represents the deviation between the desired course and the actual course of the ship, $\chi$. To address sudden changes in the desired course when a waypoint is passed, we adopt the approach proposed by [86] and redefine the desired course to allow for a weighted transition between the current and subsequent path segments, resulting in smoother course adjustments. In particular, we set:

$$\chi_d = \left\{\left[1-\frac{x_e}{d(P_k, P_{k+1})}\right]\chi_{P_k} + \frac{x_e}{d(P_k, P_{k+1})}\chi_{P_{k+1}}\right\}  - \arctan(k \cdot y_e),$$

where $d(P_k, P_{k+1})$ represents the Euclidean distance between waypoints $P_k$ and $P_{k+1}$.

§.§ Collision risk assessment

Estimating the collision risk with nearby target ships is a core task in maritime guidance, navigation, and control systems [81]. In the maritime literature, two concepts hold special importance. The first concept, the ship domain, was originally introduced by [110] and specifies a safe area around a vessel that should not be entered by any other ship. While various shapes of ship domains have been explored in the literature [106], we adopt a symmetric domain configuration that allows for one ship length of space in front of the USV's bow, and one ship width to the USV's starboard, stern, and port side. Furthermore, we define a collision event when the midship position of a target ship is at or inside our own ship's ship domain.

The second concept is the closest point of approach (CPA) [59], which includes the measures distance (DCPA) and time (TCPA) to the closest point of approach. The CPA quantifies the criticality of a scenario by assuming that both the own ship and the target ship maintain their speed and course. Several recent studies have built upon these two concepts and defined specific collision risk metrics to assess the severity of a situation [76, 34, 119].

§.§ Reinforcement learning

Reinforcement learning is a fundamental pillar of artificial intelligence that involves an agent learning through iterative trial-and-error interactions with an environment [104]. Its roots can be traced back to dynamic programming, which is why RL is sometimes referred to as approximate dynamic programming [8]. In this learning paradigm, the agent's interactions with the environment are formalized as a Markov decision process (MDP) [89], represented by a tuple $(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$. Here, $\mathcal{S}$ denotes the state space of the system, $\mathcal{A}$ represents the action space, $\mathcal{P}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0,1]$ is the state transition probability function, $\mathcal{R}: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ is a bounded reward function, and $\gamma \in [0,1)$ is a discount factor that balances the importance of immediate and future rewards. The sum of discounted rewards is called return. The agent's objective within this framework is to optimize for a policy $\pi: \mathcal{S} \times \mathcal{A} \rightarrow [0,1]$, a mapping from states to probability distributions over actions, that maximizes the expected return.

The methods to achieve this task are commonly divided into value-based and policy gradient algorithms [104]. Value-based methods typically learn a policy-dependent action-value for each state-action pair $(s,a)$, denoted $Q^{\pi}(s,a)$, which represents the expected return when being in state $s$, executing action $a$, and following policy $\pi$ afterward. Crucially, under fulfillment of regularity conditions, an optimal policy $\pi^*$ can be derived by behaving greedily to the optimal action-value function $Q^*$, where the latter fulfills $Q^*(s,a) = \max_{\pi}Q^{\pi}(s,a)$ for all $s \in \mathcal{S}$ and $a \in \mathcal{A}$ [89]. Policy gradient algorithms, on the other hand, directly optimize a parametrized policy $\pi^{\theta}$ with respect to a performance measure such as the expected return under policy $\pi^{\theta}$ over the initial state distribution. The set $\theta$ is the parameter set of the policy $\pi^{\theta}$, for example, the biases and weights of a neural network. Lastly, several state-of-the-art algorithms belong to the class of actor-critic algorithms, which are policy gradient algorithms that use action-value estimates during the gradient step. In particular, the policy is referred to as the actor, and the action-value function is the critic [29].

§.§ RL algorithm LSTM-TD3

In practical applications, fully-observed systems, where the agent has access to the complete state vector, are uncommon due to sensor noise, delays, or other disturbances [72]. Recognizing this reality, we extend the MDP formalism to encompass partially-observable Markov decision processes (POMDPs) [50] by introducing an observation space $\mathcal{O}$ and an observation function $\mathcal{Z}: \mathcal{S} \times \mathcal{A} \times \mathcal{O} \rightarrow [0,1]$. Consequently, at each time step $t$, the system resides in a state $s_t \in \mathcal{S}$, the agent receives an observation $o_t \in \mathcal{O}$ generated according to $\mathcal{Z}$, and then selects an action $a_t \in \mathcal{A}$. The system transitions to the next state $s_{t+1} \in \mathcal{S}$ based on $\mathcal{P}$ and provides a reward $r_t$ determined by $\mathcal{R}$. During implementation, the state input $s \in \mathcal{S}$ for an action-value function and policy is replaced by an observation $o \in \mathcal{O}$. For example, in our PF module, the observation vector includes the deviation to the local path and the currently acting environmental forces, while the action is a change in the rudder angle.

In this study, we use the Long-Short-Term-Memory-based Twin Delayed Deep Deterministic Policy Gradient (LSTM-TD3) algorithm proposed by [72] for both the LPP and PF agents. Building upon prior work by [61] and [29], the LSTM-TD3 is an actor-critic algorithm maintaining an actor $\mu$ with parameter set $\theta_{\mu}$ and two critics $Q_1$ and $Q_2$ with parameter sets $\theta_{1}$ and $\theta_{2}$, respectively. Crucially, all networks incorporate LSTM layers [40], which process the information of several times steps and offer a robust solution to address the challenges posed by partial observability. The optimization of the critics uses a DQN-style update [74] with the clipped double $Q$-Learning target proposed in [29], while the actor is optimized based on the deterministic policy gradient theorem of [102].

Throughout the paper, we set the history length to $h = 2$, processing the last two time steps in addition to the current one. While the algorithm is the same, the architecture of the implemented actor and critic networks differ for our LPP and PF agents and will be explained in detail in the upcoming Sections <ref> and <ref>.

§ LOCAL PATH PLANNING MODULE

§.§ Configuration of the RL agent

§.§.§ Observation space

The LPP agent constitutes the core of the LPP module, as illustrated in Figure <ref>. Its responsibility is to generate a reliable local plan while taking into account surrounding target ships, traffic rules, and the waterway geometry. This module does not consider environmental forces such as winds, waves, and currents, as those fall under the responsibility of the PF unit.

The observation for the LPP agent at time $t$, denoted as $o_{t}^{\rm LPP}$, is defined by stacking three vector components: $o_{\rm OS,t}^{\rm LPP}$, which summarizes information about the own ship; $o_{\rm IW,t}^{\rm LPP}$, which describes the navigational area; and $o_{\rm TS,t}^{\rm LPP}$, which delivers information about the surrounding target ships. Specifically, we have:

$$o_{t}^{\rm LPP} = \left(\left(o_{\text{OS},t}^{\rm LPP}\right)^\top, \left(o_{\text{IW},t}^{\rm LPP}\right)^\top, \left(o_{\text{TS},t}^{\rm LPP}\right)^\top \right)^\top.$$

Own ship observation. For the observation about the own ship, $o_{\rm OS,t}^{\rm LPP}$, the following features are included:

$$o_{\text{OS},t}^{\rm LPP} = \left(\frac{U_{\text{OS},t}}{U_{\rm scale}}, \frac{[\psi_{\text{OS},t} - {\chi_{P_k,t}]_{-\pi}^{\pi}}}{\pi}, \frac{[\chi^{\rm global}_{e,t}]_{-\pi}^{\pi}}{\pi}, \frac{y^{\rm global}_{e,t}}{y_{\rm scale}} \right)^{\top},$$

where $U_{\text{OS},t}$ represents the speed of the own ship at time $t$, $\psi_{\text{OS},t}$ is the heading of the own ship, $\chi_{P_k,t}$ is the angle between the two active waypoints of the own ship (as shown in Figure <ref>), $y^{\rm global}_{e,t}$ is the cross-track error on the global path, and $\chi^{\rm global}_{e,t}$ is the course error derived from the VFG method. The superscript $global$ indicates that these features are specific to the LPP agent and are computed with respect to the global path. The VFG gain parameter is set to $k^{\rm LPP} = 0.001$. The function $[\cdot]_{a}^{a+2\pi}: \mathbb{R} \rightarrow [a, a+2\pi)$ is used to transform an angle to a desired domain; see [119]. Additionally, we have scaling parameters $U_{\rm scale} = \unit[3]{m/s}$ and $y_{\rm scale} = \unit[64]{m}$, where 64 meters corresponds to one length between perpendiculars ($L_{pp}$) of the downscaled KVLCC2 tanker.

Waterway observation. To ensure the agent can plan without running aground, it requires information about the navigational area. We construct a vector, denoted as $o_{\text{IW}, t}^{\rm LPP}$, which contains proximities to the navigational boundary in 10 different directions; see Figure <ref>:

\begin{equation}\label{eq:obs_coast}
o_{\text{IW},t}^{\rm LPP} = \left(1-\frac{d^{H}(\gamma_{1})}{d^{H}_{\rm scale}}, \ldots, 1-\frac{d^{H}(\gamma_{10})}{d^{H}_{\rm scale}} \right)^\top.
\end{equation}

The value $\gamma_i$ represents the $i$-th component, where $i = 1, \ldots, 10$, in radians of the vector with degree values $(0, 20, 45, 90, 135, 180, 225, 270, 315, 340)^\top$. We carefully selected these angles to provide the agent with a comprehensive view of its surroundings while avoiding the need for a high-dimensional feature vector that would require specifying angles, for example, for every degree around the clock. The function $d^H: [0, 2\pi) \rightarrow [0, d^H_{\text{scale}}]$ calculates the distance to either the coastline or the global path of the opposing traffic for a given angle.

Visualization of $o_{\text{IW}, t}^{\rm LPP}$, representing the LPP units' awareness of the geometry of the waterway. The reversed global path is the path used to generate opposing traffic.

During implementation, we examine for each angle up to 50 logarithmically scaled distances within a maximum range of $d^H_{\text{scale}} = \unit[1]{NM}$, leading to a normalization of the fractions in (<ref>) to the interval $[0,1]$. We determine if a particular point is below the required water depth for the tanker or if it lies on the opposite side of the opposing traffic's path. Additionally, we follow the approach in [39] by reversing the signs of the fractions to provide the agent with information about the proximity rather than the distance to either the coastline or the opposing path.

Target ship observation. For the observation of the $i$-th target ship at time $t$, denoted as $o_{\text{TS},i,t}$, the following features are considered:

\begin{equation}\label{eq:TS_obs}
\resizebox{1.\hsize}{!}{$
o_{\text{TS},i,t} = \left(
\frac{d^{i}_{\text{OS},t}-D(\alpha^{i}_{\text{OS},t})}{d_{\rm scale}},
\frac{[\alpha^{i}_{\text{OS},t}]_{-\pi}^{\pi}}{\pi},
\frac{[\psi_{i,t}-{\chi_{P_k, t}}]_{-\pi}^{\pi}}{\pi},
\frac{U_{i,t}- U_{\text{OS},t}}{U_{\rm scale}},
\sigma_{i,t},
\frac{t^{\rm cpa}_{i,t}}{t_{\rm norm}},
\frac{d^{\rm cpa,*}_{i,t}}{d_{\rm norm}}
\right)^{\top}.$}
\end{equation}

Here, $d^{i}_{\text{OS},t}$ represents the Euclidean distance between the own ship and the $i$-th target ship, $\alpha^{i}_{\text{OS},t}$ is the relative bearing of ship $i$ from the perspective of the own ship, $\psi_{i,t}$ and $U_{i,t}$ are the heading and speed of the target ship, respectively. The function $D: [0,2\pi) \rightarrow \mathbb{R}$ computes the ship domain around the own ship for a certain angle. Thus, the distance between the target ship's midpoint and the own ship's ship domain is provided to the agent, directly reflecting our definition of a collision event. The binary variable $\sigma_{i,t}$ indicates whether the other vessel is traveling in the same direction as the own ship and is defined as follows:

\[ \sigma_{i,t} =
\begin{cases}
-1 & \text{if} \quad  \vert[\psi_{i,t}-\psi_{\text{OS},t}]_{-\pi}^{\pi}\vert \geq \pi/2, \\
1 & \rm else.
\end{cases}
\]

The collision risk with the other ship is captured by $t_{i,t}^{\rm cpa}$, the TCPA between the own ship and ship $i$, and the variable $d_{i,t}^{\rm cpa, *}$. The latter follows [119] and is defined as $d_{i,t}^{\rm cpa, *} = \max{[0, d_{i,t}^{\rm cpa} - D(\alpha^{i, \rm cpa}_{\text{OS},t})]}$, where $d_{i,t}^{\rm cpa}$ is the regular DCPA and $\alpha^{i, \rm cpa}_{\text{OS},t}$ is the relative bearing of the ship $i$ from the perspective of the own ship at the closest point of approach. Thus, we consider the ship domain in the computation of the DCPA metric similar to as we do it for the present distance to the target ship. The scaling parameters $t_{\rm norm} = \unit[300]{s}$ and $d_{\rm norm} = \unit[0.25]{NM}$ are used.

The complete target ship observation vector is constructed as:
$$o_{\text{TS}, t}^{\rm LPP} = \left(\left(o_{\text{TS},1,t}\right)^\top, \ldots, \left(o_{\text{TS},N_t,t}\right)^\top \right)^\top,$$

where $N_t$ is the number of present target ships at time step $t$. The target ships within this vector are sorted in descending order based on their distance to the own ship. Furthermore, a target ship is only included if its distance to the own ship is less than $d_{\rm scale} = \unit[0.5]{NM}$, which is considered a reasonable range for practical operations on IWs.

§.§.§ Neural network architecture

The varying number of surrounding ships, denoted as $N_t$, introduces a challenge for conventional feed-forward neural networks, which typically rely on a fixed input size. To overcome this challenge, we leverage the spatial-temporal recurrent neural network architecture proposed by [119] and integrate it into actor-critic methods. This adaptation empowers the model to effectively handle continuous action spaces. The schematic representation of our proposed architecture is illustrated in Figure <ref>. In the figure, the concatenation operator is denoted as $\bowtie$, the fully connected (FC) layers have 64 neurons, and the number of hidden units in the LSTM layers is 64.

In particular, we incorporate the spatial-temporal recurrent structure into both the actor and the two critics of the LSTM-TD3 algorithm. The spatial recurrent component loops over surrounding vessels, while the temporal recurrent component loops over time steps. This design choice equips our LPP agent with resilience against partial observability, while allowing it to adapt seamlessly to scenarios involving varying numbers of target ships.

The actor's architecture closely mirrors that presented in [119], except that we apply a hyperbolic tangent activation at the last layer to generate actions within the range $[-1,1]^{\vert \mathcal{A}\vert}$, where $\vert \mathcal{A}\vert$ denotes the cardinality of the action space. Further details on the action space will be elucidated in Section <ref>. Formally, the actor network can be described as follows:

\begin{align*}
z_{\mu, t-l} &= f_{\mu,l}\left(o_{\text{OS},t-l}^{\rm LPP}, o_{\text{IW},t-l}^{\rm LPP}, o_{\text{TS},t-l}^{\rm LPP}; \theta_{f_{\mu,l}}\right) \quad \text{for} \quad l = 0, \ldots, h,\\
\mu\left(o_{(t-h):t}^{\rm LPP}; \theta_{\mu}^{\rm LPP}\right) &= g_{\mu}\left(z_{\mu,t-h}, \ldots, z_{\mu,t-1}, z_{\mu,t}; \theta_{g_{\mu}}\right),
\end{align*}

where the functions $f_{\mu,l}$ with parameter sets $\theta_{f_{\mu,l}}$ for $l=0, \ldots,h$ are the spatial recurrent components, and the function $g_{\mu}$ with parameter set $\theta_{g_{\mu}}$ represents the temporal recurrency. The notation $o_{(t-h):t}^{\rm LPP} = \cup_{l=0}^{h}o_{t-l}^{\rm LPP}$ is used to indicate that the actor network is a function of the past observations, while the complete parameter set of the actor is $\theta_{\mu}^{\rm LPP} = \left(\cup_{l=0}^{h} \theta_{f_{\mu,l}}\right) \cup \theta_{g_{\mu}}$.

Similarly, the critics' architecture follows the principles outlined in [119], with the difference that we concatenate the output of the temporal LSTM with the actions supplied by the actor. Formally, we have for critic $Q_j$ with $j \in \{1,2\}$:

\begin{align*}
z_{j, t-l} &= f_{j,l}\left(o_{\text{OS},t-l}^{\rm LPP}, o_{\text{IW},t-l}^{\rm LPP}, o_{\text{TS},t-l}^{\rm LPP}; \theta_{f_{j,l}}\right) \quad \text{for} \quad l = 0, \ldots, h,\\
Q_j\left(o_{(t-h):t}^{\rm LPP}, a_{t}^{\rm LPP}; \theta_{j}^{\rm LPP}\right) &= g_{j}\left(z_{j,t-h}, \ldots, z_{j,t-1}, z_{j,t}, a_{t}^{\rm LPP}; \theta_{g_{j}}\right),
\end{align*}

with the spatial recurrent functions $f_{j,l}$ for $l = 0, \ldots,h$, parametrized with sets $\theta_{f_{j,l}}$, and the function $g_j$ with parameter set $\theta_{g_j}$, which represents the temporal recurrent component. The critics evaluate the action $a_{t}^{\rm LPP}$ provided by the actor, and the complete parameter set of critic $Q_j$ is denoted $\theta_{j}^{\rm LPP} = \left(\cup_{l=0}^{h} \theta_{f_{j,l}}\right) \cup \theta_{g_j}$.

If there are no target ships present, we artificially create a no-risk ship with $o_{\text{TS},1,t} = (1, -1, 1, -1, 1, -1, 1)^\top$ to fulfill the requirement of having at least one target ship for the recurrence loop in the network architecture.

Neural network architecture for the LPP agent; adapted from [119]. We apply a ReLU activation after each FC layer, except for the last one of the actor and the critic, which use a tanh and linear activation, respectively.

§.§.§ Action space

We define a one-dimensional action space ($\vert \mathcal{A}\vert = 1$) for the LPP unit, which represents a change in the own ship's heading. At time step $t$, the agent generates an action $a_{t}^{\rm LPP} \in [-1,1]$ that is used to update the own ship's heading $\psi_{\text{OS},t}$ according to the following formula:

$$\psi_{\text{OS},t+1} = \psi_{\text{OS},t} + a_{t}^{\rm LPP} \cdot a^{\rm LPP},$$

where $a^{\rm LPP} = 10^\circ$. To ensure realistic and feasible paths, we only apply the agent's action every four time steps. Given our simulation step size of 5 seconds, this means that the planner can adjust the heading by a maximum of ten degrees every 20 seconds.

§.§.§ Reward function

Designing an appropriate reward function is crucial in RL applications as it provides feedback to the agent's actions. By incorporating insights from [119] and [86], we have identified five reward components that facilitate LPP on IWs. These components address global-path following, collision avoidance, traffic rule adherence, and comfort considerations.

The first two components focus on following the global path. We introduce a cross-track error-based reward, denoted as $r^{\rm LPP}_{y_e, t}$, and a course error-based component, denoted as $r^{\rm LPP}_{\chi_e, t}$. They are defined as follows:

\begin{equation*}
r^{\rm LPP}_{y_e, t} = \exp \left(-k^{\rm LPP}_{y_e} \cdot \frac{\vert y^{\rm global}_{e, t} \vert}{y_{e, \text{norm}}} \right), \quad
r^{\rm LPP}_{\chi_e, t} = \exp \left(-k^{\rm LPP}_{\chi_e} \cdot \vert [\chi^{\rm global}_{e, t}]_{-\pi}^{\pi} \vert \right),
\end{equation*}

where the power weights are $k^{\rm LPP}_{y_e} = 2$ and $k^{\rm LPP}_{\chi_e} = 4$, and the normalization length is $y_{e, \text{norm}} = \unit[128]{m}$, which corresponds to $2 L_{pp}$ of the KVLCC2 replica ship. These values are chosen to appropriately weigh the importance of the respective error terms in the reward computation.

The third reward component, denoted as $r^{\rm LPP}_{\text{coll}, t}$, penalizes collision events with other ships and leaving the navigational area. We, therefore, define the binary variables $\sigma_{\text{ground}, t}$ and $\sigma_{\text{lane}, t}$, which take the value one if the ship runs aground or crosses the global path of the opposing traffic, respectively, and zero otherwise. Both situations indicate leaving the navigational area and are therefore considered collision events. Further, we define $\sigma_{\text{coll}, i,t}$ as a binary variable that takes the value one if the own ship has a collision with target ship $i$ at time step $t$, and zero otherwise.

In addition, research on COLAV for ASVs on open waters has shown the benefit of including a distance-based collision reward component [127]. We also include such a signal into $r^{\rm LPP}_{\text{coll}, t}$ since it has the advantage of permanently yielding feedback for the RL agent, also in non-collision events. In particular, we will impose an elliptical collision reward around a target ship, where we encourage the agent to maintain a larger longitudinal distance from the target ship while allowing for smaller lateral distances, as the latter is necessary for overtaking maneuvers. This behavior is specified via the function $f : [0,2\pi) \times \mathbb{R} \rightarrow \mathbb{R}$ as follows:

$$f(\alpha, d) = \exp\left\{ \frac{-\left[d \cdot \sin(\alpha)\right]^2}{e_{\rm norm}^2}\right\} \cdot \exp\left\{ \frac{-\left[d \cdot \cos(\alpha)\right]^2}{n_{\rm norm}^2}\right\},$$

where $e_{\text{norm}} = 3B$ and $n_{\text{norm}} = 1L_{pp}$, with $B$ being the width of the ship. On this basis, we introduce $r^{\rm LPP}_{\text{coll}, t}$ as:

\begin{equation}\label{eq:r_coll}
\resizebox{1.\hsize}{!}{$
r^{\rm LPP}_{\rm coll, t} = k_{\rm coll} \cdot \left( \sigma_{\rm ground, t} + \sigma_{\rm lane, t} + \sum_{i=1}^{N_t} \sigma_{\rm coll, i,t}\right) - \underset{i=1,\ldots,N_t}{\text{max}} f\left[\alpha_{i,t}^{\rm OS}, d^{i}_{\text{OS},t}-D\left(\alpha^{i}_{\text{OS},t}\right)\right]
\end{equation}

where the collision weight $k_{\rm coll}$ is set to $-10$, the variable $\alpha_{i,t}^{\text{OS}}$ represents the relative bearing of the own ship from the perspective of target ship $i$, and $\alpha_{\text{OS},t}^{i}$ is the relative bearing of target ship $i$ from the perspective of the own ship.

The fourth reward component, $r^{\rm LPP}_{\rm rule, t}$, concerns the compliance with traffic rules, which is investigated for each target ship separately. We define the binary variable $\sigma_{\text{rule},i, t}$ that takes value one if at time step $t$ a traffic rule with respect to target ship $i$ is violated. A traffic rule violation consists of three components in this study, which must be fulfilled simultaneously. First, the target ship needs to be close enough to be relevant. Second, it needs to travel in the same direction as the own ship. Third, one of the two behavioral rules we enforce during simulation is violated. As described in Section <ref>, we require that other ships should be overtaken on their port side, corresponding to §23 (1) of [11], and that if the own ship is being overtaken by another vessel, it should facilitate the maneuver by making space for the target ship. The latter requirement represents §23 (2) of the regulation, and we introduce the variable $\sigma_{\text{spd}, t}$ to check for its fulfillment. The variable takes the value one if all target ships traveling in the same direction as the own ship are faster than the own ship, and zero otherwise.

In summary, we define $r^{\rm LPP}_{\rm rule, t} = k_{\rm rule} \cdot \sum_{i=1}^{N_t} \sigma_{\rm rule,i, t}$, where we set the constant to $k_{\rm rule} = -2$, and define the binary violation variable $\sigma_{\rm rule,i, t}$ as follows:

\begin{equation}\label{eq:sigma_rule}
\resizebox{1.\hsize}{!}{$
\sigma_{\rm rule,i, t} =
\begin{cases}
1 & \text{if} \quad
\left\{d_{\rm OS,t}^{i} \leq g(\alpha_{i,t}^{\rm OS})\right\}
\land
\left\{\vert[\psi_{i,t}-\psi_{\text{OS},t}]_{-\pi}^{\pi}\vert < \frac{\pi}{2}\right\}\\
& \land
\left\{\left[\left(U_{\rm OS,t} > U_{i,t}\right)
\land
\left(\frac{\pi}{2} \leq \alpha_{i,t}^{\rm OS} \leq \pi \right)\right]
\lor\left[ \sigma_{\rm spd, t} \land  \left( \frac{3}{2}\pi \leq \alpha_{i,t}^{\rm OS} < 2\pi  \right) \right]\right\}\\
0 & \rm else.
\end{cases}$}
\end{equation}

The function $g: [0,2\pi) \rightarrow \mathbb{R}$ calculates a bearing-dependent distance to determine if the target ship is within a range that requires consideration of traffic rules. Specifically, we select a lateral distance of [0.25]NM and a longitudinal distance of [0.5]NM from the target ship, and $g(\cdot)$ is defined to linearly interpolate between the four resulting corner points.

The fifth component is a comfort reward, $r^{\rm LPP}_{\rm comf,t}$, and should prevent the agent from frequently selecting large heading changes, resulting in a stable and smooth local path. Thus, we define: $r^{\rm LPP}_{\rm comf,t} = -\left(a_{t}^{\rm LPP}\right)^2$. Finally, we aggregate all five reward components into a single scalar via:

$$r^{\rm LPP}_t =
r^{\rm LPP}_{y_e, t} \omega^{\rm LPP}_{y_e} +
r^{\rm LPP}_{\chi_e,t} \omega^{\rm LPP}_{\chi_e} +
r^{\rm LPP}_{\rm coll,t} \omega^{\rm LPP}_{\rm coll} +
r^{\rm LPP}_{\rm rule,t} \omega^{\rm LPP}_{\rm rule} +
r^{\rm LPP}_{\rm comf,t} \omega^{\rm LPP}_{\rm comf},

where we experimentally determined the weights as follows: $\omega^{\rm LPP}_{y_e} = \frac{4}{19} \approx 0.211$, $\omega^{\rm LPP}_{\chi_e} = \frac{1}{19} \approx 0.053$, $\omega_{\rm coll} = \omega^{\rm LPP}_{\rm rule} = \frac{6}{19} \approx 0.316$, and $\omega^{\rm LPP}_{\rm comf} = \frac{2}{19} \approx 0.105$.

§.§ Training environment

§.§.§ Waterway generation

To establish the simulation environment, we first sample a global path by interchanging straight and curved segments following the method described in <cit.>. In the following, we denote a real-valued uniform distribution with support $[a,b]$ as $\mathcal{U}(a,b)$, an integer-valued uniform distribution with support $[a,b]$ as $\mathcal{DU}(a,b)$, and an exponential distribution with expectation $\beta$ as $Exp(\beta)$. The length of each straight river interval is generated according to $\unit[400]{m} + \mathcal{DU}(0, 32) \cdot \unit[50]{m}$, and the radii of the curves are sampled from $\unit[1000]{m} + \mathcal{DU}(0, 4000) \cdot \unit[1]{m}$. Additionally, the curve angles are generated from $60^\circ + \mathcal{DU}(0, 40) \cdot 1^\circ$. We randomly sample left or right curves, respectively. Furthermore, we construct another path by imposing a fixed offset of 200 meters to the global path. The second path is referred to as reversed global path and is responsible for generating the opposing traffic.

Once we have established the global path, we proceed to create an IW by sampling water depths around it, following [86]. Initially, we sample a maximum water depth in meters from $\text{clip}\left[Exp(35), 20, 100\right]$ with the clipping operation: $\text{clip}(x, l, u) = \min\left[u, \max(x,l)\right]$. The generated waterway has a maximum width of 500 meters. To introduce some variability to the depth data, we incorporate noise by adding realizations from $\mathcal{U}(-2, 2)$ in meters. We resample a generated waterway at the beginning of every fifth episode to keep the computational effort low.

§.§.§ Target ship generation

To initialize the target ships, we incorporate insights from the recent work conducted by [35]. Our aim is to create training scenarios that pose significant challenges to the agent, increasing the risk of collisions with other ships. In each episode, we randomly sample $N \sim \mathcal{DU}(0,10)$ target ships, encompassing a range of speeds, including both slower and faster vessels than the own ship, as well as ships traveling in opposing directions. The base vessel speed in our simulation is $U_{\rm base} = \unit[3]{m/s}$, and at each episode, we randomly sample the own ship's speed from $\mathcal{U}(0.8, 1.2) \cdot U_{\rm base}$.

To initiate target ship $i$, where $i = 1, \ldots, N$ if $N > 0$, there is a 15% probability of sampling a faster ship with a speed of $U_i \sim \mathcal{U}(1.3, 1.5) \cdot U_{\rm base}$. In this case, the vessel $i$ is initiated behind the own ship and travels in the same direction. The initial distance to the own ship along the global path, denoted $d_i$, is generated via $d_i \sim \mathcal{U}(0.3, 0.7) \cdot \unit[1]{NM}$, ensuring the vessel will create a threat for the own ship in the future. In the remaining 85% of cases, vessel $i$ is slower with $U_i \sim \mathcal{U}(0.4, 0.8) \cdot U_{\rm base}$ and is initiated in front of the own ship. In this case, the vessel $i$ is randomly assigned to travel in the same or opposing direction. When traveling in the opposing direction, we sample $d_i \sim \mathcal{U}(1.1, 1.9) \cdot \unit[1]{NM}$ and place vessel $i$ on the reversed global path. Otherwise, we have $d_i \sim \mathcal{U}(0.3, 0.7) \cdot \unit[1]{NM}$.

To introduce further variability and prevent the target ships from consistently spawning exactly on their respective paths, we introduce small positional noise to the target vessel's initial positions. An episode ends if a maximum of 150 steps has been reached or if the agent is more than $\unit[0.5]{NM}$ away from the global path. A screenshot of the full environment including waterway and target ships is shown in Figure <ref>. The own ship is depicted in red and travels in the south-east direction in this case, while the target ships are grey if they travel in the same direction as the own ship, and golden otherwise.

Screenshot of the simulation environment for the LPP agent (red) with four target ships (grey in the agent's direction, golden otherwise). The latitude and longitude values are artificial and serve as orientation.

§.§.§ Target ship behavior modeling

One of the key considerations in simulating AVSs is the control mechanism for the target ships. For a comprehensive overview of this topic, we refer to [136]. In our study, we focus on the non-communicative scenario, where there is no explicit exchange of intentions or planned trajectories among vessels. In many existing works on ASVs, the assumption of linearly moving target ships with no course and speed changes is commonly adopted, particularly in open water scenarios [33, 25, 127, 128, 95]. However, while trained policies based on this assumption can generalize to non-linear target ship motions, as shown by [119], the linearity and non-reactiveness assumptions are unrealistic in practical scenarios, particularly in the context of IWs.

To address this limitation, we employ a simple rule-based controller for the target ships during the training of the LPP agent. The basis of this controller is the VFG method, as outlined in Section <ref>. Additionally, the target ships are capable of performing basic overtaking maneuvers, in compliance with §23(1) of the collision regulations outlined in [11]. In head-on scenarios, they avoid collisions by executing a turn to starboard. For a detailed description of the complete controller, please refer to Algorithm <ref> in <ref>. Importantly, considering the recent work of [1], we incorporate a 20% probability of generating a non-cooperative target vessel which is solely controlled based on VFG and does not perform any COLAV maneuvers.

Finally, we recognize that the environmental design and the definition of the state, action, and reward spaces for the LPP agent depend on various constants and hyperparameters. However, as mentioned earlier, we introduce noise in all fundamental aspects of the environmental generation. This includes the initial generation of the waterway, the number of target ships, and their behavior. By incorporating this approach, we can effectively optimize for a robust policy for the LPP task. The validation scenarios in Section <ref> will illustrate the effectiveness of our approach.

§ PATH FOLLOWING MODULE

§.§ Configuration of the RL agent

§.§.§ Observation space

Following the overall architecture in Figure <ref>, the PF unit takes as input the local path generated from the LPP module and controls the rudder angle under consideration of the environmental disturbances. The PF module has no information about the IW geometry or the target ships and is solely responsible for tracking the given local path.

The observation of the PF agent at time $t$ is denoted as $o_{t}^{\rm PF}$ and consists of a component describing the status of the own ship, denoted $o_{\text{OS},t}^{\rm PF}$, and a component regarding the environmental forces, denoted $o_{\text{Env},t}^{\rm PF}$. Thus, we have $o_{t}^{\rm PF} = \left(\left(o_{\text{OS}, t}^{\rm PF}\right)^\top, \left(o_{\text{Env}, t}^{\rm PF}\right)^\top\right)^\top$.

Own ship observation. The component describing the status of the own ship is defined as:

$$o_{\text{OS}, t}^{\rm PF} = \left(
\frac{u_{\text{OS},t}}{u_{\rm scale}},
\frac{v_{\text{OS},t}}{v_{\rm scale}},
\frac{\Tilde{r}_{\text{OS},t}}{\Tilde{r}_{\rm scale}},
\frac{\Dot{\Tilde{r}}_{\text{OS},t}}{\Dot{\Tilde{r}}_{\rm scale}},
\frac{\delta_{\text{OS},t}}{\delta_{\rm max}},
\frac{y_{e,t}^{\rm local}}{y_{\rm scale}},
\frac{[\chi_{e,t}^{\rm local}]_{-\pi}^{\pi}}{\pi}
\right)^\top.$$

Here, $u_{\text{OS},t}$, $v_{\text{OS},t}$, $\Tilde{r}_{\text{OS},t}$, $\Dot{\Tilde{r}}_{\text{OS},t}$, $\delta_{\text{OS},t}$ are the surge velocity, sway velocity, yaw rate, change in yaw rate, and rudder angle of the own ship, respectively. The scaling constants are: $u_{\rm scale} = \unit[3]{m/s}$, $v_{\rm scale} = \unit[0.2]{m/s}$, $\Tilde{r}_{\rm scale} = \unit[0.002]{rad/s}$, $\Dot{\Tilde{r}}_{\rm scale} = 8 \cdot \unit[10^{-5}]{rad/s^2}$, and $\delta_{\rm max} = 20^\circ$. The variables $y_{e,t}^{\rm local}$ and $\chi_{e,t}^{\rm local}$ are the cross-track error and the course error from the VFG method for the local path, which uses a gain parameter of $k^{\rm PF} = 0.01$.

Environmental force observation. Further, we define the environmental observation component as follows:

\begin{equation*}
\resizebox{1.\hsize}{!}{$
\begin{split}
o_{\text{Env}, t}^{\rm PF} = \biggl(
&\frac{V_{c,t}}{V_{\rm c, norm}},
\frac{[\beta_{c,t} -\psi_{\text{OS},t}]_{-\pi}^{\pi}}{\pi},
\frac{V_{wi,t}}{V_{wi, \text{norm}}},
\frac{[\beta_{wi,t} -\psi_{\text{OS},t}]_{-\pi}^{\pi}}{\pi},
\frac{[\beta_{wa,t} -\psi_{\text{OS},t}]_{-\pi}^{\pi}}{\pi},\\
&\frac{\zeta_{wa, t}}{\zeta_{wa, \text{norm}}},
\frac{T_{wa, t}}{T_{wa, \text{norm}}},
\frac{\lambda_{wa, t}}{\lambda_{wa, \text{norm}}},
\frac{H_t}{H_{\rm norm}}\biggl)^\top,
\end{split}$}
\end{equation*}

where $V_{c,t}$, $\beta_{c,t}$, $V_{wi,t}$, $\beta_{wi,t}$ are the current and wind's speed and angle of attack, respectively, at the position of the own ship at time $t$. Moreover, $\beta_{wa,t}$, $\zeta_{wa, t}$, $T_{wa, t}$, $\lambda_{wa, t}$, are the wave's angle, height, period, and length. Finally, we included the water depth $H_t$ at the own ship's position into the observation vector since it also affects the dynamics via the shallow water corrections outline in Section <ref>. The scaling parameters are set to: $V_{c, \text{norm}} = \unit[0.5]{m/s}$, $V_{wi, \text{norm}} = \unit[15]{m/s}$, $\zeta_{wa, \text{norm}} = \unit[2]{m}$, $T_{wa, \text{norm}} = \unit[7]{s^{-1}}$, and $\lambda_{wa, \text{norm}} = H_{\rm norm} = \unit[100]{m}$.

§.§.§ Neural network architecture

We use the network architecture outlined in [37] for both actor $\mu$ and the critics $Q_j$, $j \in \{1,2\}$ of the PF unit, which is similar to the original proposal of [72]. The architecture is visualized in Figure <ref>, where the FC layers use 128 neurons and the number of hidden units of the LSTM layers is 128. In particular, the actor $\mu$, parametrized by $\theta_{\mu}^{\rm PF}$, processes the observations $o_{(t-h):t}^{\rm PF} = \cup_{l=0}^{h}o_{t-l}^{\rm PF}$ to yield an action $\mu\left(o_{(t-h):t}^{\rm PF}; \theta_{\mu}^{\rm PF}\right)$. The critic $Q_j$, $j \in \{1,2\}$, which is parametrized by the set $\theta_{j}^{\rm PF}$, evaluates a given action $a_{t}^{\rm PF}$ to produce an action-value estimate $Q_j\left(o_{(t-h):t}^{\rm PF}, a_{t}^{\rm PF}; \theta_{j}^{\rm PF}\right)$. Following [72], the networks process the past observations $o_{t-h}^{\rm PF},\ldots,o_{t-1}^{\rm PF}$ through an LSTM layer, which constitutes a memory extraction unit. The current observation $o_{t}^{\rm PF}$ is separately processed by fully-connected layers, which [72] refer to as the current feature extraction unit.

Neural network architecture used for the PF agent; adapted from [37]. We apply a ReLU activation after each FC layer, except for the last one of the actor and the critic, which use a tanh and linear activation, respectively.

§.§.§ Action space

The PF agent controls the rudder angle of the own ship. Building on the observation defined in the last subsection, the agent computes an action $a_{t}^{\rm PF} \in [-1,1]$ which adjusts the rudder angle as follows:

$$\delta_{\text{OS},t+1} = \text{clip}\left(\delta_{\text{OS},t} + a^{\rm PF}_{t} \cdot a^{\rm PF},  -\delta_{\rm max}, \delta_{\rm max}\right),$$

where the clipping operation ensures the absolute value of the rudder angle does not exceed $\delta_{\rm max} = 20^\circ$. Further, we set $a^{\rm PF} = 5^\circ$, which results in combination with our simulation step size of 5 seconds in realistic rudder changes. To stress again: the PF agent adjusts the rudder angle of the own ship, while the LPP agent directly modifies the heading angle.

§.§.§ Reward function

The reward function for the PF unit builds on [86] and consists of three components: a cross-track error reward, $r^{\rm PF}_{y_e,t}$, a course error component, $r^{\rm PF}_{\chi_e,t}$, and a comfort reward, $r^{\rm PF}_{\text{comf},t}$. Similar to the LPP agent's reward function of Section <ref>, we define:

\begin{align*}
r^{\rm PF}_{y_e,t} &= \exp \left(-k_{y_e}^{\rm PF} \vert y_{e,t}^{\rm local}\vert \right), \quad r_{\text{comf}, t}^{\rm PF} = -\left(a^{\rm PF}_{t}\right)^2,\\
r^{\rm PF}_{\chi_e,t} &= \begin{cases}
k_{\rm turn} & \text{if}\;\; \vert \chi_{e,t}^{\rm local}\vert \geq \frac{\pi}{2}\\
\exp \left(-k_{\chi_e}^{\rm PF} \vert \chi_{e,t}^{\rm local}\vert \right) & \text{else.}
\end{cases}
\end{align*}

We set the constants $k_{y_e}^{\rm PF} = 0.05$, $k_{\chi_e}^{\rm PF} = 5$, and $k_{\rm turn} = -10$. The condition with the large negative penalty for the course error is included to prevent the agent from completely turning around and following the path in the wrong direction. The reward components are aggregated to the reward for the PF agent at time $t$, denoted $r^{\rm PF}_{t}$, as follows:

$$r^{\rm PF}_{t} =
r^{\rm PF}_{y_e, t} \omega^{\rm PF}_{y_e} +
r^{\rm PF}_{\chi_e,t} \omega^{\rm PF}_{\chi_e} +
r^{\rm PF}_{\rm comf,t} \omega^{\rm PF}_{\rm comf},$$

where a small grid search yielded the equal weights: $\omega^{\rm PF}_{y_e} = \omega^{\rm PF}_{\chi_e} = \omega^{\rm PF}_{\rm comf} = \frac{1}{3}$.

§.§ Training environment

The IW generation for the PF unit is identical to the one described in Section <ref>. However, in this module, we need to sample the current, wind, and wave conditions since it is the PF unit's primary responsibility to account for these. The respective angles of attack in radians are separately sampled from $\mathcal{U}(0, 2\pi)$. Moreover, we sample a current speed from $\text{clip}\left[Exp(0.2), 0, 0.5\right] \cdot \unit[1]{m/s}$ and a wind speed from $\mathcal{U}(0, 15) \cdot \unit[1]{m/s}$. The wave height, length, and period are sampled from $\text{clip}\left[Exp(0.1), 0.01, 2\right] \cdot \unit[1]{m}$, $\text{clip}\left[Exp(20), 1, 100\right] \cdot \unit[1]{m}$, and $\text{clip}\left[Exp(1), 0.5, 7\right] \cdot \unit[1]{s^{-1}}$, respectively. Furthermore, we introduce zero-mean Gaussian noise to each value when queried by the own ship, enhancing the robustness of the policy.

The termination conditions for an episode are as follows: if more than 500 steps have elapsed, if the agent strays more than 400 meters away from the local path, or if the water depth at the own ship's position becomes insufficient, rendering the shallow water approximations infeasible.

§ RESULTS AND VALIDATION

§.§ Training details

The LPP and PF agents are separately trained for $2 \cdot 10^6$ and $3 \cdot 10^6$ steps, respectively. The remaining hyperparameters are the same for both agents and are outlined in Table <ref>. All experiments were conducted using Python 3.8.6 [115] and the PyTorch deep learning library [84] version 1.10.0. The computational hardware employed for these experiments consisted of Intel(R) Xeon(R) CPUs E5-2680 v3 (12 cores) running at 2.50 GHz. We make the source code to this study publicly available in a GitHub repository in [120] to ensure full reproducability.

Hyperparameter     Value

Batch size     32

Discount factor     0.99

History length ($h$)     2

Learning rate (actor)     0.0001

Learning rate (critic)     0.0001

Loss function     Mean squared error

Max. replay buffer size     $5 \cdot 10^5$

Min. replay buffer size     5,000

Optimiser     Adam [55]

Policy update delay     2

Soft update rate     0.001

Target policy smoothing noise     0.2

Target policy smoothing noise clip     0.5

List of hyperparameters. For a detailed description of each parameter we refer to [29] and [72].

During the training process, we average every 5000 steps the test return, which is the sum of rewards, of 3 evaluation episodes. To enhance clarity, we apply exponential smoothing to these values. Additionally, we perform the experiment ten times under different random initialization of the neural networks. This enables us to compute confidence intervals that better represent the training performance, which are shown in Figure <ref>. The blue line in each of the plots is the respective mean over ten independent runs, and the light blue area around it represents the 95% pointwise confidence interval. The orange line is the run whose final policy is validated in the upcoming section. The LPP unit's training plot has a larger variance due to its dependence on five distinct reward components. In contrast, the PF unit's training is less variable since path following is a more straightforward control task, minimizing spatial and angular deviation from the local path.

[Local path planning unit]
[Path following unit]
The two plots show the test return's development during training for the two agents.

In the upcoming sections, we separately validate the proposed LPP and PF modules on simulation examples. Afterward, we employ the complete architecture described in Section <ref> in validation scenarios on real AIS data.

§.§ Validation: Local path planning module

§.§.§ Setup

We have developed a comprehensive testing procedure to thoroughly evaluate the performance of the LPP unit. The procedure consists of six distinct setups, which are among the most challenging and safety-critical scenarios encountered on inland waterways:

* overtaking a vessel train,

* overtaking an overtaker,

* overtaking under oncoming traffic,

* overtaking an overtaker under oncoming traffic,

* getting overtaken,

* and navigating along static obstacles.

The inclusion of static obstacles along the ship's path is particularly important to assess the LPP agent's generalization capabilities, as the agent has not been exposed to such obstacles during training. We analyze the behavior of the DRL agent separately in each of these six setups on a straight waterway segment, a left curve, and a right curve, respectively. This results in a total of 18 different scenarios being studied.

Moreover, we compare the performance of the DRL planning agent for the straight waterway segment with a state-of-the-art APF method, building on the recent proposals of [62] and [122]. The detailed functionality of the APF approach is described in <ref>. We choose three performance metrics to compare the two methods thoroughly. Following [47], the first metric is the controller effort (CE) over a trajectory, which measures the average commanded heading change of the respective planning method. Formally, we define:

$$\text{CE}_{\rm LPP} = \frac{1}{\Delta_{\psi} \cdot T}\sum_{t=1}^{T} \vert \psi_{\text{OS},t} - \psi_{\text{OS},t-1}\vert,$$

where $T$ is the length of the trajectory. Note that the CE is normalized by $\Delta_\psi$, which is the maximum possible heading change between consecutive time steps. As described in Section <ref>, this value is $10^\circ$ for the DRL agent, while we reduced it to $2^\circ$ for the APF method, which heavily increased the latter method's performance. Generally, the CE should be as small as possible to generate smooth local paths that allow for fuel efficient operations.

The second performance metric for the LPP task is the mean cross-track error (MCTE), which is defined by [47] as follows:

$$\text{MCTE}_{\rm LPP} = \frac{1}{L_{pp} \cdot T}\sum_{t=0}^{T} \vert y_{e,t}^{\rm global}\vert,$$

which includes a normalization by the length between perpendiculars $L_{pp} = \unit[64]{m}$ of the vessel. A smaller MCTE indicates a planned path close to the global path, reflecting strong tracking capabilities.

Finally, incorporating safety considerations into our evaluation, we introduce a third performance metric named MinDist. This metric is the minimum distance encountered with respect to any target ship along the trajectory. Formally, we have:

$$\text{MinDist} = \frac{1}{L_{pp}} \cdot \min_{t \in \{0,\ldots, T\}} \min_{i \in \{0, \ldots, N_t\}} \left[d_{\text{OS},t}^{i}-D\left(\alpha_{\text{OS},t}^{i}\right) \right],$$

where we consider the ship domain of the own ship similar to the target ship observation in (<ref>). Generally, a sufficiently large MinDist indicates a safe maritime operation.

§.§.§ Analysis

The trajectories for the straight case of the DRL agent and the APF method are visualized in Figures <ref> and <ref>, respectively. The black trajectories in these figures correspond to the respective planning agent, while the colorized ones are the target ships controlled by Algorithm <ref>. The purple and grey dotted lines are the global and reversed global paths. Further, Table <ref> contains the corresponding performance metrics. Additionally, Figure <ref> provides plots of the cross-track and course errors to the global path, the selected action by the DRL agent, and the distance to the target ships. The results for the curved scenarios alongside the initial speed configurations of each vessel can be found in <ref>.

Trajectories of the LPP validation scenarios of the DRL agent on a straight river segment. Note that the latitude and longitude values are artificial and serve as orientation.

Trajectories of the LPP validation scenarios of the APF method on a straight river segment. Note that the latitude and longitude values are artificial and serve as orientation.

Global cross-track and course error, selected actions, and distances to the target ships during validation of the LPP agent on a straight waterway segment.

2*Scenario     2c$\text{CE}_{\rm LPP}$     2c$\text{MCTE}_{\rm LPP}$     2cMinDist
DRL     APF     DRL     APF     DRL     APF

1     0.067     0.565      0.954     0.576     1.197     0.434

2     0.040     0.283     0.523     0.388     1.136     0.639

3     0.052     0.343     0.381     0.349     0.558     0.423

4     0.054     0.374     0.502     0.456      0.618     0.655

5     0.064     0.195     0.390     0.172     1.810     0.928

6     0.064     0.710     0.801     0.855     0.727     0.589

Average     0.057     0.412     0.592     0.466     1.008     0.611

Performance comparison of the DRL agent with the APF method for the LPP task. All metrics are dimensionless.

Focusing on Scenario 1 in Figures <ref> and <ref>, we can observe the successful overtaking maneuver performed by the LPP agent on a vessel train consisting of four target ships. Initially, the agent steers to the port side to avoid a collision with the nearest target ship and then overtakes each vessel one by one until the entire vessel train has been passed. Importantly, this behavior complies with §23(1) of the regulations specified in [11], as the agent performs overtaking maneuvers on the target vessel's port side. Once the overtaking maneuver is completed safely, the agent returns to the global path until the cross-track and course errors are close to zero again. Moreover, the action selection during the maneuver is relatively moderate since the agent avoids large consecutive heading changes.

Similar successful results can be observed in the overtaking cases of Scenarios 2 to 4, demonstrating the LPP agent's proficiency in executing advanced maneuvers while maintaining appropriate safe distances from the target ship. Furthermore, the agent's action selection demonstrates a balanced approach in these scenarios, indicating the successful integration of the comfort reward component. In Scenario 5, we can observe the agent's compliance with §23(2), as it enables overtaking of the target ship by slightly moving towards starboard, which directly reflects the traffic rule reward component defined in (<ref>). Of particular note is Scenario 6, where the agent effectively navigates around static obstacles while maintaining a minimum distance of approximately 50 meters.

Comparing the DRL agent to the APF method, we observe that the APF trajectories are more unstable and clearly display the undesired change between stronger and weaker repulsive forces when passing target ships; see, for instance, Scenario 6 of Figure <ref>. Remarkably, as shown in Table <ref>, this unstable behavior leads to a slightly smaller average MCTE of the APF method in comparison to the DRL agent. However, this circumstance comes at the expense of the increased criticality of the APF trajectories. Generally, although the APF approach does not produce collisions, the MinDist of the DRL controller is on average 65% larger. Further following Table <ref>, the CE is massively reduced by the DRL approach, which is in line with the visually unstable behavior of the APF method.

Lastly, we emphasize that the hyperparameters of the APF method have been carefully tuned to increase its overall performance during the six scenarios on the straight waterway segment. However, deploying this APF configuration without further tuning on the curved waterway segments is not feasible, so we limit comparing the DRL and APF methods to the straight waterway case. On the contrary, our DRL agent can seemingly handle different waterway curvatures without further adaptations, as shown in <ref>.

§.§ Validation: Path following module

§.§.§ Setup

We evaluate the performance of our PF agent by subjecting it to various environmental conditions, including the three major forces: currents, winds, and waves. Each force is tested separately in both moderate and extreme scenarios, resulting in a total of six validation scenarios. In each scenario, the agent is tasked with following a straight path initially unaffected by environmental forces. Subsequently, a force field perpendicular to the path is introduced, followed by another segment with zero forces. Finally, the force direction is reversed in the last segment. This testing procedure enables us to assess the agent's adaptability to different environmental conditions.

For comparison purposes, we follow [83] and [86] and include a PID controller for the rudder angle, which was optimized using the particle swarm optimization (PSO) approach outlined in [20]; see <ref> for details. It should be emphasized that the PID controller is specifically optimized for the validation scenarios, while the RL agent's training encompasses a broader range of scenarios.

Furthermore, we follow [47] and consider the CE and the MCTE to quantitatively compare the performance of the two methods. However, contrary to Section <ref>, the CE for PF measures the average absolute rudder angle, while the MCTE in this task is computed with respect to the local instead of the global path. Formally, we set:

\begin{equation*}
\text{CE}_{\rm PF} = \frac{1}{\delta_{\rm max} \cdot T}\sum_{t=0}^{T} \vert \delta_{\text{OS},t}\vert, \quad \quad
\text{MCTE}_{\rm PF} = \frac{1}{B \cdot T}\sum_{t=0}^{T} \vert y_{e,t}^{\rm local}\vert,
\end{equation*}

where $B = \unit[11.6]{m}$ is the width of the downscaled KVLCC2 tanker.

§.§.§ Analysis

The results are presented in Figures <ref> and <ref>, where each column refers to a testing scenario for a separate environmental force, while the other two forces are set to zero. From top to bottom, the rows depict the local cross-track error, the local course error, the surge and sway velocities, the yaw rate, and the rudder angle, respectively. Afterward, the rows include the specific attributes for each force; for example, the speed and angle of attack of the currents. In addition, Table <ref> displays the CE and MCTE performance metrics.

Validation results for path following under moderate environmental conditions.

Validation results for path following under extreme environmental conditions.

2*Scenario     2c$\text{CE}_{\rm PF}$     2c$\text{MCTE}_{\rm PF}$
DRL     PID     DRL     PID

1     0.032     0.137     0.056     0.202

2     0.021     0.058     0.036     0.087

3     0.041     0.078     0.067     0.105

4     0.114     0.386     0.498     0.615

5     0.165     0.311     0.460     0.488

6     0.298     0.419     0.165     0.613

Average     0.112     0.232     0.214     0.352

Performance comparison of the DRL agent with the PID controller for the PF task. All metrics are dimensionless.

Generally, the DRL agent demonstrates remarkable performance across all six scenarios, effectively adapting to force fields by promptly adjusting the rudder angle to steer back to the desired path. As a result, it successfully maintains minimal cross-track and course error. Notably, the inclusion of a comfort reward component has proven to be beneficial, as the agent generally exhibits smooth and moderate changes in the rudder angle. In contrast, the PID controller exhibits the typical undesired oscillation behavior, which persists even after applying the PSO technique. The challenging aspect lies in the heterogeneity of the scenarios, where the transition dynamics of the environment vary significantly under different force fields. This variability likely poses a difficulty for the PID controller in achieving stable and consistent performance. Following Table <ref>, the DRL agent strongly outperforms the PID controller both in terms of CE and MCTE. In particular, the CE of the DRL is, on average, approximately 48% of the CE of the APF method, while the MCTE of the DRL method is simultaneously only circa 61% of the MCTE of the APF approach.

§.§ Validation: Complete architecture

After validating the LPP and PF units in separation, in this subsection, we aim to conduct a comprehensive evaluation of the complete architecture introduced in Section <ref>. We deploy it in simulation on the lower part of the river Elbe in northern Germany. To ensure a realistic simulation, we specifically choose the date of January 29, 2022, and utilize the actual AIS vessel trajectories and environmental disturbances observed on that day. It is worth noting that the presence of the storm named Malik in Middle Europe on that date adds an extra challenge to ASV. We manually specify a global path from Lighthouse Tinsdal to the Elbe estuary close to Cuxhaven, which is illustrated in Figure <ref>. Further information regarding the data sources and the trajectory interpolation of the target ships are deferred to <ref>.

To test the architecture's generalization capabilities, we perform the same test for three different base speeds (in m/s) of the own ship: $U \in \{3, 4, 5\}$, even though both agents have only been trained on $U = \unit[3]{m/s}$. Figure <ref> presents the angular and spatial deviations of the local and global paths, respectively, while selected COLAV maneuvers are displayed in Figure <ref>.

Upon analyzing Figure <ref>, we observe that the maximum spatial deviation from the global path reaches approximately 200 meters, while the vast majority of deviations are smaller than 40 meters in absolute value. These deviations are relatively low, considering the fairway width of the Elbe is typically between 600 and 2000 meters in the selected segment. The local cross-track error, which measures the deviation from the planned local path, is naturally of smaller magnitude and exceeds 20 meters only in a few selected cases. Additionally, we notice a slight increase in the spatial deviation from both the local and global paths as the speed increases. However, this increase remains moderate and demonstrates the successful generalization ability of both agents. Moreover, the angular deviations show resilience to changes in speed and, importantly, do not differ significantly between the global and local levels. This is likely attributed to the replanning of the local path, which presents the PF agent with a new path to react to.

Global path for validation of the complete architecture based on AIS data.

Empirical distributions of the global and local cross-track and course errors over the complete journey from Lighthouse Tinsday to the Elbe estuary close to Cuxhaven.

[Head-On scenario at $U = 3 m/s$]
[Head-On scenario at $U = 5 m/s$]
[Enabling overtaking at $U = 3 m/s$]
[Overtaking at $U = 4 m/s$]
Encounter scenarios based on real depth and AIS data using the complete architecture for ASV control.

Analyzing the maneuvers depicted in Figure <ref>, it becomes evident that the framework possesses the capability to execute COLAV actions effectively. The model successfully performs starboard turns in head-on scenarios (Panels (a) and (b)), creates space for overtaking vessels (Panel (c)), and overtakes slower vessels on their portside (Panel (d)). These observations align with the validation results presented in Section <ref>, further highlighting the architecture's suitability for real-world scenarios. Notably, the maneuvers prove successful for all three selected speeds.

While the architecture demonstrates strong performance in various validation scenarios, it is essential to note that collisions have occurred in rare cases during the AIS testing routine. However, these collisions arise due to the nature of the testing format, where the ASV is deployed into scenarios with static target ship trajectories. Consequently, the target ships remain unaware of the presence of the own ship, as illustrated in Figure <ref>. Consider, for instance, Panel (c) of Figure <ref>, where the own ship makes room for a faster target ship. If the target ship suddenly executes a starboard turn while being alongside the own ship, a collision will occur regardless of the own ship's reaction. As stated by [68], it is impossible to avoid a collision if the target ship intentionally moves towards the own ship.

§.§ Discussion and practical challenges

The validation scenarios outlined in Sections <ref>, <ref>, and <ref> have thoroughly assessed the effectiveness of the LPP and PF agents, both individually and in tandem, establishing a comprehensive evaluation of our autonomous system. Notably, the validation using AIS data in Section <ref> delves into the behavior of our two-level system within the context of human vessel trajectories, demonstrating successful handling of various real-world encounter situations. However, we recognize the practical challenges associated with the interaction between autonomous systems and traditional, human-controlled vessels in maritime operations [54]. For instance, an autonomous vessel needs to accurately estimate the intention of the conventional ship, while the conventional similarly needs to predict the future trajectory of the ASV [88].

Recently, [91] presented several options for enhancing safety in encounters between conventional ships and ASVs, including the consideration of broadcasting intentions as a form of communication. We emphasize that such a broadcasting mechanism can be easily integrated into our architecture since the acquired information could be used during the planning iterations of the LPP agent. Generally, as emphasized by [1], collaboration through communication between vessels is a critical component of maritime operations, particularly in the context of autonomous systems.

§ CONCLUSION

The use of autonomous surface vessels for inland waterway transportation shows promise in creating a sustainable and economically attractive transportation system. Our study introduces a two-level architecture for ASVs operating on inland waterways based on DRL, employing separate agents for local path planning and path following. We consider relevant environmental disturbances, adhere to traffic rules, and validate our approach using simulated and real AIS trajectories.

We acknowledge the limitations of our work, which highlight avenues for future research. Firstly, we do not address limited visibility or sensor faults, which are critical high-risk scenarios in actual maritime operations. Secondly, as emphasized in Section <ref>, seafarers usually exchange communication signals when operating on inland waterways. Such information can be incorporated into our LPP unit, thereby removing the simplifying assumption of linear target ship movement during the planning iterations. Lastly, our focus on simulation experiments calls for validating the proposed architecture in real-world waterways. Addressing these limitations will enable further advancements in developing ASVs for inland waterway transportation, enhancing safety and effectiveness while paving the way for a sustainable transport system.

§ ACKNOWLEDGMENTS

The authors thank the members of the RL Dresden Group, especially Martin Treiber and Fabian Hart, for their constructive feedback on this work. Furthermore, the authors thank Thor Fossen for his insightful answers regarding the consideration of environmental forces in the simulation. The authors also acknowledge the Center for Information Services and High Performance Computing at TU Dresden for providing the resources for high-throughput calculations. Further, the authors thank the European Maritime Safety Agency for providing the AIS data for the validation scenarios. Finally, the authors express their gratitude to the Weiße Flotte Dresden for sharing their extensive experience in vessel operations. Their detailed explanations of the functionalities of the inland vessel 'Gräfin Kosel' have greatly enriched this research. Niklas Paulig was funded by BAW - Bundesanstalt für Wasserbau (Mikrosimulation des Schiffsverkehrs auf dem Niederrhein), Germany.

[1]
Akdağ, M., Solnør, P., & Johansen, T. A. (2022).
Collaborative collision avoidance for maritime autonomous surface ships: A review.
Ocean Engineering, 250:110920.

[2]
Al Enezy, O., van Hassel, E., Sys, C., & Vanelslander, T. (2017).
Developing a cost calculation model for inland navigation.
Research in Transportation Business & Management, 23:64–74.

[3]
Amin, O. M. & Hasegawa, K. (2010).
Generalised mathematical model for ship manoeuvrability considering shallow water effect.
In Conference Proceedings of the Japan Society of Naval Architects and Ocean Engineers, volume 10, pages 531–534.

[4]
Ankudinov, V., Miller, E., Jakobsen, B., & Daggett, L. (1990).
Manoeuvring performance of tug/barge assemblies in restricted waterways.
In International Conference on Marine Simulation and Ship Maneuverability, volume 90, pages 515–525.

[5]
Annamalai, A. S., Sutton, R., Yang, C., Culverhouse, P., & Sharma, S. (2015).
Robust adaptive control of an uninhabited surface vehicle.
Journal of Intelligent & Robotic Systems, 78:319–338.

[6]
Bačkalov, I., Vidić, M., & Rudaković, S. (2023).
Lessons learned from accidents on some major european inland waterways.
Ocean Engineering, 273:113918.

[7]
Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., & Wang, Z. (2020).
Autonomous navigation of stratospheric balloons using reinforcement learning.
Nature, 588(7836):77–82.

[8]
Bertsekas, D. (2019).
Reinforcement learning and optimal control.
Athena Scientific.

[9]
Breivik, M. & Fossen, T. I. (2009).
Guidance laws for autonomous underwater vehicles.
Underwater vehicles, 4:51–76.

[10]
Breu, D. A. & Fossen, T. I. (2011).
L1 adaptive and extremum seeking control applied to roll parametric resonance in ships.
In 2011 9th IEEE International Conference on Control and Automation (ICCA), pages 871–876. IEEE.

[11]
Bundesministerium für Digitales und Verkehr (1998).

[12]
Cao, S., Fan, P., Yan, T., Xie, C., Deng, J., Xu, F., & Shu, Y. (2022).
Inland waterway ship path planning based on improved RRT algorithm.
Journal of Marine Science and Engineering, 10(10):1460.

[13]
Castañeda, H., Rodriguez, J., & Gordillo, J. L. (2021).
Continuous and smooth differentiator based on adaptive sliding mode control for a quad-rotor mav.
Asian Journal of Control, 23(2):661–672.

[14]
Chen, L., Negenborn, R. R., & Lodewijks, G. (2016).
Path planning for autonomous inland vessels using A* BG.
In International Conference on Computational Logistics, pages 65–79. Springer.

[15]
Cheng, Y. & Zhang, W. (2018).
Concise deep reinforcement learning obstacle avoidance for underactuated unmanned marine vessels.
Neurocomputing, 272:63–73.

[16]
Chun, D.-H., Roh, M.-I., Lee, H.-W., Ha, J., & Yu, D. (2021).
Deep reinforcement learning-based collision avoidance for an autonomous ship.
Ocean Engineering, 234:109216.

[17]
de Barros, B. R. C., de Carvalho, E. B., & Junior, A. C. P. B. (2022).
Inland waterway transport and the 2030 agenda: Taxonomy of sustainability issues.
Cleaner Engineering and Technology, page 100462.

[18]
Ding, F., Zhang, Z., Fu, M., Wang, Y., & Wang, C. (2018).
Energy-efficient path planning and control approach of USV based on particle swarm optimization.
In OCEANS 2018 MTS/IEEE Charleston, pages 1–6.

[19]
Donha, D. C., Desanj, D., Katebi, M., & Grimble, M. (1998).
H$_\infty$ adaptive controllers for auto-pilot applications.
International Journal of Adaptive Control and Signal Processing, 12(8):623–648.

[20]
Eberhart, R. C. & Shi, Y. (2000).
Comparing inertia weights and constriction factors in particle swarm optimization.
In Proceedings of the 2000 Congress on Evolutionary Computation, volume 1, pages 84–88. IEEE.

[21]
E.U. Copernicus Marine Service (2023a).
Global ocean 1/12° physics analysis and forecast updated daily.
Accessed: 2023-04-17.

[22]
E.U. Copernicus Marine Service (2023b).
Global ocean daily gridded sea surface winds from scatterometer.
Accessed: 2023-04-17.

[23]
E.U. Copernicus Marine Service (2023c).
Global ocean waves analysis and forecast.
Accessed: 2023-04-17.

[24]
Faltinsen, O. (1993).
Sea loads on ships and offshore structures, volume 1.
Cambridge University Press.

[25]
Fan, Y., Sun, Z., & Wang, G. (2022).
A novel reinforcement learning collision avoidance algorithm for USVs based on maneuvering characteristics and COLREGs.
Sensors, 22(6):2099.

[26]
Fossen, T. I. (2021).
Handbook of Marine Craft Hydrodynamics and Motion Control, 2nd Edition.
John Wiley & Sons.

[27]
Fossen, T. I., Breivik, M., & Skjetne, R. (2003).
Line-of-sight path following of underactuated marine craft.
IFAC proceedings volumes, 36(21):211–216.

[28]
Fossen, T. I. & Pettersen, K. Y. (2014).
On uniform semiglobal exponential stability (USGES) of proportional line-of-sight guidance laws.
Automatica, 50(11):2912–2917.

[29]
Fujimoto, S., Hoof, H., & Meger, D. (2018).
Addressing function approximation error in actor-critic methods.
In International Conference on Machine Learning, pages 1587–1596. PMLR.

[30]
Gan, L., Yan, Z., Zhang, L., Liu, K., Zheng, Y., Zhou, C., & Shu, Y. (2022).
Ship path planning based on safety potential field in inland rivers.
Ocean Engineering, 260:111928.

[31]
Gonzalez-Garcia, A., Castañeda, H., & Garrido, L. (2020).
Usv path-following control based on deep reinforcement learning and adaptive control.
In Global Oceans 2020: Singapore–US Gulf Coast, pages 1–7. IEEE.

[32]
Goodfellow, I., Bengio, Y., & Courville, A. (2016).
Deep learning.
MIT press.

[33]
Guo, S., Zhang, X., Zheng, Y., & Du, Y. (2020).
An autonomous path planning model for unmanned ships based on deep reinforcement learning.
Sensors, 20(2):426.

[34]
Ha, J., Roh, M.-I., & Lee, H.-W. (2021).
Quantitative calculation method of the collision risk for collision avoidance in ship navigation using the cpa and ship domain.
Journal of Computational Design and Engineering, 8(3):894–909.

[35]
Hart, F. & Okhrin, O. (2022).
Enhanced method for reinforcement learning based dynamic obstacle avoidance by assessment of collision risk.
arXiv preprint arXiv:2212.04123.

[36]
Hart, F., Okhrin, O., & Treiber, M. (2023).
Vessel-following model for inland waterways based on deep reinforcement learning.
Ocean Engineering, 281:114679.

[37]
Hart, F., Waltz, M., & Okhrin, O. (2021).
The impact of missing velocity information in dynamic obstacle avoidance based on deep reinforcement learning.
arXiv preprint arXiv:2112.12465.

[38]
Hart, P. E., Nilsson, N. J., & Raphael, B. (1968).
A formal basis for the heuristic determination of minimum cost paths.
IEEE transactions on Systems Science and Cybernetics, 4(2):100–107.

[39]
Heiberg, A., Larsen, T. N., Meyer, E., Rasheed, A., San, O., & Varagnolo, D. (2022).
Risk-based implementation of COLREGs for autonomous surface vehicles using deep reinforcement learning.
Neural Networks, 152:17–33.

[40]
Hochreiter, S. & Schmidhuber, J. (1997).
Long short-term memory.
Neural computation, 9(8):1735–1780.

[41]
Hofbauer, F. & Putz, L.-M. (2020).
External costs in inland waterway transport: An analysis of external cost categories and calculation methods.
Sustainability, 12(14):5874.

[42]
Huang, Y., Chen, L., Chen, P., Negenborn, R. R., & Van Gelder, P. (2020).
Ship collision avoidance methods: State-of-the-art.
Safety science, 121:451–473.

[43]
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021).
How to train your robot with deep reinforcement learning: lessons we have learned.
The International Journal of Robotics Research, 40(4-5):698–721.

[44]
International Maritime Organization (1972).
COLREG: Convention on the International Regulations for Preventing Collisions at Sea.

[45]
International Maritime Organization (2023).
AIS transponders.
Accessed June 2, 2023.

[46]
International Telecommunication Union (2014).
Technical characteristics for an automatic identification system using time-division multiple access in the vhf maritime mobile band.
Recommendation ITU: Geneva, Switzerland.

[47]
Jadhav, A. K., Pandi, A. R., & Somayajula, A. (2023).
Collision avoidance for autonomous surface vessels using novel artificial potential fields.
Ocean Engineering, 288:116011.

[48]
Jin, Y. & Misra, S. (2023).
Controlling fracture propagation using deep reinforcement learning.
Engineering Applications of Artificial Intelligence, 122:106075.

[49]
Johansen, T. A., Perez, T., & Cristofaro, A. (2016).
Ship collision avoidance and COLREGS compliance using simulation-based control behavior selection with predictive hazard assessment.
IEEE Transactions on Intelligent Transportation Systems, 17(12):3407–3422.

[50]
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998).
Planning and acting in partially observable stochastic domains.
Artificial Intelligence, 101(1-2):99–134.

[51]
Kang, K., Belkhale, S., Kahn, G., Abbeel, P., & Levine, S. (2019).
Generalization through simulation: Integrating simulated and real data into deep reinforcement learning for vision-based autonomous flight.
In International Conference on Robotics and Automation, pages 6008–6014. IEEE.

[52]
Khatib, O. (1985).
Real-time obstacle avoidance for manipulators and mobile robots.
In Proceedings. 1985 IEEE International Conference on Robotics and Automation, volume 2, pages 500–505.

[53]
Kijima, K. & Nakiri, Y. (1990).
Prediction method of ship manoeuvrability in deep and shallow waters.
In International Conference on Marine Simulation and Ship Maneuverability, page 311.

[54]
Kim, T.-e., Perera, L. P., Sollid, M.-P., Batalden, B.-M., & Sydnes, A. K. (2022).
Safety challenges related to autonomous ships in mixed navigational environments.
WMU Journal of Maritime Affairs, 21(2):141–159.

[55]
Kingma, D. P. & Ba, J. (2014).
Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980.

[56]
Kuwata, Y., Wolf, M. T., Zarzhitsky, D., & Huntsberger, T. L. (2013).
Safe maritime autonomous navigation with COLREGS, using velocity obstacles.
IEEE Journal of Oceanic Engineering, 39(1):110–119.

[57]
Last, P., Bahlke, C., Hering-Bertram, M., & Linsen, L. (2014).
Comprehensive analysis of automatic identification system (AIS) data in regard to vessel movement prediction.
The Journal of Navigation, 67(5):791–809.

[58]
Lefeber, E., Pettersen, K. Y., & Nijmeijer, H. (2003).
Tracking control of an underactuated ship.
IEEE Transactions on Control Systems Technology, 11(1):52–61.

[59]
Lenart, A. S. (1983).
Collision threat parameters for a new radar display and plot technique.
The Journal of Navigation, 36(3):404–410.

[60]
Li, L., Wu, D., Huang, Y., & Yuan, Z.-M. (2021).
A path planning strategy unified with a COLREGs collision avoidance function based on deep reinforcement learning and artificial potential field.
Applied Ocean Research, 113:102759.

[61]
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015).
Continuous control with deep reinforcement learning.
arXiv preprint arXiv:1509.02971.

[62]
Liu, W., Qiu, K., Yang, X., Wang, R., Xiang, Z., Wang, Y., & Xu, W. (2023a).
COLREGS-based collision avoidance algorithm for unmanned surface vehicles using modified artificial potential fields.
Physical Communication, 57:101980.

[63]
Liu, X., Zhang, W., Shao, C., Wang, Y., & Cong, Q. (2023b).
Autonomous intelligent control of earth pressure balance shield machine based on deep reinforcement learning.
Engineering Applications of Artificial Intelligence, 125:106702.

[64]
Liu, Y., Bu, R., & Gao, X. (2018).
Ship trajectory tracking control system design based on sliding mode control algorithm.
Polish Maritime Research, 25(3):26–34.

[65]
Liu, Y. & Bucknall, R. (2015).
Path planning algorithm for unmanned surface vehicle formations in a practical maritime environment.
Ocean engineering, 97:126–144.

[66]
Liu, Z., Zhang, Y., Yu, X., & Yuan, C. (2016).
Unmanned surface vehicles: An overview of developments and challenges.
Annual Reviews in Control, 41:71–93.

[67]
Lyu, H. & Yin, Y. (2018).
Fast path planning for autonomous ships in restricted waters.
Applied Sciences, 8(12):2592.

[68]
Lyu, H. & Yin, Y. (2019).
COLREGS-constrained real-time path planning for autonomous ships using modified artificial potential fields.
The Journal of Navigation, 72(3):588–608.

[69]
Ma, J., Jia, C., Shu, Y., Liu, K., Zhang, Y., & Hu, Y. (2021).
Intent prediction of vessels in intersection waterway based on learning vessel motion patterns with early observations.
Ocean Engineering, 232:109154.

[70]
Martinsen, A. B. & Lekkas, A. M. (2018).
Curved path following with deep reinforcement learning: Results from three vessel models.
In OCEANS 2018 MTS/IEEE Charleston, pages 1–8. IEEE.

[71]
Matsuo, Y., LeCun, Y., Sahani, M., Precup, D., Silver, D., Sugiyama, M., Uchibe, E., & Morimoto, J. (2022).
Deep learning, reinforcement learning, and world models.
Neural Networks.

[72]
Meng, L., Gorbet, R., & Kulić, D. (2021).
Memory-based deep reinforcement learning for POMDPS.
In International Conference on Intelligent Robots and Systems, pages 5619–5626. IEEE.

[73]
Meyer, E., Heiberg, A., Rasheed, A., & San, O. (2020).
COLREG-compliant collision avoidance for unmanned surface vehicle using deep reinforcement learning.
IEEE Access, 8:165344–165364.

[74]
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015).
Human-level control through deep reinforcement learning.
Nature, 518(7540):529–533.

[75]
Motwani, A., Sharma, S., Sutton, R., & Culverhouse, P. (2013).
Interval kalman filtering in navigation system design for an uninhabited surface vehicle.
The Journal of Navigation, 66(5):639–652.

[76]
Mou, J. M., Van Der Tak, C., & Ligteringen, H. (2010).
Study on collision avoidance in busy waterways by using AIS data.
Ocean Engineering, 37(5-6):483–490.

[77]
Munim, Z. H., Dushenko, M., Jimenez, V. J., Shakil, M. H., & Imset, M. (2020).
Big data and artificial intelligence in the maritime industry: a bibliometric review and future research directions.
Maritime Policy & Management, 47(5):577–597.

[78]
Negenborn, R. R., Goerlandt, F., Johansen, T. A., Slaets, P., Valdez Banda, O. A., Vanelslander, T., & Ventikos, N. P. (2023).
Autonomous ships are on the horizon: here’s what we need to know.
Nature, 615(7950):30–33.

[79]
Nelson, D. R., Barber, D. B., McLain, T. W., & Beard, R. W. (2007).
Vector field path following for miniature air vehicles.
IEEE Transactions on Robotics, 23(3):519–529.

[80]
Öztürk, Ü., Akdağ, M., & Ayabakan, T. (2022).
A review of path planning algorithms in maritime autonomous surface ships: Navigation safety perspective.
Ocean Engineering, 251:111010.

[81]
Öztürk, Ü. & Cicek, K. (2019).
Individual collision risk assessment in ship navigation: A systematic literature review.
Ocean Engineering, 180:130–143.

[82]
Papoulias, F. A. (1993).
On the nonlinear dynamics of pursuit guidance for marine vehicles.
Journal of Ship Research, 37(04):342–353.

[83]
Paramesh, S. & Rajendran, S. (2021).
A unified seakeeping and manoeuvring model with a PID controller for path following of a KVLCC2 tanker in regular waves.
Applied Ocean Research, 116:102860.

[84]
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., & Chintala, S. (2019).
Pytorch: An imperative style, high-performance deep learning library.
Advances in Neural Information Processing Systems, 32:8026–8037.

[85]
Paulig, N. (2023).
pytsa: trajectories interpolation from AIS records.

[86]
Paulig, N. & Okhrin, O. (2023).
Robust path following on rivers using bootstrapped reinforcement learning.
arXiv preprint arXiv:2303.15178.

[87]
Peng, Z., Liu, E., Pan, C., Wang, H., Wang, D., & Liu, L. (2023).
Model-based deep reinforcement learning for data-driven motion control of an under-actuated unmanned surface vehicle: Path following and trajectory tracking.
Journal of the Franklin Institute, 360(6):4399–4426.

[88]
Porathe, T. & Rødseth, Ø. J. (2019).
Simplifying interactions between autonomous and conventional ships with e-navigation.
In Journal of Physics: Conference Series, volume 1357, page 012041. IOP Publishing.

[89]
Puterman, M. L. (2014).
Markov decision processes: discrete stochastic dynamic programming.
John Wiley & Sons.

[90]
Ringbom, H. (2019).
Regulating autonomous ships—concepts, challenges and precedents.
Ocean Development & International Law, 50(2-3):141–169.

[91]
Rødseth, Ø. J., Wennersberg, L. A. L., & Nordahl, H. (2023).
Improving safety of interactions between conventional and autonomous ships.
Ocean Engineering, 284:115206.

[92]
Rohács, J. & Simongati, G. (2007).
The role of inland waterway navigation in a sustainable transport system.
Transport, 22(3):148–153.

[93]
Rong, H., Teixeira, A., & Soares, C. G. (2022).
Ship collision avoidance behaviour recognition and analysis based on AIS data.
Ocean Engineering, 245:110479.

[94]
Sakamoto, T. & Baba, E. (1986).
Minimisation of resistance of slowly moving full hull forms in short waves.
In Proceedings of Sixteenth Symposium on Naval Hydrodynamics, pages 598–613.

[95]
Sawada, R., Sato, K., & Majima, T. (2021).
Automatic ship collision avoidance using deep reinforcement learning with LSTM in continuous action spaces.
Journal of Marine Science and Technology, 26(2):509–524.

[96]
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017).
Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347.

[97]
Serigstad, E., Eriksen, B.-O. H., & Breivik, M. (2018).
Hybrid collision avoidance for autonomous surface vehicles.
IFAC-PapersOnLine, 51(29):1–7.

[98]
Sharma, S., Naeem, W., & Sutton, R. (2012).
An autopilot based on a local control network design for an unmanned surface vehicle.
The Journal of Navigation, 65(2):281–301.

[99]
Siciliano, B., Khatib, O., & Kröger, T. (2008).
Springer handbook of robotics, volume 200.

[100]
Siegwart, R., Nourbakhsh, I. R., & Scaramuzza, D. (2011).
Introduction to autonomous mobile robots.
MIT press.

[101]
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018).
A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.
Science, 362(6419):1140–1144.

[102]
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014).
Deterministic policy gradient algorithms.
In International Conference on Machine Learning, pages 387–395. Pmlr.

[103]
Singh, Y., Sharma, S., Sutton, R., Hatton, D., & Khan, A. (2018).
A constrained A* approach towards optimal path planning for an unmanned surface vehicle in a maritime environment containing dynamic obstacles and ocean currents.
Ocean Engineering, 169:187–201.

[104]
Sutton, R. S. & Barto, A. G. (2018).
Reinforcement Learning: An Introduction.
Cambridge: The MIT Press.

[105]
Szepesvári, C. (2010).
Algorithms for reinforcement learning.
Synthesis lectures on artificial intelligence and machine learning, 4(1):1–103.

[106]
Szlapczynski, R. & Szlapczynska, J. (2017).
Review of ship safety domains: Models and applications.
Ocean Engineering, 145:277–289.

[107]
Taimuri, G., Matusiak, J., Mikkola, T., Kujala, P., & Hirdaris, S. (2020).
A 6-DoF maneuvering model for the rapid estimation of hydrodynamic actions in deep and shallow waters.
Ocean Engineering, 218:108103.

[108]
Tam, C. & Bucknall, R. (2010).
Path-planning algorithm for ships in close-range encounters.
Journal of Marine Science and Technology, 15:395–407.

[109]
Tam, C., Bucknall, R., & Greig, A. (2009).
Review of collision avoidance and path planning methods for ships in close range encounters.
The Journal of Navigation, 62(3):455–476.

[110]
Toyoda, S. & Fujii, Y. (1971).
Marine traffic engineering.
The Journal of Navigation, 24(1):24–34.

[111]
Treiber, M. & Kanagaraj, V. (2015).
Comparing numerical integration schemes for time-continuous car-following models.
Physica A: Statistical Mechanics and its Applications, 419:183–195.

[112]
Tu, E., Zhang, G., Rachmawati, L., Rajabally, E., & Huang, G.-B. (2017).
Exploiting AIS data for intelligent maritime navigation: A comprehensive survey from data to methodology.
IEEE Transactions on Intelligent Transportation Systems, 19(5):1559–1582.

[113]
Vagale, A., Bye, R. T., Oucheikh, R., Osen, O. L., & Fossen, T. I. (2021a).
Path planning and collision avoidance for autonomous surface vehicles II: a comparative study of algorithms.
Journal of Marine Science and Technology, 26(4):1307–1323.

[114]
Vagale, A., Oucheikh, R., Bye, R. T., Osen, O. L., & Fossen, T. I. (2021b).
Path planning and collision avoidance for autonomous surface vehicles I: a review.
Journal of Marine Science and Technology, 26:1292–1306.

[115]
Van Rossum, G. & Drake, F. L. (2009).
Python 3 Reference Manual.
CreateSpace, Scotts Valley, CA.

[116]
Vanneste, A., Vanneste, S., Vasseur, O., Janssens, R., Billast, M., Anwar, A., Mets, K., De Schepper, T., Mercelis, S., & Hellinckx, P. (2022).
Safety aware autonomous path planning using model predictive reinforcement learning for inland waterways.
In Annual Conference of the IEEE Industrial Electronics Society, pages 1–6.

[117]
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019).
Grandmaster level in starcraft II using multi-agent reinforcement learning.
Nature, 575(7782):350–354.

[118]
Waltz, M. & Okhrin, O. (2022).
Addressing maximization bias in reinforcement learning with two-sample testing.
arXiv preprint arXiv:2201.08078.

[119]
Waltz, M. & Okhrin, O. (2023).
Spatial–temporal recurrent reinforcement learning for autonomous ships.
Neural Networks, 165:634–653.

[120]
Waltz, M. & Paulig, N. (2022).
RL Dresden Algorithm Suite.

[121]
Wan, L., Su, Y., Zhang, H., Shi, B., & AbouOmar, M. S. (2020).
An improved integral light-of-sight guidance law for path following of unmanned surface vehicles.
Ocean Engineering, 205:107302.

[122]
Wang, P., Gao, S., Li, L., Sun, B., & Cheng, S. (2019).
Obstacle avoidance path planning design for autonomous driving vehicles based on an improved artificial potential field algorithm.
Energies, 12(12):2342.

[123]
Wang, X., Liu, Z., & Cai, Y. (2017).
The ship maneuverability based collision avoidance dynamic support system in close-quarters situation.
Ocean Engineering, 146:486–497.

[124]
Wang, Y., Cao, J., Sun, J., Zou, X., & Sun, C. (2023).
Path following control for unmanned surface vehicles: A reinforcement learning-based method with experimental validation.
IEEE Transactions on Neural Networks and Learning Systems.

[125]
Woo, J., Yu, C., & Kim, N. (2019).
Deep reinforcement learning-based controller for path following of an unmanned surface vehicle.
Ocean Engineering, 183:155–166.

[126]
Xu, H. & Guedes Soares, C. (2023).
Review of path-following control systems for maritime autonomous surface ships.
Journal of Marine Science and Application, 22(2):153–171.

[127]
Xu, X., Cai, P., Ahmed, Z., Yellapu, V. S., & Zhang, W. (2022a).
Path planning and dynamic collision avoidance algorithm under COLREGs via deep reinforcement learning.
Neurocomputing, 468:181–197.

[128]
Xu, X., Lu, Y., Liu, G., Cai, P., & Zhang, W. (2022b).
COLREGs-abiding hybrid collision avoidance algorithm based on deep reinforcement learning for USVs.
Ocean Engineering, 247:110749.

[129]
Yasukawa, H. & Yoshimura, Y. (2015).
Introduction of MMG standard method for ship maneuvering predictions.
Journal of Marine Science and Technology, 20(1):37–52.

[130]
Yu, H., Meng, Q., Fang, Z., Liu, J., & Xu, L. (2023).
A review of ship collision risk assessment, hotspot detection and path planning for maritime traffic control in restricted waters.
The Journal of Navigation, pages 1–27.

[131]
Zhai, P., Zhang, Y., & Shaobo, W. (2022).
Intelligent ship collision avoidance algorithm based on DDQN with prioritized experience replay under COLREGs.
Journal of Marine Science and Engineering, 10(5):585.

[132]
Zhang, X., Yang, G., Zhang, Q., Zhang, G., & Zhang, Y. (2017).
Improved concise backstepping control of course keeping for ships using nonlinear feedback technique.
The Journal of Navigation, 70(6):1401–1414.

[133]
Zhang, Y., Chen, P., Chen, L., & Mou, J. (2023).
A path planning method for the autonomous ship in restricted bridge area based on anisotropic fast marching algorithm.
Ocean Engineering, 269:113546.

[134]
Zhao, Y., Ma, Y., & Hu, S. (2021).
Usv formation and path-following control via deep reinforcement learning with random braking.
IEEE Transactions on Neural Networks and Learning Systems, 32(12):5468–5478.

[135]
Zhao, Y., Qi, X., Ma, Y., Li, Z., Malekian, R., & Sotelo, M. A. (2020).
Path following optimization for an underactuated usv using smoothly-convergent deep reinforcement learning.
IEEE Transactions on Intelligent Transportation Systems, 22(10):6208–6220.

[136]
Zhou, Y., Daamen, W., Vellinga, T., & Hoogendoorn, S. (2019).
Review of maritime traffic models from vessel behavior modeling perspective.
Transportation Research Part C: Emerging Technologies, 105:323–345.

[137]
Zhuge, D., Wang, S., Zhen, L., & Psaraftis, H. N. (2023).
Data-driven modeling of maritime transportation: Key issues, challenges, and solutions.
DOI: 10.1016/j.eng.2022.12.009.

Appendix section

§ GERMAN MARITIME TRAFFIC RULES

The following paragraph is an excerpt from the German regulation for maritime shipping lanes [11], which we consider in our research. We first state the original legal text in German, followed by the official English translation.

23 Überholen

(1) Grundsätzlich muß links überholt werden. Soweit die Umstände des Falles es erfordern, darf rechts überholt

(2) Das überholende Fahrzeug muß unter Beachtung von Regel 9 Buchstabe e und Regel 13 der Kollisionsverhütungsregeln die Fahrt so weit herabsetzen oder einen solchen seitlichen Abstand vom vorausfahrenden Fahrzeug einhalten, daß kein gefährlicher Sog entstehen kann und während des ganzen
Überholmanövers jede Gefährdung des Gegenverkehrs ausgeschlossen ist. Das vorausfahrende Fahrzeug muß das Überholen soweit wie möglich erleichtern.

23 Overtaking

(1) As a rule, an overtaking vessel shall pass the vessel being overtaken on the latter vessel’s port side. If the circumstances of the case so require, the overtaking vessel may pass the vessel being overtaken on the latter vessel’s starboard side.

(2) The overtaking vessel, acting in compliance with the provisions of Rule 9(e) and Rule 13 of the International Regulations for Preventing Collisions at Sea, 1972, as amended, shall slacken her speed so much, respectively, shall give the vessel being overtaken such a wide berth that no dangerous suction or wash can develop and that no vessel proceeding in the opposite direction will be put at any risk for the entire duration of the overtaking process. The vessel being overtaken shall facilitate the overtaking vessel’s action to the greatest possible extent.

Appendix section

§ TARGET SHIP CONTROL

In Algorithm <ref>, we specify the behavior of the target ships during the training and validation phases of the LPP agent. The returned angle from the algorithm is assigned as the new heading for the controlled target ship. The phrase closest ship in the algorithm refers to the surrounding ship with the closest Euclidean distance between the vessel's midpoints.

$B, L_{pp}$: width and length between perpendiculars of the controlled ship

$\chi_{d}$: desired course from VFG (with gain parameter $k = 0.001$)

$\chi_{P_k}$: path angle between current waypoints

$\Tilde{d}$: distance to closest ship

$U_0, U_1$: speed of the controlled ship and its closest surrounding ship

$t^{\rm CPA}, d^{\rm CPA}$: time and distance to CPA with the closest ship

$\alpha$: relative bearing of the controlled ship from the perspective of its closest surrounding ship

$\sigma$: boolean, True if closest ship travels in reversed direction, False otherwise

Turn right in case of opposing traffic
($t^{\rm CPA}$ $>$ 0) $\land$ ($d^{\rm CPA}$ $<$ 2B) $\land$ ($\Tilde{d} \leq$ 10Lpp) $\land$ $\sigma$
return $\chi_{d} + 5^{\circ}$

Overtake left
($U_0 > U_1$) $\land$ ($\Tilde{d} \leq$ 10Lpp) $\land$ ($135^{\circ} \leq \alpha \leq 315^{\circ}$)
$y_e > 0$
return $\psi_{dc} - 8^\circ$

return $\chi_{P_k} - 8^\circ\cdot\exp\left[(y_e/5B) \log(4)\right]$

Default VFG guidance

return $\chi_{d}$

Rule-based heading control for target ships on inland waterways

Appendix section

§ BASELINE: ARTIFICIAL POTENTIAL FIELD METHOD

We select an APF approach as a baseline method for the local path planning task. The approach is based on the recent proposals of [62] and [122], which we slightly adapt to improve the method's performance on inland waterway scenarios. According to the original proposal from the robotics domain of [52], an APF can be constructed by superpositioning attractive and repulsive forces. The attractive forces $F_{\text{att},t}$ at time step $t$ pull the planning agent toward the desired goal position, while the repulsive forces $F_{\text{rep},t}$ push the agent away from obstacles to avoid collisions. The resulting total force $F_t = F_{\text{att},t} + F_{\text{rep},t}$ is used to derive a desired heading as follows:

$$\psi_{d,t} = \operatorname{arctan}\left(F_{e,t}/F_{n,t}\right),$$

where $F_{n,t}$ and $F_{e,t}$ are the north and east components of $F_t$, respectively. Based on the desired heading $\psi_{d,t}$, the new heading can be set via:

$$\psi_{\text{OS}, t+1} = \psi_{\text{OS}, t} + \operatorname{clip}\left(\psi_{d,t} - \psi_{\text{OS}, t}, -\Delta_{\psi}, \Delta_{\psi} \right),$$

where $\psi_{\text{OS},t}$ is the heading of the own ship of time $t$ and $\Delta_{\psi}$ is the maximum possible heading change between time steps, considering the limited maneuverability of the vessel. Figure <ref> visualizes the attractive and repulsive forces imposed in this study, which we describe in detail in the following.

Visualization of the forces of the APF method building on [62] and [122].

Following [62], we construct the attractive forces as a sum of two components: $F_{\text{att}, t}= F_{\text{att1}, t} + F_{\text{att2},t}$. The first component, $F_{\text{att1}, t}$, is the goal-oriented attractive force:

\begin{equation*}
F_{\text{att1}, t} =
\begin{cases}
k_{a1} \cdot d_{\text{OS},t}^{\rm G} \cdot \vec{n}_{\rm OG} & \text{if} \quad d_{\text{OS},t}^{\rm G} < d^*,\\
k_{a1} \cdot d^* \cdot \vec{n}_{\rm OG} & \rm else,
\end{cases}
\end{equation*}

where $k_{a1}$ and $d^*$ are constants, $d_{\text{OS},t}^{\rm G}$ is the Euclidean distance between the own ship and the goal point at time $t$, and $\vec{n}_{\rm OG}$ is the unit vector pointing from the own ship toward the goal. We thereby define the goal as a future waypoint of the global path, which is updated as the vessel progresses. The second component, $F_{\text{att2},t}$, is responsible for pulling the vessel back toward the path to the goal if the cross-track error is too large. Formally, we have:

\begin{equation*}
F_{\text{att2}, t} =
\begin{cases}
k_{a2} \cdot \vert y_{e,t} \vert \cdot \vec{n}_{\rm OP} & \text{if} \quad y_{e,t} \geq d_l,\\
0 & \rm else,
\end{cases}
\end{equation*}

where $k_{\rm a2}$ and $d_l$ are constants, $y_{e,t}$ is the cross-track error of the own ship at time $t$, and $\vec{n}_{\rm OP}$ is the unit vector from the own ship perpendicular to the linear path between the initial vessel position and the goal; see [62] and Figure <ref>.

Denoting the repulsive force with each target ship $i$ at time $t$ as $F_{\text{rep},i,t}$, the overall repulsive force is attained via summation:
$$F_{\text{rep},t} = \sum_i F_{\text{rep},i,t}.$$
Similar to the attractive forces, we construct the repulsive force with relation to target ship $i$ as the sum of two components. The first component, $F_{\text{rep1},i, t}$, is the conventional repulsive force pushing the vessel away from the obstacle [52], while the second component, $F_{\text{rep2},i, t}$, being orthogonal to $F_{\text{rep1},i, t}$, is inspired by [62] and ensures that overtaking occurs on the portside of the target ships.

Crucially, we impose two constraints that must be fulfilled to achieve a non-zero repulsive force from an obstacle. The first follows the proposal of [122] and requires the own ship to be in an ellipse around a target ship, allowing the repulsive force to differ in the lateral and longitudinal direction of the target vessel. Note that this approach is similar to how we specified the collision reward component of the LPP agent in Section <ref>. The second constraint is that the TCPA between the own ship and the target ship at time $t$, denoted $t_{i,t}^{\rm cpa}$, has to be larger or equal to zero. We found this second constraint crucial to improving the performance of the APF methods on our inland waterway scenarios since it eliminates repulsive forces after a vessel has been overtaken. Aggregating these thoughts, we specify:

\begin{equation*}
F_{\text{rep},i, t} =
\begin{cases}
F_{\text{rep1},i, t} + F_{\text{rep2},i, t} \quad \text{if} & \left[p_{\text{OS},t} \in \operatorname{Ell}(i, a_{\rm lon}, a_{\rm lat})\right] \land (t_{i,t}^{\rm cpa} \geq 0),\\
0 & \rm else,
\end{cases}
\end{equation*}

where $p_{\text{OS},t}$ is the current north-east position of the own ship, and $\operatorname{Ell}(i, a_{\rm lon}, a_{\rm lat})$ is the set of points inside the ellipse around target ship $i$. The semi-major axis of the ellipse has length $a_{\rm lon}$ and points in the direction of the heading of the target ship, while the semi-minor axis has length $a_{\rm lat}$.

Following [52], we thereby set:

$$F_{\text{rep1},i, t} = k_{r1} \cdot \left(\frac{1}{d_{\text{OS},t}^{i}} - \frac{1}{d_0}\right) \cdot \frac{1}{\left(d_{\text{OS},t}^{i}\right)^2} \cdot \vec{n}_{\rm TO},$$

where $k_{\rm r1}$ and $d_0$ are constant coefficients, and $d_{\text{OS},t}^{i}$ is the Euclidean distance between the own ship and target ship $i$ at time $t$. The unit vector $\vec{n}_{\rm TO}$ points from the target ship toward the own ship. Lastly, we define:

\begin{equation}\label{eq:F_rep2it}
F_{\text{rep2},i, t} =
\begin{cases}
k_{r2} \cdot \vec{n}_{\rm OT\perp} & \text{if} \quad
\vert[\psi_{i,t}-\psi_{\text{OS},t}]_{-\pi}^{\pi}\vert < \frac{\pi}{2},\\
0 & \rm else,
\end{cases}
\end{equation}

where $k_{\rm  r2}$ is a constant, $\vec{n}_{\rm OT\perp}$ is the unit vector that is perpendicular to portside on the unit vector from the own ship to the target ship. The variables $\psi_{i,t}$ and $\psi_{\text{OS},t}$ are the heading of the target ship $i$ and the own ship, respectively. Hence, the condition in (<ref>) requires that the vessels travel in opposing directions. Otherwise, the overtaking-related force $F_{\text{rep2},i, t}$ is set to zero.

We empirically determine all parameters via a grid search, leading to the values: $\Delta_\psi = 2^\circ$, $d^* = \unit[0.5]{NM}$, $d_l = \unit[50]{m}$, $k_{\rm a1} = 1$, $k_{\rm a2} = 0.1$, $k_{\rm r1} = 0.1$, $k_{\rm r2} = 0.1$, $a_{\rm lat} = d_0 = \unit[0.5]{NM}$, and $a_{\rm lon} = \unit[0.04]{NM}$.

Appendix section

§ FURTHER VALIDATION: LOCAL PATH PLANNING AGENT

In the following, we provide further details on the discussed validation scenarios from Section <ref>, including the results for curved waterway segments of the LPP agent. As emphasized in Section <ref>, the APF method cannot be directly transferred to curved scenarios since its hyperparameters have been carefully tuned for the straight waterway case. Table <ref> includes the initial speed configuration of the target ships, while the speed of the own ship is set to $\unit[3]{m/s}$ in all cases. Furthermore, Figures <ref> and <ref> display the trajectories of the DRL planning agent for right and left curves, respectively. The black trajectories in these figures correspond to the DRL agent, while the colorized ones are the target ships controlled by Algorithm <ref>. The purple and grey dotted lines are the global and reversed global paths. Moreover, we display additional metrics during the curved scenarios in Figures <ref> and <ref>, respectively.

Similar to the findings for the straight waterway segment, the LPP agent demonstrates the successful execution of all necessary maneuvers while effectively maintaining safe distances and avoiding consecutive large changes in heading. It is worth highlighting that the agent's need to navigate through curved paths in the scenarios depicted in Figures <ref> and <ref> is also reflected in the average action values as shown in Figures <ref> and <ref>, respectively. In these figures, we observe a slightly positive average action in the right curve scenario, while a slightly negative average action is observed in the left curve scenario.

Scenario     Waterway     $U_1$ [m/s]      $U_2$ [m/s]
$U_3$ [m/s]     $U_4$ [m/s]     $U_5$ [m/s]

1     straight     1.50     1.50     1.50     1.50     1.50

2     straight     1.05     1.65     -     -     -

3     straight     1.20     2.10     1.20     -     -

4     straight     1.20     2.10     1.20     1.65     -

5     straight     4.50     -     -     -     -

6     straight     0.00     0.00     0.00     0.00     0.0

7     right curve     1.50     1.50     1.50     1.50     1.50

8     right curve     1.05     1.65     -     -     -

9     right curve     1.20     2.10     1.20     -     -

10     right curve     1.20     2.10     1.20     1.65     -

11     right curve     4.50     -     -     -     -

12     right curve     0.00     0.00     0.00     0.00     0.0

13     left curve     1.50     1.50     1.50     1.50     1.50

14     left curve     1.05     1.65     -     -     -

15     left curve     1.20     2.10     1.20     -     -

16     left curve     1.20     2.10     1.20     1.65     -

17     left curve     4.50     -     -     -     -

18     left curve     0.00     0.00     0.00     0.00     0.0

Initial target ship speeds for the validation scenarios in the LPP task. The variable $U_i$, $i=1,\ldots,5$, is the speed of target ship $i$.