Bridging Active Exploration and Uncertainty-Aware Deployment Using
  Probabilistic Ensemble Neural Network Dynamics
In recent years, learning-based control in robotics has gained significant
attention due to its capability to address complex tasks in real-world
environments. With the advances in machine learning algorithms and
computational capabilities, this approach is becoming increasingly important
for solving challenging control problems in robotics by learning unknown or
partially known robot dynamics. Active exploration, in which a robot directs
itself to states that yield the highest information gain, is essential for
efficient data collection and minimizing human supervision. Similarly,
uncertainty-aware deployment has been a growing concern in robotic control, as
uncertain actions informed by the learned model can lead to unstable motions or
failure. However, active exploration and uncertainty-aware deployment have been
studied independently, and there is limited literature that seamlessly
integrates them. This paper presents a unified model-based reinforcement
learning framework that bridges these two tasks in the robotics control domain.
Our framework uses a probabilistic ensemble neural network for dynamics
learning, allowing the quantification of epistemic uncertainty via Jensen-Renyi
Divergence. The two opposing tasks of exploration and deployment are optimized
through state-of-the-art sampling-based MPC, resulting in efficient collection
of training data and successful avoidance of uncertain state-action spaces. We
conduct experiments on both autonomous vehicles and wheeled robots, showing
promising results for both exploration and deployment.
Taekyung Kim, Jungwi Mun, Junwon Seo, Beomsu Kim, Seongil Hong
Bridging Active Exploration and Uncertainty-Aware Deployment Using Probabilistic Ensemble Neural Network Dynamics

###### Abstract

In recent years, learning-based control in robotics has gained significant attention due to its capability to address complex tasks in real-world environments. With the advances in machine learning algorithms and computational capabilities, this approach is becoming increasingly important for solving challenging control problems in robotics by learning unknown or partially known robot dynamics. Active exploration, in which a robot directs itself to states that yield the highest information gain, is essential for efficient data collection and minimizing human supervision. Similarly, uncertainty-aware deployment has been a growing concern in robotic control, as uncertain actions informed by the learned model can lead to unstable motions or failure. However, active exploration and uncertainty-aware deployment have been studied independently, and there is limited literature that seamlessly integrates them. This paper presents a unified model-based reinforcement learning framework that bridges these two tasks in the robotics control domain. Our framework uses a probabilistic ensemble neural network for dynamics learning, allowing the quantification of epistemic uncertainty via Jensen-Renyi Divergence. The two opposing tasks of exploration and deployment are optimized through state-of-the-art sampling-based MPC, resulting in efficient collection of training data and successful avoidance of uncertain state-action spaces. We conduct experiments on both autonomous vehicles and wheeled robots, showing promising results for both exploration and deployment.

## I Introduction

Learning-based robotic control has been an increasingly active research topic in recent decades, pushing the boundary
of robotic control tasks with new capabilities through model-free reinforcement learning [1, 2, 3, 4], model-based reinforcement learning [5, 6, 7, 8, 9, 10, 11], and robot dynamics learning [12, 13, 14, 15]. There are two key aspects that are commonly taken into account while deploying an intelligent robot. The first is to assure safety, and the second, if at all possible, is to achieve maximum performance.

Model-based methods have been regarded as being advantageous in robotic applications over pure model-free strategies for various reasons. Firstly, sample efficiency is essential since real-world samples are highly expensive in terms of time, labor, and finances [16]. Secondly, humans are primarily more intuitive about how to incorporate prior knowledge into a model compared to a policy or value function [14, 17]. Lastly, models are task-agnostic and may thus be utilized to optimize arbitrary cost functions, whereas the majority of model-free policies are bounded to a specific task.

These benefits have led to recent control paradigms that include dynamics learning into a model-based control framework, outperforming the conventional methods that rely on manually tuned models based on human insights into physics [10, 18, 19]. This scheme may also be described as a model-based reinforcement learning problem, as the model is continuously trained to provide a more accurate representation of the system, while an optimization-based controller serves as the policy. Such a control framework possesses the inherent flexibility to dynamically adjust during execution, for instance, to changing target speeds or actuator limits. Gaussian Process (GP) is a typical regression method in model learning problems due to its advantageous features, such as the ability to cope with small data samples and incorporate prior knowledge [6, 20]. However, it does not scale well to large datasets and to high-dimensional systems. To mitigate this issue, recent robotics studies have benefited from neural networks owing to their universal function approximation properties with high model capacity.

The limitation of these model-based control frameworks is that the accuracy of the learned model drastically affects the control performance [21]. However, the real-world is fraught with uncertainty, making the model learning problem complex and challenging. There are two primary viewpoints within the robotics community that tackle inevitable uncertainty in opposing ways.

A spontaneous solution would be to prevent the robot from entering uncertain state-action spaces to evade unpredictable motions. This is a straightforward strategy, given that an inaccurate prediction of the dynamics model might result in substantial losses in control performance and, in the worst case, instability. We refer to such an uncertainty-averse strategy as _uncertainty-aware deployment_ (or safety-aware deployment), and it is usually applied during execution after completing the robot's training.

The exact opposite of the above strategy is to deliberately visit unexplored state-action spaces that provide high uncertainty. The hypothesis is that by trying to explore uncertain segments of the learned model, it can learn more precise representation of the system with fewer samples and discover behaviors with higher performance. Such uncertainty-seeking strategy is known as _active exploration_, and it is considered during the early phase of training data exploration.

We argue that these opposite yet analogous strategies addressing uncertainty must be entailed concurrently in a learning-based control framework. No matter how much data is gathered through active exploration, the uncertainty that eventually has not been eliminated needs to be monitored and avoided throughout robot deployment. Conversely, when we take the model uncertainties for risk-averse planning, they are only meaningful if the model is trained with the right data; otherwise, the controller will produce erroneous motions. Nevertheless, these two approaches have largely been studied independently, and there is little literature that integrates them in a seamless manner.

This paper introduces a unified model-based reinforcement learning framework in a robot dynamics learning setting, seamlessly bridging active exploration and uncertainty-aware deployment to fulfill both safety and maximum performance. Fig. 1 shows that the proposed framework consists of two phases. In _exploration phase_, a parallelized ensemble neural network serves as the robot dynamics and outputs the estimated posterior distribution of the next state. In _deployment phase_, the neural network dynamics trained during the active exploration phase is applied directly to perform uncertainty-aware control. Both tasks are optimized using the state-of-the-art sampling-based Model Predictive Control (MPC), which, owing to its property, allows the insertion of arbitrary cost functions after training.

In summary, the contributions of this paper are as follows:

* We introduce a fully parallelized probabilistic ensemble neural network that is sensitive to uncertainty in order to learn robot dynamics.
* We separate epistemic uncertainty from model disagreement for active exploration.
* We transfer the neural network dynamics for uncertainty-aware deployment with minimal modification.
* We conduct extensive experiments with our framework on both autonomous vehicles and wheeled robots, outperforming the compared methods by a large margin.

## II Related Works

### _Model Learning with Uncertainty_

A regression model capable of accurately capturing the nonlinear dynamics is essential for model-based control in uncertain environments with nonlinearity. Data-driven models approximating the robot's forward dynamics have been extensively studied in numerous prior research, including GP [6, 20], time-varying linear models [22], set-membership identification [23, 24], locally weighted projection regression [25, 26], etc. GP has long been favored in dynamics learning given its simplicity and its inherent nature to account for both types of uncertainty: aleatoric uncertainty and epistemic uncertainty.

Aleatoric uncertainty arises from intrinsic stochasticities present in a system, such as observation noise and process
noise. Epistemic uncertainty, however, originates from a lack of sufficient data to uniquely determine the underlying characteristics of the target system in an approximated model. An ideal model trained with an infinite number of data may have zero epistemic uncertainty, but it is not feasible to eradicate it completely in a realistic setting.

Neural networks are known for being tractable to large training datasets while assuring constant-time inference and having the potential to learn highly complex and even non-smooth robot dynamics. The recent breakthrough in the field of dynamics learning is that neural networks may also capture both types of uncertainty by using an ensemble of probabilistic neural networks [27]. Vlastelica et al. [28] proposed to quantify epistemic uncertainty by taking empirical variances over the means and variances of the propagated posterior distribution of each ensemble. We investigate further to determine a more exact epistemic uncertainty in order to provide a better exploration bonus (which is discussed in Section IV-A).

### _Active Exploration_

Human-supervised data collection is time-consuming and can be biased by the fact that manual control of the robot may not encompass all of its possible motions. As a remedy, it was suggested that the agent should plan exploration by directing itself to the states that yield the largest information gain, therefore performing active exploration [29]. In general, the measured model uncertainty is leveraged as the exploration motivation to minimize the required amount of data to gather [30].

We argue that the exploration strategy should be _imaginary_ and _online_. If the agent is _reactive_ to immediate curiosity [31], which is the opposite to being _imaginary_, it disregards the long-term effects of control actions to the system. Consequently, it must struggle in realms of underactuated continuous control [32]. In addition, since most robotic systems are intrinsically stochastic, the real trajectories would quickly diverge from the planned ones. Therefore, it is desirable to plan future trajectory _online_ at every time step, rather than relying on the optimized trajectory found _offline_ as in [32].

Several studies address model-based active exploration based on the above hypothesis. Examples of this line of research include the work in [33, 34, 35], in which model-free policies are employed to maximize expected information gain in a specific task. In [36], GP is used for model learning, while a gradient-based MPC handles the information gain acquired from the GP model. In this work, we adopt sampling-based MPC as a policy and demonstrate that it is scalable to handle uncertainty derived from neural networks.

### _Safe Deployment_

Uncertainty-averse deployment has been addressed as a major concern throughout robotic control history, and there is a growing interest in its application to model-based reinforcement learning. The uncertain actions informed by the learned model may result in unstable motions or even catastrophic failure.

Kahn et al. [9] incorporated the estimated uncertainty as a cost to learn collision avoidance. Yu et al. [37] penalized uncertainty of the dynamics in an offline reinforcement learning setting to resolve the distributional shift issue. Wang et al. [38] treated trajectories having greater uncertainty than a certain threshold as constraint conditions in an application of rough terrain navigation. Similar to [38], we penalize uncertainty with both soft and hard costs to maximize the control performance while ensuring safety.

## III Methods

### _Probabilistic Ensemble Neural Network_

Consider a general continuous time system dynamics \(\dot{\mathbf{x}}_{t+1}=\mathbf{F}(\mathbf{x}_{t},\mathbf{u}_{t})\) where \(\mathbf{F}\) is a nonlinear function, \(\mathbf{x}_{t}\in\mathbb{R}^{n}\) and \(\mathbf{u}_{t}\in\mathbb{R}^{m}\) are the observed state vector and applied action input at time \(t\), respectively. To account for both aleatoric and epistemic uncertainty, we use Probabilistic Ensemble Neural Network (PENN) to approximate the unknown dynamics \(\mathbf{F}\) via data-driven manner [27, 39].

Each neural network predictive model \(\mathbf{E}_{b}\) of the ensemble members outputs a Gaussian distribution conditioned on the model input:

\[\mathbf{E}_{b}(\mathbf{x}_{t},\mathbf{u}_{t};\theta_{b})=\mathcal{N}(\mathbf{ \mu}_{\theta_{b}}(\mathbf{x}_{t},\mathbf{u}_{t}),\mathbf{\Sigma}_{\theta_{b}}( \mathbf{x}_{t},\mathbf{u}_{t}))\,, \tag{1}\]

where \(\mathbf{\mu}\) and \(\mathbf{\Sigma}\) are the mean and the diagonal covariance parameterized by the model parameter \(\theta_{b}\). Given a collected dataset of transitions \(\mathcal{D}\triangleq\{(\mathbf{x}_{t},\mathbf{u}_{t},\dot{\mathbf{x}}_{t+1}) \}_{t=0}^{N-1}\) with the size of \(N\), the probabilistic model can be trained with the Gaussian Negative Log-Likelihood (NLL) loss function:

\[\begin{split}\mathcal{L}_{\text{train}}(\theta_{b})=\sum_{t=0}^{N -1}\big{[}\mathbf{\mu}_{\theta_{b}}-\dot{\mathbf{x}}_{t+1}\big{]}^{\top}\mathbf{ \Sigma}_{\theta_{b}}^{-1}\big{[}\mathbf{\mu}_{\theta_{b}}-\dot{\mathbf{x}}_{t+1} \big{]}\\ +\log\left[\det\mathbf{\Sigma}_{\theta_{b}}\right]\,.\end{split} \tag{2}\]

Note that random initialization is applied individually to each ensemble member, resulting in mutually independent initial weights. Consequently, the only difference between the ensemble members is their initial model parameters.

Consequently, the stochastic dynamics function predicting the next state can be interpreted as a Gaussian Mixture Model (GMM) of the \(B\) ensemble members:

\[\mathbf{F}(\theta_{1:B})=\sum_{b=1}^{B}\pi_{b}\mathbf{E}_{b}(\mathbf{x}_{t}, \mathbf{u}_{t};\theta_{b})\,,\quad 0\leq\pi_{b}\leq 1\,. \tag{3}\]

Here, we simply use equal weights \(\forall\pi_{b}=\frac{1}{B}\). Then, the next state of the system during iterative trajectory rollouts can be defined as:

\[\mathbf{x}_{t+1}=\mathbf{x}_{t}+\mathbb{E}\left[\mathbf{E}_{b}(\mathbf{x}_{t}, \mathbf{u}_{t};\theta_{b})\right]\Delta t\,,\quad b\sim\{1,\dots,B\}\,. \tag{4}\]

### _Model Predictive Active Exploration_

To boost the model learning process, we introduce an active exploration strategy in the _training phase_. We encourage the robot to seek automatically to visit currently unforeseen state-action pairs for training data. To this end, ensemble model disagreement \(D\) can be used as exploration bonuses [33]. One
possible solution is to measure the empirical variance over the predictive samples \(\tilde{\mathbf{x}}_{t+1}\) from the posterior distribution of each ensemble member:

\[\mathfrak{U}(\mathbf{x}_{t},\mathbf{u}_{t})=\mathrm{Var}\left(\left\{\tilde{ \mathbf{x}}_{t+1}\sim\mathbf{E}_{b}(\mathbf{x}_{t},\mathbf{u}_{t})\right\}_{b= 1}^{B}\right), \tag{5}\]

which is referred as _uncertainty sampling_. However, this method cannot distinguish epistemic uncertainty from aleatoric uncertainty. Importantly, the robot should only be inquisitive about the epistemic uncertainty and not the aleatoric uncertainty; otherwise, it will be obscure if this exploration bonus resulted from a lack of knowledge or from inherent and irreducible system stochasticity, such as model ambiguity or sensor noise [32].

Kullback-Leibler (KL) divergence is the most well-known metric for measuring the disagreement between distributions to obtain epistemic uncertainty. However, KL divergence is asymmetric and can only be used to determine the distance between two distributions. Therefore, we employ Jensen-Renyi Divergence (JRD)

\[\mathrm{JRD}\left(\mathbf{E}_{1:B};\alpha\right)\triangleq H_{\alpha}\left( \sum_{b=1}^{B}\pi_{b}\mathbf{E}_{b}\right)-\sum_{b=1}^{B}\pi_{b}H_{\alpha} \left(\mathbf{E}_{b}\right), \tag{6}\]

where the Renyi entropy [40] of a random variable \(X\) and its corresponding density function \(p(x)\) is defined as:

\[H_{\alpha}(X\mid x\in X)=\frac{1}{1-\alpha}\log\int p(x)^{\alpha}\mathrm{d}x\,. \tag{7}\]

While an approximation of the JRD can be obtained via Monte-Carlo sampling, it is computationally expensive, thereby precluding real-time implementation. To circumvent this problem, Wang et al. [41] introduce a closed-form JRD with quadratic entropy (\(\alpha=2\)) for analytical measurement of the divergence of a GMM:

\[\begin{split} D(\mathbf{x}_{t},\mathbf{u}_{t};\mathbf{E}_{1:B})=- \log&\Bigg{[}\frac{1}{B^{2}}\sum_{i,j}^{B}\mathfrak{D}\left( \mathbf{E}_{i},\mathbf{E}_{j}\right)\Bigg{]}+\\ &\frac{1}{B}\sum_{i}^{B}\log\left[\mathfrak{D}\left(\mathbf{E}_{i },\mathbf{E}_{i}\right)\right],\end{split} \tag{8}\]

where

\[\mathfrak{D}\left(\mathbf{E}_{i},\mathbf{E}_{j}\right)=\frac{1}{|\mathbf{ \Phi}|^{\frac{1}{2}}}\exp\left(-\frac{1}{2}\mathbf{\Delta}^{\top}\mathbf{\Phi }^{-1}\mathbf{\Delta}\right), \tag{9}\]

\(\mathbf{\Phi}=\mathbf{\Sigma}_{\theta_{i}}+\mathbf{\Sigma}_{\theta_{j}}\) and \(\mathbf{\Delta}=\boldsymbol{\mu}_{\theta_{i}}-\boldsymbol{\mu}_{\theta_{i}}\).

Using this metric, Shyam et al. [33] presented a novel active exploration strategy by measuring the ensemble model disagreement along the future predictions. The proposed method trained a model-free policy to harness this time-varying and intricate exploration bonus. However, due to the sample inefficiency and the inability to adjust the trained policy to achieve modified task-specific objectives, this method has difficulty being used to robotic applications. To make these limitations tractable, we here employ Model Predictive Path Integral (MPPI) control [8] as a policy.

MPPI is the state-of-the-art sampling-based MPC that enables real-time implementation with the aid of Graphic Processing Units (GPUs) due to its parallelizable structure [18]. Denoting by \(U=\{\mathbf{u}_{0},\ldots,\mathbf{u}_{T-1}\}\) with a fixed time horizon \(T\), the MPPI algorithm seeks an optimal control sequence \(U^{*}\) such that:

\[U^{*}=\operatorname*{argmin}_{U}\mathbb{E}\left[\phi(\mathbf{x}_{T})+\sum_{t= 0}^{T-1}\mathcal{L}\left(\mathbf{x}_{t},\mathbf{u}_{t}\right)\right], \tag{10}\]

where \(\phi\left(\cdot\right)\) is a state-dependent terminal cost. In general, the running cost with the quadratic control cost with control weight matrix \(\mathbf{R}\) takes the form:

\[\mathcal{L}\left(\mathbf{x}_{t},\mathbf{u}_{t}\right)=q\left(\mathbf{x}_{t} \right)+\frac{1}{2}\mathbf{u}_{t}^{\top}\mathbf{R}\mathbf{u}_{t}\,. \tag{11}\]

Note that the state-dependent running cost \(q\left(\cdot\right)\) can be arbitrary functions, allowed by the sampling-based optimization scheme that is capable of solving non-convex problems. Consequently, we can include the information gain obtained from the model disagreement (8) into the controller's objective (11) in order to jointly achieve task-dependent cost optimization and active exploration:

\[\mathcal{L}_{\text{active}}\left(\mathbf{x}_{t},\mathbf{u}_{t}\right)=q\left( \mathbf{x}_{t}\right)+\frac{1}{2}\mathbf{u}_{t}^{\top}\mathbf{R}\mathbf{u}_{t }-w_{D}\,D(\mathbf{x}_{t},\mathbf{u}_{t})\,, \tag{12}\]

where \(w_{D}>0\) is a weighting constant. As a result, the robot is encouraged to choose behaviors that lead to exploring uncertain state-action spaces of the dynamic model, while still attempting to minimize the task-specific error.

The above characteristic facilitates the robotics applications of this method in three ways. First, the controller actively seeks states and actions having high epistemic uncertainty, i.e., for which there are little training data, in the vicinity of the set task objective. Thus, the trained model is completely tailored for instantaneous deployment. Second, it is conceivable to determine when the _exploration phase_ may be terminated; specifically, it is when the robot optimizes mostly for the task cost. As training data accumulate, the ensemble models will gradually converge toward increasingly similar predictions, resulting in a progressive decrease in model disagreement. Finally, the robot will be able to adapt to the changes in task costs and to solve the modified tasks without re-training.

### _Uncertainty-Aware Deployment_

After sufficient training time, the robot may be deployed to perform some designated tasks. In the _deployment phase_, we generally regard that there are no human supervisors and no further dynamics update. Recent research has shown that parallel ensemble neural networks are capable of learning dynamics online [42]. However it is orthogonal to the goal of this paper, and our method also can benefit from this consideration. For successful operations, the robot must avoid unpredictable situations, which can be accomplished by incorporating uncertainty into safety violations, regardless of the source of uncertainty. For instance, a data point may have no disagreement across ensembles due to sufficient training
data (low epistemic uncertainty), but we must still avoid this point if the variance of the predicted posterior distribution is high (high aleatoric uncertainty).

In this phase, we use a variant of the MPPI algorithm called Smooth MPPI (SMPPI) [43]. SMPPI shares the same information-theoretic roots as MPPI, and it also benefits from the structure of parallel trajectory evaluation. However, SMPPI lifts the control variables as derivative actions, so that the noisy sampling is performed on a higher order domain \(\dot{U}=\{\dot{\mathbf{u}}_{0},\ldots,\dot{\mathbf{u}}_{T-1}\}\). Let us denote the resulting optimal control trajectory after the optimization process as \(\dot{U}_{i+1}^{*}\) at MPC iteration step \(i\). Then, the optimal action trajectory \(U_{i+1}^{*}\), which will be used as the actual commands sent to the robot, is obtained by the integral with respect to the MPC iteration horizon: \(U_{i+1}^{*}=U_{i}^{*}+\dot{U}_{i+1}^{*}\Delta i\), where \(\Delta i\) represents the control updating period of MPC and is commonly equal to \(\Delta t\). This new action trajectory update law allows the controller to rapidly respond to changing environments while using significantly lower sampling variance, thus alleviating the chattering in resulting commands. Furthermore, we can apply an extra action smoothing cost \(\Omega(U)\) along trajectory rollouts:

\[\Omega(U)=\sum_{t=1}^{T-1}(\mathbf{u}_{t}-\mathbf{u}_{t-1})^{\top}\mathbf{\omega} \left(\mathbf{u}_{t}-\mathbf{u}_{t-1}\right), \tag{13}\]

where \(\mathbf{\omega}\in\mathbb{R}^{m\times m}\) is the weighting parameter. Such action cost was not eligible in the MPPI baseline with non-affine dynamics because it violates the information theoretic derivation [43]. As a result of these smoothing effects, SMPPI is not suitable for rapid exploration in underactuated continuous control systems but is beneficial for deployment.

To achieve risk-averse planning, taking into account both aleatoric and epistemic uncertainty, we employ the results of _uncertainty sampling_ (5) as uncertainty measurements. A simple addition of the quantified uncertainty to the cost function would make the controller uncertainty-aware, as the following form:

\[\mathcal{L}_{\text{naive}}\left(\mathbf{x}_{t},\dot{\mathbf{u}}_{t},\mathbf{u }_{t}\right)=q\left(\mathbf{x}_{t}\right)+\frac{1}{2}\dot{\mathbf{u}}_{t}^{ \top}\mathbf{R}\dot{\mathbf{u}}_{t}\,+w_{1}\mathbb{U}(\mathbf{x}_{t},\mathbf{ u}_{t}), \tag{14}\]

where \(w_{1}>0\) is a weighting constant. It is important to point out that the measured uncertainty through sampling is not differentiable with respect to state and action. Sampling-based MPC is capable of optimizing any arbitrarily crafted cost functions, making such a form of uncertainty tractable. Similarly, we take use of this trait by imposing an impulse-like penalty where uncertainty exceeds a threshold \(\xi\):

\[\mathcal{L}_{\text{hybrid}}\left(\mathbf{x}_{t},\dot{\mathbf{u}} _{t},\mathbf{u}_{t}\right)=q\left(\mathbf{x}_{t}\right)+\frac{1}{2}\dot{ \mathbf{u}}_{t}^{\top}\mathbf{R}\dot{\mathbf{u}}_{t}\,+w_{1}\mathbb{U}( \mathbf{x}_{t},\mathbf{u}_{t})+\\ w_{2}\,I\Big{(}\Big{\{}\Big{|}\mathbb{U}(\mathbf{x}_{t},\mathbf{ u}_{t})\Big{|}>\xi\Big{\}}\Big{)}, \tag{15}\]

where \(w_{2}\gg w_{1}\) is a weighting constant and \(I\) is an indicator function. It is an intuitive consideration because we never want to allow the robot to take actions that are completely unpredictable.

Following is the final objective function of SMPPI:

\[\dot{U}^{*}=\operatorname*{argmin}_{U}\mathbb{E}\Bigg{[}\phi( \mathbf{x}_{T})+ \Omega(U+\dot{U}\Delta i) \tag{16}\] \[+\sum_{t=0}^{T-1}\mathcal{L}_{\text{hybrid}}\left(\mathbf{x}_{t}, \dot{\mathbf{u}}_{t},\mathbf{u}_{t}\right)\Bigg{]}.\]

### _Implementing Neural Network Vehicle Dynamics_

Given that we have made no assumption on the specific robotic platform in our framework, we argue that it is applicable to any kind of robotic system. In this work, we focus on autonomous vehicle and wheeled robot applications to validate our idea. We choose the robot's state and action input based on the dynamic bicycle model since it provides a good trade-off between model accuracy and simplicity for real-time implementation [13]. The dynamic state variable \(\mathbf{x}=(v_{x},v_{y},r)^{\top}\) consists of longitudinal velocity, lateral velocity, and yaw rate. These states can be easily measured by using GPS and IMU sensors. The action input \(\mathbf{u}=(\delta,v_{des})^{\top}\) consists of the steering angle and desired speed. The desired speed is sent to the low-level Proportional-Integral (PI) controller, which determines the throttle and brake.

Since these state and action variables are simplified representations of the full robot dynamics, there exists irreducible model ambiguity. For example, the roll and pitch motions are not considered in the bicycle model. Automatic gear shifting encompasses non-smooth and time-varying dynamics characteristics, hence increasing the lower bound of model bias. We alleviate this problem by providing the history of state-action pairs to the neural network input so that it can extract contextual information [44, 45]. The history length \(H\) must be carefully determined, because as \(H\) increases, the state-action space that has to be discovered becomes exponentially larger, making the exploration problem more difficult. We select \(H=4\) in this work through an ablation study in the history length (see Section VI-A). Although LSTM [46] and GRU [47] show slightly better prediction performances than Multi-Layer Perceptron (MLP) (see Section VI-A), we choose MLP since its operations are fully parallelizable.

We take the parallel ensemble MLP implementation by _philipball_[48] and adapt it for our task. Let us denote the weight and the bias of a layer of an ensemble model \(\mathbf{E}_{b}\) as \(\mathbf{W}_{b}\) and \(\mathbf{b}_{b}\), respectively. Then, the parameters of all ensemble models can be represented in the form of batches of matrices: \(\mathbf{W}=[\mathbf{W}_{1}^{\top},\ldots,\mathbf{W}_{B}^{\top}]^{\top}\), \(\mathbf{b}=[\mathbf{b}_{1},\ldots,\mathbf{b}_{B}]^{\top}\). First, we initialize the parameters of each ensemble model according to the size of the layer input \(n_{\text{in}}\):

\[\mathbf{W}_{b},\,\mathbf{b}_{b}\sim\mathcal{U}\left(-\sqrt{\frac{1}{n_{\text{ in}}}},+\sqrt{\frac{1}{n_{\text{in}}}}\right),\forall b\in\{1,\ldots,B\}, \tag{17}\]

where \(\mathcal{U}\) is uniform distribution. While training the model with collected data, the forward propagation of each ensemble model is computed through

\[\mathbf{o}_{b}=\mathbf{W}_{b}^{\top}\mathbf{z}+\mathbf{b}_{b},\quad\forall b \in\{1,\ldots,B\}, \tag{18}\]
where \(\mathbf{z}\) and \(\mathbf{o}\) are the input and the output of the layer, respectively. The gradients of the Gaussian NLL loss function (2) can be obtained through automatic differentiation packages such as Pytorch autograd[49]. This process is identical to the common way we use MLP. While using the model as the dynamics of the robot, on the other hand, the forward propagation of the entire ensemble model is defined as batch matrix-matrix multiplication and addition:

\[\left[\mathbf{o}_{1},\ldots,\mathbf{o}_{B}\right]^{\top}=\mathbf{W}^{\top} \odot\left[\mathbf{z},\ldots,\mathbf{z}\right]^{\top}\oplus\mathbf{b}, \tag{19}\]

where \(\odot\) and \(\oplus\) are element-wise operations. This parallelized forward propagation (19) can be implemented using Pytorch badDmm function. Therefore, with the aid of sufficient GPU resources, we can simultaneously predict the next state and obtain uncertainty using PENN dynamics in a fixed-time duration regardless of the number of ensembles. This strategy maximizes the benefits of the parallel sample evaluation property of the MPPI framework.

## IV Experiments

Our experimental evaluations address the following key questions: **(Q1)** Can our method explore the state-action space with sufficient sample efficiency (see Section IV-A)? **(Q2)** Can the PENN taught with active exploration be used for long horizon planning, even if the task objective has been modified after training (see Section IV-B)? **(Q3)** Can our method be turned directly into an uncertainty-aware controller with minimal modification (see Section IV-C)? **(Q4)** Is our method scalable to other robot platforms (see Section IV-D)?

We evaluate **Q1-Q3** in a high-fidelity vehicle dynamics simulator - IPG CarMaker. We build the modeling and simulation system by integrating the algorithms using ROS 2 [50]. A Land Rover Defender 110 is used as the control vehicle. The low-level PI controller runs at 100 Hz, and the control parameters \(K_{\text{P}},K_{\text{I}}\) are set to {0.35, 1.0} for throttle and {0.18, 0.3} for brake. To simulate sensor noise, we add 10% of i.i.d. Gaussian noise to each dynamic state observation in \(\mathbf{x}\). MPPI is used in the active exploration experiments **(Q1)**, and SMPPI is used in the deployment experiments **(Q2-Q3)**. Table I lists the parameters of two controllers, while the remaining parameters not specified in this paper are taken from the SMPPI implementation [43].

We use a four-layer MLP with the hidden layer sizes {40, 80, 120, 40} throughout all experiments. Rectified Linear Unit (ReLU) [51] is used as activation functions. Similar to [27], we use five ensembles (\(B=5\)) for real-time implementation.

### _Model Predictive Active Exploration_

**Experimental Setup.** In this experiment, we evaluate the effectiveness of the active exploration strategy **(Q1)**. We let the robot seek for training data on a large flat ground with 0.7 friction coefficient. The task objective is to maintain the 50 km/h target speed (\(v_{\text{target}}\)) while staying inside the space boundary. Accordingly, the task-dependent running cost \(q(\mathbf{x}_{t})\) is defined as follows:

\[\begin{split} q(\mathbf{x}_{t})&=\alpha_{1}\text{ Track}(\mathbf{x}_{t})+\alpha_{2}\text{Speed}(\mathbf{x}_{t})\,,\\ \text{Track}(\mathbf{x}_{t})&=(0.9)^{t}\,10000\, \textbf{M}(p_{x},p_{y})\,,\\ \text{Speed}(\mathbf{x}_{t})&=\left(\sqrt{{v_{x}}^{2}+{ v_{y}}^{2}}-v_{\text{target}}\right)^{2}.\end{split} \tag{20}\]

\(\alpha_{(.)}\) are empirically tuned weighting constants. The track cost imposes a hard penalty to prevent collisions. The given map **M** indicates whether the robot position \((p_{x},p_{y})\) is at the outside of the space boundary.

We assume that there are human monitors who supervise the _exploration phase_. They disengage the robot's autonomy before the robot reaches the space boundary or before the robot enters dangerous situations. We empirically assume this period as 30 s. While the human supervisors setting up the robot for a new trial, the PENN is trained with the collected data offline. We refer to this process as one _iteration_ for exploration.

**Methods Evaluated.** We evaluate the following methods including ours:

#### Iv-A1 Ours

a model predictive active exploration strategy, which optimizes the objective function of (12). The epistemic uncertainty measured by JRD (8) serves as the information gain. We set the information gain weighting constant \(w_{D}\) as 100.

#### Iv-A2 Uncertainty Sampling (US)

an uncertainty sampling counter-part of our method, which uses the uncertainty quantified through sampling over each ensemble posterior distribution as the information gain. The substitution of JRD (8) with uncertainty sampling (5) in (12) is used as its running cost. We use the same weighting \(w_{D}=100\) as our method since the quantified information gain in both methods is of similar magnitude in our implementation.

#### Iv-A3 Jensen-Renyi Divergence Reactive Exploration (JDRX)

a reactive counter-part of our method [33]. We simply implement this based on our method by setting the planning horizon to \(T=1\), which explores greedily without planning based on the experience collected so far.

#### Iv-A4 Random Noise (RN)

a random exploration strategy, which injects noise in the action input. Its objective function is (11) and there is no information gain. We sample random noise from the Gaussian distribution with the same variances as the sampling distribution used by MPPI.

All the compared methods are evaluated with the same control parameters and the same model training setting with a fixed random seed. While training the model offline, we

\begin{table}
\begin{tabular}{c|c|c} Parameters & MPPI [18] & SMPPI [43] \\ \hline \(\Delta t\) & 0.1 s & 0.1 s \\ \(\Delta i\) & 0.1 s & 0.1 s \\ \(K\) & 5,000 & 5,000 \\ \(T\) & 10 & 35 \\ Sampling Variance & \(\text{Diag}(4.0,3.0)\) & \(\text{Diag}(1.6,0.4)\) \\ \(\mathbf{\omega}\) & N/A & \(\text{Diag}(0.4,0.1)\) \\ \end{tabular}
\end{table} TABLE I: Control parameters of MPPI and SMPPI. Note that the action smoothing cost with \(\mathbf{\omega}\) only applies to SMPPI.
randomly split the collected data \(\mathcal{D}\) into the training and test sets with a 7:3 ratio. The PENN is trained using the Adam optimizer with a learning rate of 0.001. We save the model that shows the best test sets performance during 10 epochs, and use this model as the robot dynamics \(\mathbf{F}\) in the next iteration.

**Experimental Results.**

We first evaluate how much of the state-action space each method has covered. We compute the convex hull [52] with the collected state and action data \(\{\mathbf{x}_{t},\mathbf{u}_{t}\}_{t=0}^{N-1}\) at each iteration, and analyze the volume enlargement of the convex hulls [53] (see Fig. 2). The results indicate that JDRX cannot explore the state-action space efficiently. Also, the volumes of ours and RN show comparable magnitudes while consistently exceeding those of US.

We further analyze the collected data points in detail. Similar to as Spielberg et al. [54], we focus on the state envelope of sideslip \(\beta\) and yaw rate \(r\), since precise motion predictions with large sideslip angles and rotational speed are crucial to the success of high-speed driving on sharp curves. The results are shown in Fig 3. JDRX is not able to collect driving data with high yaw rates. This is because it reacts immediately to the trivial information gain near the current state and is unable to plan for higher information gain in the future. On the other hand, ours always covers the largest state space among the compared methods. This is because ours can plan over a longer horizon to gather drifting data by ignoring the aleatoric uncertainty exudes from plain driving. Our method has collected a substantial amount of data with \(\beta\) around 0.2 rad (approximately 11.5 \({}^{\circ}\)), and even data with \(\beta\) around 0.3 rad (approximately 17.2 \({}^{\circ}\)). Lastly, we stress that the data collected by RN is not helpful for achieving the driving tasks, despite it showing a similar enlargement in the convex hull to our method. We will further discuss this in Section IV-B.

### _Direct Deployment_

**Experimental Setup.** We take the resulted PENNs of active exploration as the robot dynamics and directly evaluate their performances in long horizon planning (**Q2**). We load the best models at each iteration and monitor the control performance improvements of the PENNs of the compared methods. The performances are evaluated in a race track with three moderate curves and five sharp curves (see Fig. 3(a)). We set the friction coefficient of the track to 0.7.

In this experiment, the objective of the robot is to drive through the race track at a speed as close to \(v_{\text{target}}\) as possible, while staying inside of the track. We modify the task-dependent cost \(q(\mathbf{x}_{t})\) as follows:

\[q(\mathbf{x}_{t})=\alpha_{1}\text{Track}(\mathbf{x}_{t})+\alpha_{2}\text{Speed }(\mathbf{x}_{t})+\alpha_{3}\text{Stable}(\mathbf{x}_{t}). \tag{21}\]

The track cost and the speed cost are nearly identical to those in the exploration phase. The given map \(\mathbf{M}\) is modified accordingly to the new race track. The target speed \(v_{\text{target}}\) is adjusted to 45 km/h. We add the stabilizing cost that imposes a hard penalty on large sideslip angles \(\beta\) to prevent the robot from losing its stability:

\[\begin{split}\text{Stable}(\mathbf{x}_{t})&=10000\, I\left(\{|\beta|>0.3\}\right)\,,\\ \beta&=-\text{arctan}\left(\frac{v_{y}}{\|v_{x}\|} \right)\,.\end{split} \tag{22}\]

We set the planning horizon to \(T=35\), which is 3.5 s, to compare the performances in long horizon planning. Note that no information gain is provided to the objective function, so the controllers are uncertainty-neutral.

**Experimental Results.** The results are shown in Fig. 4. At the sharpest corners of \(\#3\), \(\#7\), and \(\#8\), in order to drive through them without losing significant speed, the vehicle should gently slow down before entering the corners and drift continuously. If the vehicle cannot perform drifting maneuvers, it will nearly stop in the middle of the corners or even collide with the track boundary. Moreover, if the predictions of the dynamics are not accurate, the vehicle may enter a saturated drifting region and become uncontrollable.

First, we count the number of times that each method completes the whole track without driving outside of the given boundary (see Fig. 3(b)). JDRX is not able to complete the whole lap. Reactive exploration limits the robot to acquiring only insignificant information gain in the vicinity of the current state. As a result, it fails to collect high-speed driving data for training and to produce accurate predictions in states with high speeds. RN shows improvement as it collects more data through repeated iterations. However, it only relies on coincidental actions that lead to collecting novel data. Our method achieves the best results among the compared methods, and its performance steadily improves as more data is collected.

Interestingly, the performance of US improves in the early stages, but after 100 iterations, its performance continuously decreases. It implies that collecting more data does not necessarily improve the prediction performance. Since US cannot separate the source of uncertainty, it may repeatedly collect the same data showing naive motions. Therefore, the model is trained to make increasingly better predictions on these superfluous data while neglecting the data representing important driving characteristics.

We also analyze the control performance in terms of speed (see Fig. 3(c)) and stability (see Fig. 3(d)) during each trial. Our method has effectively regulated the speed cost compared to the other methods. The results also show that RN violates the stabilizing constraints far more frequently than our method. It demonstrates that the collected data from RN do not effectively contribute to achieving the given task, since they omit crucial training data such as high-speed drifting

Fig. 2: The volumes of the convex hulls along exploration iterations.
maneuvers. Only our method successfully drives through the race track at high speeds without losing stability.

### _Uncertainty-Aware Deployment_

**Experimental Setup.** We assumed that the _exploration phase_ was performed in a large open space under human supervision. However, in _deployment phase_, there is no such monitoring to guarantee safety. In this experiment, we validate that our active exploration policy can be transferred to an uncertainty-aware controller with minimal modification.

To highlight the influence of the uncertainty-aware control strategy, we augment the dynamics to be terrain-aware by feeding the elevation map to the neural network. We assume that a 2.5D elevation map with a 0.1 m grid size is provided by one of the existing methods that can generate these maps from raw sensor input in real time [55, 56]. The map encoder network consists of an 8-channel convolutional layer and a 4-channel convolutional layer, both with a kernel size of 3 and a stride of 2 [57]. The output of the encoder is concatenated with the history of state-action pairs and fed into the original MLP model that had been used throughout the previous experiments. We use the data collected by our method, which shows the best performance in direct deployment experiments, to train the augmented dynamics. During training, the elevation map is randomly generated from the Gaussian distribution of \(\mathcal{N}(0,0.2)\).

Following our prior work [58], we design a realistic off-road environment with various components including large bumps and randomly patterned rough terrains (see Fig. 5). We also add non-traversable elements such as trees, bushes, and wooden pillars, and they are included in the given map **M** for collision avoidance.

**Methods Evaluated.** We take the same task-dependent cost function (21) used in the direct deployment experiments. The target speed \(v_{\text{target}}\) is adjusted to 30 km/h. The key difference between these two deployment tasks is that we here include uncertainty in the control objective. We evaluate the following uncertainty-averse strategies:

#### Vi-C1 Uncertainty-Neutral (UN)

a uncertainty neutral strategy that does not take uncertainty into account while planning. It uses the same objective function as the one used in direct deployment experiments.

#### Vi-C2 Naive Penalty (NP)

a simple uncertainty-aware strategy that penalizes a linear cost on quantified uncertainty. It uses (14) as the objective function with \(w_{1}=1000\).

#### Vi-C3 Hybrid

an uncertainty-aware strategy that has both soft and hard costs. It uses (15) as the objective function with \(w_{1}=1000\), \(w_{2}=10000\), and \(\xi=0.15\). Note that the hard cost penalizes states that are estimated to have uncertainty greater than \(\xi\) with the same magnitude as a collision with the track boundary.

#### Vi-C4 Conservative

a conservative strategy that controls the vehicle as stable as possible. It uses (15) as the objective function with \(w_{1}=0\), \(w_{2}=10000\), and \(\xi=0.02\).

**Experimental Results.** We visualize the trajectories taken by the compared methods in Fig. 5. To illustrate the vehicle's stability, we display the rotational impacts exerted on the vehicle during driving onto the trajectories. We simply use angular momentum, obtained by the squared sum of roll rate and pitch rate. We normalize the angular momentum by the maximum angular momentum value across all trials. Therefore, a trajectory with a darker color represents that the vehicle is more unstable. The angular and vertical motions of the vehicle during experiments are shown in Table II for quantitative evaluation.

Although UN can control the vehicle at the reference speed while avoiding collisions with obstacles and staying on track, it fails to circumvent uncertain regions because it is blind to uncertainty. It drives over the bumps at Scenario #1 and Scenario #5, eventually resulting in overturning. NP has opted for terrain with less uncertainty and, as a result, has chosen moderate terrain rather than the high bumps (see Scenario #1 and Scenario #5). However, it fails to circumvent the largest bump at Scenario #3. On the other hand, Hybrid decides to slow down before the bump and take a detour to satisfy the safety criteria, despite the speed penalty. Conservative is a reasonable option in practical applications because it minimizes the damage to the robot. It takes the most secure trajectory as can be seen at Scenario #4 and Scenario #5 due to the low uncertainty threshold. The trajectory taken by

Fig. 3: The scatter plots of collected data during active exploration. The data are from 10, 30, 100, and 300 iterations, respectively. During all iterations, ours using JRD information gain covers the largest state spaces of sideslip angle and yaw rate compared to other methods.
Conservative also shows that the vehicle rarely experienced terrain impacts except at the inevitable bump at Scenario #2. Hybrid shows a good balance between safety and agility. It avoids regions with high degrees of uncertainty, which might irreversibly damage or destroy the vehicle, while navigating the terrains with low levels of uncertainty at the desired speed. Table II demonstrates that the methods with hard uncertainty penalty show a significantly lower degree of undesirable vertical motions and angular motions compared to the other methods.

### _Experiments on Wheeled Robots_

In this section, we conduct a sanity check to ensure that our framework is scalable to other types of robot platforms. We use a 1:5 scale wheeled robotics testbed as the novel target system. We examine our framework in two robotics simulators: Gazebo [59] and Nvidia Isaac Sim [60]. In each simulator, we conduct the active exploration experiments **(Q1)** and direct deployment experiments **(Q2)** with the best performing exploration strategy. Note that we do not modify any settings in our framework except for adjusting the target speed according to the robot's specifications (\(v_{\text{target}}\) = 15 km/h) in both exploration and deployment phases. Other experimental settings are identical to those described in previous sections, including parameters of neural networks, controllers, and objective functions. Fig. 6 shows the simulated environments and the resulting trajectories in deployment tests taken by

\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{\#} & \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Vertical Vel. [m/s]**} & \multicolumn{2}{c}{**Vertical Acc. [m/s\({}^{2}\)]**} & \multicolumn{2}{c}{**Roll Rate [rad/s]**} & \multicolumn{2}{c}{**Pitch Rate [rad/s]**} & \multicolumn{2}{c}{**Roll Acc. [rad/s\({}^{2}\)]**} & \multicolumn{2}{c}{**Pitch Acc. [rad/s\({}^{2}\)]**} \\ \cline{3-13}  & & **Mean** & **Max** & **Mean** & **Max** & **Mean** & **Max** & **Mean** & **Max** & **Mean** & **Max** & **Mean** & **Max** \\ \hline
1 & Uncertainty-Neutral (UN) & 2.07 & 28.23 & 0.13 & 1.09 & 0.28 & 2.46 & 0.43 & **1.10** & 2.22 & 30.67 & 1.64 & 12.78 \\
2 & Naive Penalty (NP) & 1.97 & 23.30 & 0.05 & 0.38 & 0.33 & 2.06 & 0.44 & 2.22 & 2.73 & 33.97 & 1.80 & 8.02 \\ \hline
3 & Hybrid & 1.42 & **9.37** & **0.04** & **0.20** & 0.21 & **1.56** & **0.39** & 1.15 & 1.85 & **18.37** & **1.28** & **5.91** \\
4 & Conservative & **1.40** & 10.60 & **0.04** & 0.22 & **0.14** & 1.60 & 0.42 & 1.11 & **1.16** & 22.23 & 1.46 & 6.93 \\ \hline \hline \end{tabular}
\end{table} TABLE II: The average and maximum motions of the vehicle across all trials. The vertical motions include vertical velocity and vertical acceleration. The angular motions include velocities and accelerations of roll and pitch, respectively. Hybrid and Conservative experience lower terrain impacts than UN and NP by a large margin.

Fig. 4: (a) The race track designed in the IPG CarMaker simulator for direct deployment experiments. We visualize the vehicle trajectory taken by our method at 300 iterations for driving 10 laps in a counter-clockwise direction. (b) The number of times that each method completes the whole lap during every 50 iterations. (c) The average speed cost for each saved model along exploration iterations. The shaded areas denote a 95% confidence interval. (d) The number of trials of each method that violates the stabilizing constraints at least once, i.e., having larger sideslip angles than 0.3 rad, during every 50 iterations.
the robots. The results suggest that our framework has the potential to be applied to other robot platforms with continuous state and action space, actively exploring training data, and safely controlling the robot.

## V Conclusion

In this paper, we presented a unified model-based reinforcement learning framework with dynamics learning that bridges active exploration and uncertainty-aware deployment in the robotics control domain. The dynamics is learned using a fully parallelized probabilistic ensemble neural network that is sensitive to uncertainty. For active exploration, the epistemic uncertainty can be quantified by measuring the ensemble disagreement via Jensen-Renyi Divergence. Two opposing task objectives for exploration and deployment can be optimized by the state-of-the-art sampling-based MPC. Our extensive experiments demonstrate that the unified framework can be applied to both autonomous vehicles and wheeled robots. Our framework shows promising results for both exploration and deployment, in that the robot efficiently collects useful training data samples that are essential to achieving the task, and successfully avoids uncertain regions by imposing an uncertainty penalty. We hope that this work will serve as a stepping stone towards more general learning-based robotic applications under uncertainty.

## VI Appendix

### _Additional Details for Learned Dynamics_

To perform an ablation study on the neural network dynamics, we collect a human-controlled driving dataset. We build a race track in the IPG CarMaker simulator, modeled after a kart circuit located in Kirchlengern, Germany [43]. The length of the track is 1016 m and it has two moderate curves and four sharp curves. The overall structure of the track is similar to that in Fig. 6c. The driving data collected on the track consists of 1) zig-zag driving at low speeds, 2) high-speed driving, and 3) sliding maneuvers, in both clockwise and counter-clockwise directions. We collected a total of 70 minutes of data at a rate of 10 Hz. We split the data into 70\(\%\) for training and 30\(\%\) for testing after randomizing the data to remove temporal correlations.

We evaluate the test performance of the compared models with respect to each state variable in \(\mathbf{x}=(v_{x},v_{y},r)^{\top}\), which are the longitudinal velocity \(v_{x}\), the lateral velocity \(v_{y}\), and the yaw rate \(r\). We use the Root Mean Square Error (RMSE) as the performance metric. We also display the average test RMSE for the convenience of comparison.

**Model Comparison.** First, we compare the pure MLP model and the Recurrent Neural Network (RNN) model. In this experiment, we use a deterministic model without ensemble and use L2 loss function. The other parameters of the MLP model follow those mentioned in Section IV. In RNN models, we replace the first layer of the MLP model as GRU [47] and LSTM [46], respectively, with the same hidden layer size as the MLP model. The best test performances of each model during 1000 epochs are shown in Table III. The RNN models show slightly better performances than the MLP model. However, we choose MLP to benefit from the parallelizable structure of the ensemble MLP model.

**History Comparison.** In this experiment, we use PENN model as described in Section III-D. We evaluate the perfor

Fig. 5: Visualization of uncertainty-aware navigation results on the vehicle simulator. We display the rotational impacts exerted on the vehicle onto the trajectories. The maximum value of rotational impact during a three-second window is used for visual clarity.
mances of the PENNs with different lengths of state-action history \(H\). The results are shown in Table IV. The lengths of 3, 4, and 5 produce the best results among the compared models. When the history length exceeds 5, the prediction performance generally decreases. In accordance with prior literature employing the same strategy [44, 45, 54, 61], we determine the history length to be 4.

### _Sim-to-Real Transfer_

We successfully transferred our algorithm to real-world settings for uncertainty-aware deployment tasks. We integrated our algorithm with global path planning and online traversability map generation using a LiDAR sensor. These experiments were conducted on our off-road testbeds. The experimental results can be found on our project page: [https://taekyung.me/rss2023-bridging](https://taekyung.me/rss2023-bridging).

## VII Acknowledgment

This work was supported by the Korean Government (2023).
