# Object Manipulation in Marine Environments using Reinforcement Learning

Ahmed Nader  Muhayy Ud Din  Mughni Irfan  Irfan Hussain  Khalifa University
Center for Autonomous Robotic Systems - Khalifa University, UAE.

###### Abstract

Performing intervention tasks in the maritime domain is crucial for safety and
operational efficiency. The unpredictable and dynamic marine environment makes
the intervention tasks such as object manipulation extremely challenging. This
study proposes a robust solution for object manipulation from a dock in the
presence of disturbances caused by sea waves. To tackle this challenging
problem, we apply a deep reinforcement learning (DRL) based algorithm called
Soft. Actor-Critic (SAC). SAC employs an actor-critic framework; the actors
learn a policy that minimizes an objective function while the critic evaluates
the learned policy and provides feedback to guide the actor-learning process.
We trained the agent using the PyBullet dynamic simulator and tested it in a
realistic simulation environment called MBZIRC maritime simulator. This
simulator allows the simulation of different wave conditions according to the
World Meteorological Organization (WMO) sea state code. Simulation results
demonstrate a high success rate in retrieving the objects from the dock. The
trained agent achieved an 80 percent success rate when applied in the
simulation environment in the presence of waves characterized by sea state 2,
according to the WMO sea state code.

###### keywords:

Marine robotics, Dynamic grasping, Reinforcement learning, Maritime
environment, waves disturbances.

††thanks: Sponsor and financial support anonymous.

, , , and

##  1 Introdcution

Marine robotics has gained significant importance in recent years, where
autonomous systems are increasingly used as practical solutions to
environmental monitoring and maritime security challenges such as illegal
fishing, piracy, smuggling, and human trafficking Ahmed et al. (2023). To
effectively address these issues, marine robots, especially unmanned surface
vessels (USVs), must perform tasks such as autonomous surveillance, tracking,
and intervention-based activities involving manipulation Sun et al. (2021) Din
et al. (2023).

By equipping USVs with manipulators, these vessels can perform a wide range of
manipulation tasks autonomously; for instance, in maritime security, these
manipulation tasks allow USVs to actively respond to incidents, such as
assisting in the interception of suspicious vessels and retrieving contraband
Real-Arce et al. (2015); Lin et al. (2023); Trischitta et al. (2012); Jakob et
al. (2011). Similarly, in port logistics and maritime transportation Yang et
al. (2023), USVs with manipulators can assist in transporting cargo between
docks and ships within port terminals, optimizing loading and unloading
processes and reducing turnaround times for vessels, an integral part of this
transportation task is the successful retrieval of objects from the dock.
CORDIS (2013)

This paper addresses the challenge, where a USV equipped with a manipulator
retrieves objects from the dock in the presence of disturbances due to sea
waves. The USV will experience dynamic and unpredictable movements. These
disruptions will significantly increase the complexity of grasping tasks for
the manipulator. The challenges posed by wave-induced disturbances
significantly impact tasks involving robotic arms on the water surface.
Current research primarily focuses on stabilizing the end-effector pose of
robot arms mounted on ships in the presence of ocean waves and currents, with
applications such as stabilizing a UAV landing platform Xu et al. (2021), not
actual manipulation tasks. Woolfrey et al. (2021) and From et al. (2011) have
demonstrated this using Model Predictive Control (MPC). This approach can be
extended to our case; however, a significant drawback is its dependency on
obtaining an accurate plant model, which is particularly challenging in
maritime environments.

Figure 1: Training and testing environments for object manipulation. Fig-A:
Pybullet training environment for object grasping. Fig-B: shows the testing
environment.

We propose applying deep reinforcement learning (DRL) to accomplish the
manipulation task, as it offers several benefits for grasping in dynamic
environments. Unlike classical control techniques, DRL methods do not depend
on an explicit environment model. By learning directly from interactions, DRL
eliminates the need for an accurate environmental model and only requires a
training environment. Additionally, DRL provides online and closed-loop
control strategies, which are highly effective in resisting disturbances.
These characteristics make DRL well-suited for tasks in dynamic environments,
allowing for adaptive and robust grasping performance under uncertainty.

Training a DRL agent in a dynamic environment is highly sample inefficient,
often necessitating millions of steps to converge Amarjyoti (2017). However,
our findings demonstrate that an agent trained in a static environment can
perform satisfactorily even when facing wave-induced disturbances. This
suggests that dynamic training may be bypassed in our case. The trained agent
was tested in a realistic simulation environment, which included various wave
scenarios mimicking real-world conditions according to the WMO Sea state code
NOAA NationalCenters for environmental Information (2014).

In summary, the key contributions of this work are:

  1. 1.

Investigate the challenge of retrieving objects from the dock, subject to wave
disturbances, using a manipulator-equipped USV, as Previous research primarily
concentrated on stabilizing the end effector pose of the robot arm in the
presence of waves, neglecting actual manipulation tasks.

  2. 2.

Experiment with DRL to carry out the retrieving task. Our findings reveal that
effective results can be achieved by training in a static environment,
bypassing the need for explicitly training with wave disturbance signals

  3. 3.

Develop realistic test simulations for the retriving task across diverse sea
conditions, following the WMO Sea State code, and conduct an in-depth
comparison and analysis of the results.

##  2 Related Work

Object grasping and manipulation are crucial in robotics as they enable robots
to interact and modify their environment, making them valuable for a wide
range of applications Rosell et al. (2019). Effective object manipulation
allows robots to perform complex tasks such as picking in cluttered
environments, packaging, and handling delicate items with precision.
Furthermore, advancements in object manipulation are essential for developing
autonomous systems that can adapt to dynamic and unstructured environments.

Manipulation in the marine environment using manipulators involves specialized
robotic arms or remotely operated vehicles (ROVs) to precisely handle objects
such as underwater equipment, subsea structures, and cargo. Current research
about object manipulation using manipulators in marine environments is focused
on objects either floating or underwater. For instance, in Peñalver et al.
(2015); Du et al. (2023) vision-based grasping systems are employed that track
and grasp objects. Some studies such as Carlucho et al. (2020) used RL
paradigm for controlling manipulators attached to Autonomous Underwater
Vehicles in underwater environments. However, these applications are confined
to scenarios with minimal relative disturbances between the manipulator and
the objects.

Deep Reinforcement Learning has emerged as a promising approach for robot
grasping due to its ability to handle complex and dynamic environments. It
demonstrates potential in controlling robots for grasping both rigid
Kalashnikov et al. (2018) and deformable objects Jangir et al. (2020). For
instance, Buchler et al. (2022) demonstrated impressive results with a custom-
made high-response manipulator, showcasing robotic table tennis skills learned
from scratch. However, a significant challenge associated with DRL is its
sample inefficiency. Training an agent often requires millions of steps in
static environments Han et al. (2023), and the demands are even higher in
dynamic settings. To address this issue, Fang et al. (2019) proposed ”Dher:
Hindsight experience replay for dynamic goals”. This method utilizes failure
episodes to generate artificial success trajectories by treating the final
state as the desired goal. While effective, it currently achieves only dynamic
reaching in simulations. Another strategy to improve sample efficiency
involves transferring a trained agent from simulation to the real world
(Sim2Real). Chen and Lu (2021) pioneered this approach, demonstrating the
ability of a deep learning agent to hold a moving object in the real world.
Their work trained a Baxter robot to grasp a moving object in simulation
before transferring it to the real world, achieving a 7/30 success rate in
straight-line scenarios. Due to the inherent complexity of dynamic grasping,
direct training in dynamic environments often suffers from low success rates
and extended training times. Wu et al. (2022) tackled this challenge using
adversarial reinforcement learning. They employed two separate agents - one
for the end-effector and one for the object - where the object agent gradually
increased its movement complexity. This method achieved a 3/5 success rate for
grasping a mustard bottle. An innovative approach introduced in Xu et al.
(2023) avoids training a dynamic RL agent directly. Instead, they train an
agent to grasp a static object and then enhance its ability with a trajectory
prediction module. This approach has the potential to achieve both high sample
efficiency and good performance in dynamic environments. Both Xu et al. (2023)
and Wu et al. (2022) use Soft Actor Critic (SAC) which is the current state of
the art in DRL algorithms, A central feature of SAC is entropy regularization.
The policy is trained to maximize a trade-off between expected return and
entropy. This connection to the exploration-exploitation trade-off helps
accelerate learning and prevents premature convergence to suboptimal
solutions. Recent advancements explore using meta-parameters to adapt grasping
policies to different scenarios. These meta-parameters control various aspects
of the grasping pipeline, such as the object pose predictor’s look-ahead time
and the motion planner’s time budget. For instance, Jia et al. (2023)
demonstrates a meta-controller that learns to dynamically adjust these
parameters based on the current scene, leading to improved grasping success
rates and reduced grasping time while also managing to grasp objects in the
presence of obstacles.

This work presents a unique grasping challenge: using a reinforcement learning
(RL) agent to control a robot arm that must grasp stationary objects on the
dock while buffeted by ocean wave disturbances. This scenario departs from
research mentioned above that focused mainly on grasping of moving objects
with a static robot arm. Our approach requires the RL agent to not only
account for the object’s pose and gripper dynamics but also to compensate for
the unpredictable movements of the robot arm caused by the waves. This
presents a more complex and realistic manipulation task that pushes the
boundaries of current RL grasping techniques.

##  3 Methodology

###  3.1 Task Definition

A robotic arm is mounted on an unmanned surface vessel (USV) that will
encounter dynamic and unpredictable movements caused by waves and currents.
These disturbances significantly increase the complexity of grasping tasks for
the robotic arm. The primary objective of this study is to develop a
reinforcement learning (RL) agent capable of controlling the robotic arm to
autonomously retrieve objects from a docking station despite the external
disturbances from the USV’s movements. The RL agent must adapt to and respond
to these disturbances to grasp and successfully retrieve the objects.

To simplify the task, we make the following assumptions and constraints:

  * •

Complete knowledge of the gripper and object position and orientation in the
world at all times.

  * •

Only top grasps are considered.

Training the RL agent in a wave-induced dynamic environment is inefficient due
to its randomness, requiring extensive training steps. Instead, we propose
training the agent in a static environment to grasp randomly placed objects.
Here, the agent learns to quickly close the distance between the gripper and
the object to maximize rewards. If wave disturbances are slow and periodic,
the agent’s rate of closing the distance should outpace the rate of these
disturbances, leading to successful grasps. We hypothesize that an agent
trained in a static environment will perform well in minor wave disturbances.
We will train the agent statically and test it in a dynamic environment that
simulates waves, demonstrating its effectiveness in our results and discussion
sections.

###  3.2 Reinforcement Learning Agent Training

In RL-based approaches, an agent undergoes a learning process to make
decisions by actively engaging with its environment. The agent performs
actions within the environment, gains feedback through rewards or penalties,
and then gradually adjusts behavior to maximize its cumulative reward. The
agent perceives the environment through observations, where each observation
represents the current state of the environment; the observation space
represents all possible states that the environment can assume, while the
action space comprises all feasible actions that the agent can undertake.
Central to RL is the reward function, a function that maps the observation-
action pair to a scalar value that describes how desirable the action taken is
in light of the given observation, thereby guiding the agent’s decision-making
process; the policy is the function or the strategy that maps the agent’s
current observation to actions, the agent goal at the end is to learn a policy
that will maximize its cumulative reward. .

The RL agent will learn to control the gripper’s position and orientation to
pick up a randomly placed cube on a plane and lift it to a specified height.
To ensure only top grasps, the gripper has a fixed orientation with respect to
the cube in two degrees of freedom (pitch and roll). Therefore, the agent
controls the gripper’s position p={x,y,z} and rotation along the z-axis (yaw
angle). For the training purpose, we created a simulation environment using
Pybullet as shown in Fig. 1. The simulation environment’s structure and
functionality are consistent with the interface provided by OpenAI Gym,
allowing for the utilization of established reinforcement learning algorithms.
The RL algorithm that we selected is the Soft Actor-Critic (SAC) algorithm
introduced by Haarnoja et al. (2018)

###  3.3 Soft Actor-Critic (SAC) algorithm

Soft Actor-Critic (SAC) is a state-of-the-art model-free deep reinforcement
learning algorithm that leverages deep neural network for solving problems
with continuous action spaces. This off-policy algorithm incorporates an
actor-critic framework, shown in Fig. 2; the actor learns a stochastic policy
that determines actions based on the current state. This stochasticity allows
SAC to handle uncertainties and exploration effectively. The critic evaluates
this policy and provides feedback to guide the actor toward an optimal path.
The goal is for the actor to learn a policy that optimizes an objective
function through training. One of the main aspects that differentiates SAC
from the other actor-critic algorithms is that it tries to maximize the
entropy and reward in the objective function. Therefore, the algorithm
considers not only the expected rewards but also the entropy of the policy;
this encourages exploration without sacrificing the pursuit of high rewards
during training; equation (1) describes the objective function of the SAC
algorithm.

Figure 2: The actor-critic framework displaying the actor and critic neural
networks and the environment, the actor encapsulates the learned policy while
the citric evaluate this actor-learned policy and provide feedback to guide
the actor toward an optimal path.

$\begin{gathered}\pi=\arg\max_{\pi}\underset{\tau\sim\pi}{E}\left[\sum_{t=0}^{\infty}\gamma^{t}\left(R\left(s_{t},a_{t},s_{t+1}\right)+\right.\right.\\\
\left.\left.\alpha H\left(\pi\left(\cdot\mid
s_{t}\right)\right)\right)\right]\end{gathered}$ (1)

$\pi$ is the policy that the actor will learn, $\gamma$ is the reward discount
factor, $\tau$ is the trajectory, $R$ is the reward function, $s_{t}$ is the
observation at time $t$, $s_{t+1}$ is the observations at time $t+1$, $a_{t}$
is the actions at time $t$, $H\left(\pi\left(\cdot\mid s_{t}\right)\right)$ is
the policy’s entropy at $s_{t}$, $\alpha$ is the trade-off coefficient which
controls the trade-off between the exploration and exploitation, the selection
of the SAC algorithm over other RL algorithms for our case is driven by
several advantageous characteristics:

  1. 1.

SAC is a model-free algorithm that doesn’t require an explicit model of the
environment dynamics.

  2. 2.

The excellent exploration capabilities of SAC are desirable in our continuous
observation and action space.

  3. 3.

SAC exhibits robustness to hyper-parameters such as learning rate and reward
discount factor, making it less sensitive compared to other model-free
algorithms

The critic network is a Q value network, with an input layer of size 40, 3
hidden layers of size 256, and an output layer of size 1. The input to the
critic is the observations and actions, and the output is the expected Q
value. The Q value represents the predicted cumulative reward an agent can
obtain by taking a particular action in a given state, this network is updated
to minimize the error between its predictions and the observed reward. the
actor-network has an input layer of size 30, which the size of the
observation, and 28 hidden layers with a size of 256, and an output layer of
size 5 repressing the size of the actions, the actor aims to maximize the
Q-value according to the critic, after training within the SAC framework, the
actor encapsulates the learned policy. The actor then serves as the agent
during testing.

The reward function $R$, action space $a$ and observation space $s$ that we
used, are all defined below :

####  3.3.1 Action Space $a$

As mentioned earlier, the agent controls over the gripper’s x, y, and z
positions and the yaw angle. Additionally, the agent can adjust the gripper’s
joint angle to open and close it. The action space is continuous and is
described in equation (2).

$a_{t}=\left\\{\Delta x,\Delta y,\Delta z,\Delta\psi_{g},g_{state}\right\\}$
(2)

all the values in the action space are normalized to [-1,1]. $\Delta x$,
$\Delta y$, $\Delta z$, and $\Delta\psi_{g}$ are small changes in the gripper
position and orientation with respect to the world, and the gripper control
involves scaling them by a factor $\beta$ which is chosen to be 0.5 .
$g_{state}$ denotes the state of the gripper, with”-1” indicating that it is
fully closed, and ”1” indicating that it is fully open, $a_{t}$ represent the
action taken at time $t$ by the agent.

the new gripper’s position and orientation at time $t+1$ will be calculated
based on the action taken at time $t$ as follows:

$x_{t+1}=x_{t}+\Delta x*\beta$ (3) $y_{t+1}=x_{t}+\Delta y*\beta$ (4)
$z_{t+1}=x_{t}+\Delta z*\beta$ (5)
$\psi_{g(t+1)}=\psi_{g(t)}+\Delta\psi_{g}*\beta$ (6)

####  3.3.2 Observation Space

The observation space is described as follows:

$s_{t}=\left\\{p^{cube},p^{g},D_{cube_{-}g},g_{state},\psi_{g},\psi_{cube}\right\\}$
(7)

where, the variables $p^{cube}$ and $p^{g}$ represent the position of the cube
and the gripper in the world coordinate, $g_{state}$ is the state of the
gripper as in equation (2), and $D_{cube_{-}g}$ is the distance between the
gripper and the cube. $\psi_{g}$ and $\psi_{cube}$ denote the yaw angles of
the gripper and the cube, respectively. These angles signify the rotation
about the Z-axis in the world coordinate system.

$s_{t}$ represent the observation of the environment at time $t$, We stack
three observations from consecutive time steps and obtain the observation that
will be used during training and testing, therefore the observation at time
$t$ becomes:

$s_{t}^{stacked}=\left\\{s_{t},s_{t-1},s_{t-2}\right\\}$ (8)

this will give the agent access to information from different time steps and
will allow it to capture temporal dependencies such as velocity and
acceleration, and then learn the relation between the actions and such
dependencies implicitly.

####  3.3.3 Reward Function Design

We decomposed the task into three distinct sub-tasks: reaching, grasping, and
lifting. Successful completion of the task involves the accomplishment of each
sub-task. Therefore, We have designed a dedicated reward function for each
sub-task.

For reaching, we have two reward functions, one related to the position and
one related to the orientation. The position reward is calculated as follows :

$1-\tanh\left(1.66D_{cube_{-}g}\right)$ (9)

the orientation reward is :

$1-\tanh abs\left(\psi_{cube}-\psi_{g}\right)$ (10)

In the grasping stage, the agent receives a reward of 0.5 when both fingers of
the gripper touch the cube. For the lifting stage, a reward of 2 is granted
when both fingers touch the cube and the cube is lifted by 1 cm from the
ground. To ensure the successful completion of lifting, the cube is to be
lifted at least 20 cm off the ground, and to achieve that an additional reward
of 5 is given when the cube is grasped and at least 20 cm off ground. We
employ a dense reward scheme to enable the agent to learn rapidly. At each
time step, the agent’s total reward is the sum of these individual rewards.

###  3.4 Testing Environment

To evaluate the performance of the developed agent in achieving its primary
objective of retrieving the objects from the docking station by controlling
the robotic arm mounted on the Unmanned Surface Vehicle (USV), we employed the
MBZIRC simulator developed by Open Robotics (2022) . It is an open-source
marine simulator developed for the Mohamed Bin Zayed International Robotics
Challenge (MBZIRC). It is developed using C++, Python, and ROS2 Galactic which
is developed by Macenski et al. (2022) and used for communication among the
components. The simulator uses Gazebo Ignition developed by Open Robotics
(2021) for hydro-static and hydrodynamic simulations. The testing setup is
illustrated in Fig. 3, which includes the USV equipped with the Oberon7 arm,
featuring the same gripper used during training. Additionally, the cube on the
dock is the same size as the one used in the training phase.

Figure 3: Testing setup scene featuring an Unmanned Surface Vehicle (USV)
equipped with the Oberon7 arm and its gripper, and the object to be picked.

The MBZIRC simulator allows the simulation of various sea states. A sea state
characterizes the condition of the water surface, encompassing wave height,
period, and current speed. These parameters are based on statistical data
collected from real-world seas. This feature enables us to assess the agent’s
performance in an environment resembling real-world scenarios, providing a
robust evaluation of its capabilities.

To enable the agent to control the robotic arm within the simulator, we
recreate the observation space on which the agent was trained using
information obtained directly from the simulation. We didn’t implement any
sensors in the simulation to get this information, as we can access the ground
truth information directly using ROS topics, we then provide these
observations to the agent, allowing it to generate the necessary actions. The
changes in position and orientation obtained from the actions are scaled by
the factor previously used during agent training. These scaled actions
represent the change in the gripper pose. It is added to the current gripper
pose, yielding the next gripper pose with respect to the world as shown in
equations 3, 4, 5 and 6, since we’re considering top grasps we need to ensure
that the gripper is always pointing towards the ground, therefore the roll and
pitch angles with respect to the world are always chosen to be $\phi=0$ and
$\theta=0.5*\pi$ respectively.

The next obtained pose is then input into an inverse kinematics solver. This
solver works on an optimization problem to determine the joint angles that
best achieve the specified pose. This calculation is performed based on the
current pose of the robot’s base.

Additionally, based on the gripper state obtained from the action, we
determine the required gripper joint angle, The resulting joint angles of the
arm and the gripper are then sent to the simulator via ROS2 messages as joint
angle commands, A PID controller executes these commands within the
simulation, allowing the arm to respond and move accordingly, Fig. 4 depicts
the overview of this process.

Figure 4:  The figure depicts the dynamic interaction between the agent, the
simulator, and the inverse kinematics solver at each time step, highlighting
the process and sequence of command exchanges.

##  4 Results and Discussion

###  4.1 Training Results

As outlined in Section 2, we employed the soft actor-critic (SAC)
reinforcement learning algorithm for training. In our implementation, we used
a variation that enables the automatic tuning of the alpha parameters
throughout the training process. This adaptive adjustment optimizes the trade-
off between exploration and exploitation. Table 1 provides a detailed overview
of the hyper-parameters utilized during the training phase.The hyper-
parameters used were similar to the ones used in Zhan et al. (2021).

Table 1: Hyper-parameters of the SAC algorithm. hyper-parameter | value
---|---
$\gamma$ | 0.98
learning rate | 0.0001
initial $\alpha$ | 0.5

Each training episode consists of 100 time steps, and we employed a dense
reward scheme, meaning that the reward is computed at each time step. As shown
in Fig. 5, the training process required approximately 10,000 episodes to
converge, yielding an agent with good performance, as evidenced by the smooth
reward, which is close to the maximum reward. The task is considered
successful if, by the conclusion of episode, the agent has successfully
grasped and lifted the cube at least 20 cm off the ground. Our trained agent
achieved this criterion 93 percent of the time, demonstrating satisfactory
performance in the training environment.

Figure 5: The reward improvement during training, The original reward is
represented by the grey line, whereas the smoothed reward, achieved through a
window of size 5, is illustrated by the blue line.

###  4.2 Testing Results

By employing the MBZIRC simulator, we established the testing environment
detailed in Section 3 and illustrated in Fig. 3. This environment is utilized
to assess our trained agent’s capability to effectively grasp and retrieve
objects from the dock station under wave disturbances. We conducted tests
under three different sea states classified as 0, 1, and 2, following the
codes designated by The World Meteorological Organization, which can be found
in NOAA NationalCenters for environmental Information (2014). A comprehensive
description of these sea states is provided in table 2. During testing in sea
state one, we employed a wave amplitude of 0.1 meters, representing the
maximum amplitude for this sea state. The corresponding wave period was set to
5 seconds. In sea state two, we utilized a wave amplitude of 0.5 meters, the
maximum for this sea state, and maintained a consistent wave period of 5
seconds. This uniformity ensures consistency and facilitates the comparisons
of the agent performance in different sea states.

Table 2: WMO Sea State Code. WMO Sea State Code | Wave amplitude(m) | Characteristics
---|---|---
0 | 0 | Calm(glassy)
1 | 0 to 0.1 | Calm(rippled)
2 | 0.1 to 0.5 | Smooth (wavelets)

Similar to the training environment, successful completion of the task in the
testing environment is defined by the agent’s ability to grasp the cube from
the dock and lift it a minimum of 20 cm within 30 seconds of simulation time.
Unlike during training, we extend the time window in this scenario,
acknowledging that the agent faces additional challenges in overcoming wave
disturbances. For each sea state, we did 15 tests and calculated the success
rate, which is shown in table 3.

Table 3: Agent success rate under different sea states. Sea State Code | Agent success rate
---|---
0 | 0.933
1 | 0.87
2 | 0.8

In sea state 0, characterized by the absence of wave disturbances, the agent
achieves a success rate of 0.933, which is very similar to the success rate
observed in the training environment, which was 0.93. This alignment is
expected since no disturbances are applied to the gripper in the training and
sea state 0 testing environments. in general the results indicate that the
agent performs well even when faced with wave disturbances despite not being
trained specifically for that purpose. and this behavior can be attributed to
the following :

  1. 1.

The wave disturbances primarily affect the gripper, not the object which
remains stationary on the dock. Given that, the object is considered static
with respect to the world, providing observations akin to the agent’s
training, and since RL control operates in an online and closed-loop manner,
the agent continuously receives feedback from the environment and can
immediately adjust its actions. Consequently, the agent persistently drives
the gripper to approach the object despite disturbances attempting to push it
away.

  2. 2.

The agent can move the gripper quickly to reach the object. Reaching is a
necessary part of achieving the grasping task; therefore, the agent learned
during training how to control the gripper to efficiently close the distance
to the object as fast as possible to maximize its reward; the agent’s ability
to move quickly resulted from it implicitly learning representations of
velocity and acceleration, and associating them with specific actions to
optimize the reward, The agent’s capacity to comprehend velocity and
acceleration without them being explicitly shown in the observations is
attributed to our practice of stacking observations from multiple time steps
and use it in training.

  3. 3.

The slow and periodic nature of the sea wave disturbances causes relatively
slow fluctuations in the distance between the object and the gripper. When the
waves decrease this distance, it is advantageous for the agent since it brings
the gripper closer to the object. Therefore, the waves won’t always work
against the agent.

As a result of these factors mentioned above, the agent effectively outpaces
the slow and periodic disturbances of the waves, gradually minimizing the
distance to the object over time, ultimately leading to a successful grasp.
This is evident in Fig. 6 , where the peaks in the distance to the object
gradually diminish over time. This reduction signifies that the agent is
minimizing the distance to the object. Eventually, we observe the distance
stabilizes, indicating the agent’s successful grasp of the object. A visual
representation of the arm movement to grasp the object is evident in Fig. 7,
where we can see screenshots of the arm movement to grasp the object; we can
see in screenshot four that the arm is almost grasping the object, but in
screenshot five, it gets pushed away by sea waves. However, it manages to
resist the disturbances and eventually successfully grasps the object.

Figure 6: The change in the distance from the gripper to the object versus
time. It shows how the agent approaches the object despite the wave-induced
disturbances.

In sea state one, the success rate is similar to zero state due to low wave
amplitude and long periods, resulting in small and slow disturbances. In sea
state two, with increased amplitude but the same period, the agent’s success
rate drops significantly due to the increased speed of disturbances, making it
more difficult for the agent to resist and minimize the distance within the
allocated time to grasp the object successfully.

Figure 7:  Sequence of snapshots of the execution of the object retrieval
task. In screenshot four, the arm almost grasped the object, but in screenshot
five, the arm gets pushed away due to the waves; however, eventually, it
manages to resist the waves and grasp the object.

##  5 Conclusions

In this work, we addressed the challenging issue of retrieving objects from a
dock using a USV equipped with a robotic arm. we trained a reinforcement
learning agent based on the actor-critic framework and performed testing in a
realistic simulation using wave characteristics based on the WMO Sea state
code, which showed that when training an RL agent to grasp randomly placed
objects in static environment it acquires the ability to resist disturbance
and can effectively handle wave-induced disturbances without specific dynamic
training.

Our proposed method solely focuses on top grasps and accounts for wave-induced
disturbances only, overlooking drifting. Drifting refers to the USV moving due
to currents; unlike waves, drifting presents non-periodic disturbances which
pose a challenge for the agent to address. Moving forward, we aim to enhance
the agent’s capabilities to incorporate more complex scenarios such as mobile
manipulation for marine intervention. Furthermore, we will test the developed
approach in a real sea environment with an underwater manipulator.

This work is supported by the Khalifa University of Science and Technology
under Award No. MBZIRC-8434000194, CIRA-2021-085, FSU-2021-019,
RC1-2018-KUCARS.

## References

  * Ahmed et al. (2023) Ahmed, M., Bakht, A.B., Hassan, T., Akram, W., Humais, A., Seneviratne, L., He, S., Lin, D., and Hussain, I. (2023).  Vision-based autonomous navigation for unmanned surface vessel in extreme marine conditions.  In _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 7097–7103.  10.1109/IROS55552.2023.10341867.
  * Amarjyoti (2017) Amarjyoti, S. (2017).  Deep reinforcement learning for robotic manipulation-the state of the art.  _arXiv preprint arXiv:1701.08878_.
  * Buchler et al. (2022) Buchler, D., Guist, S., Calandra, R., Berenz, V., Scholkopf, B., and Peters, J. (2022).  Learning to play table tennis from scratch using muscular robots.  _IEEE Transactions on Robotics_ , 38(4), 1–11.  10.1109/TRO.2022.3176207.  URL https://doi.org/10.1109/TRO.2022.3176207.
  * Carlucho et al. (2020) Carlucho, I., Paula, M.D., Acosta, G.G., and Barbalata, C. (2020).  A reinforcement learning control approach for underwater manipulation under position and torque constraints.  _Global Oceans 2020: Singapore – U.S. Gulf Coast_ , 1–7.  URL https://api.semanticscholar.org/CorpusID:227162199.
  * Chen and Lu (2021) Chen, P. and Lu, W. (2021).  Deep reinforcement learning based moving object grasping.  _Information Sciences_ , 565, 62–76.  10.1016/j.ins.2021.01.077.
  * CORDIS (2013) CORDIS, c. (2013).  Final report summary - cargo-ants (cargo handling by automated next generation transportation systems for ports and terminals): Fp7: Cordis: European commission.  URL https://cordis.europa.eu/project/id/605598/reporting/de.
  * Din et al. (2023) Din, M.U., Humais, A., Akram, W., Alblooshi, M., Saoud, L.S., Alblooshi, A., Seneviratne, L., and Hussain, I. (2023).  Marine $\mathcal{X}$: Design and implementation of unmanned surface vessel for vision guided navigation.  In _2023 21st International Conference on Advanced Robotics (ICAR)_ , 226–231.  10.1109/ICAR58858.2023.10406475.
  * Du et al. (2023) Du, Z., Negenborn, R.R., and Reppa, V. (2023).  Review of floating object manipulation by autonomous multi-vessel systems.  _Annual Reviews in Control_ , 55, 255–278.  https://doi.org/10.1016/j.arcontrol.2022.10.003.  URL https://www.sciencedirect.com/science/article/pii/S1367578822001365.
  * Fang et al. (2019) Fang, M., Zhou, C., Shi, B., Gong, B., Xu, J., and Zhang, T. (2019).  Dher: Hindsight experience replay for dynamic goals.  In _7th International Conference on Learning Representations, ICLR 2019_ , 1–12.  URL https://arxiv.org/abs/1812.05905.
  * From et al. (2011) From, P.J., Gravdahl, J.T., Lillehagen, T., and Abbeel, P. (2011).  Motion planning and control of robotic manipulators on seaborne platforms.  _Control engineering practice_ , 19(8), 809–819.
  * Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018).  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In J. Dy and A. Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , 1861–1870. PMLR.  URL https://proceedings.mlr.press/v80/haarnoja18b.html.
  * Han et al. (2023) Han, D., Mulyana, B., Stankovic, V., and Cheng, S. (2023).  A survey on deep reinforcement learning algorithms for robotic manipulation.  _Sensors_ , 23(7).  10.3390/s23073762.  URL https://www.mdpi.com/1424-8220/23/7/3762.
  * Jakob et al. (2011) Jakob, M., Vanĕk, O., and Pĕchouček, M. (2011).  Using agents to improve international maritime transport security.  _IEEE Intelligent Systems_ , 26(1), 90–96.  10.1109/MIS.2011.23.
  * Jangir et al. (2020) Jangir, R., Alenya, G., and Torras, C. (2020).  Dynamic cloth manipulation with deep reinforcement learning.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 4630–4636.  10.1109/ICRA40945.2020.9196659.  URL https://doi.org/10.1109/ICRA40945.2020.9196659.
  * Jia et al. (2023) Jia, Y., Xu, J., Jayaraman, D., and Song, S. (2023).  Dynamic grasping with a learned meta-controller.  URL https://api.semanticscholar.org/CorpusID:256900757.
  * Kalashnikov et al. (2018) Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Levine, S., Tamar, A., Mancuso, M., Hadsell, R., et al. (2018).  Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation.  _Proceedings of the Conference on Robot Learning (CoRL)_ , 2018, 1–23.  URL https://arxiv.org/abs/1806.10293.
  * Lin et al. (2023) Lin, B., Xie, W., Shi, Y., Du, B., Zhang, C., and Zhang, W. (2023).  Robust target interception strategy for a usv with experimental validation.  _IEEE Robotics and Automation Letters_ , 8(11), 7042–7049.  10.1109/LRA.2023.3300235.
  * Macenski et al. (2022) Macenski, S., Foote, T., Gerkey, B., Lalancette, C., and Woodall, W. (2022).  Robot operating system 2: Design, architecture, and uses in the wild.  _Science Robotics_ , 7(66), eabm6074.  10.1126/scirobotics.abm6074.  URL https://www.science.org/doi/abs/10.1126/scirobotics.abm6074.
  * NOAA NationalCenters for environmental Information (2014) NOAA NationalCenters for environmental Information (2014).  About wmo code table 3700.  URL https://www.nodc.noaa.gov/gtspp/document/codetbls/wmocodes/table3700.html.
  * Open Robotics (2021) Open Robotics (2021).  ignition gazebo.  URL https://gazebosim.org/home.
  * Open Robotics (2022) Open Robotics (2022).  Mbzirc maritime grand challenge simulator.  URL https://github.com/osrf/mbzirc?tab=readme-ov-file.
  * Peñalver et al. (2015) Peñalver, A., Pérez, J., Fernández, J., Sales, J., Sanz, P., García, J., Fornas, D., and Marín, R. (2015).  Visually-guided manipulation techniques for robotic autonomous underwater panel interventions.  _Annual Reviews in Control_ , 40, 201–211.  https://doi.org/10.1016/j.arcontrol.2015.09.012.  URL https://www.sciencedirect.com/science/article/pii/S1367578815000516.
  * Real-Arce et al. (2015) Real-Arce, D.A., Barrera, C., Hernández, J., and Llinás, O. (2015).  Ocean surface vehicles for maritime security applications (the perseus project).  In _OCEANS 2015 - Genova_ , 1–4.  10.1109/OCEANS-Genova.2015.7271504.
  * Rosell et al. (2019) Rosell, J., Suárez, R., García, N., and Din, M.U. (2019).  Planning grasping motions for humanoid robots.  _International Journal of Humanoid Robotics_ , 16(06), 1950041.  10.1142/S0219843619500415.  URL https://doi.org/10.1142/S0219843619500415.
  * Sun et al. (2021) Sun, X., Liu, T., Yu, X., and Pang, B. (2021).  Unmanned surface vessel visual object detection under all-weather conditions with optimized feature fusion network in yolov4.  _Journal of Intelligent & Robotic Systems_, 103, 1–16.
  * Trischitta et al. (2012) Trischitta, P.R., Salloum, H., Bunin, B., and Orton, P. (2012).  Mitigating threats of small vessels to maritime security.  In _2012 IEEE Conference on Technologies for Homeland Security (HST)_ , 654–659.  10.1109/THS.2012.6459926.
  * Woolfrey et al. (2021) Woolfrey, J., Lu, W., and Liu, D. (2021).  Predictive end-effector control of manipulators on moving platforms under disturbance.  _IEEE Transactions on Robotics_ , 37(6), 2210–2217.
  * Wu et al. (2022) Wu, T., Zhong, F., Geng, Y., Wang, H., Zhu, Y., Wang, Y., and Chen, X. (2022).  Grasparl: Dynamic grasping via adversarial reinforcement learning.  URL https://arxiv.org/abs/2203.02119.
  * Xu et al. (2023) Xu, B., Hassan, T., and Hussain, I. (2023).  Improving reinforcement learning based moving object grasping with trajectory prediction.  _Intelligent Service Robotics_.  URL https://api.semanticscholar.org/CorpusID:265443444.
  * Xu et al. (2021) Xu, R., Ji, X., Hou, J., Liu, H., and Qian, H. (2021).  A predictive control method for stabilizing a manipulator-based uav landing platform on fluctuating marine surface.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 8625–8632.  10.1109/IROS51168.2021.9636055.
  * Yang et al. (2023) Yang, Y., He, S., and Sun, S. (2023).  Research on the cooperative scheduling of armgs and agvs in a seaandash;rail automated container terminal under the rail-in-port model.  _Journal of Marine Science and Engineering_ , 11(3).  10.3390/jmse11030557.  URL https://www.mdpi.com/2077-1312/11/3/557.
  * Zhan et al. (2021) Zhan, A., Zhao, R., Pinto, L., Abbeel, P., and Laskin, M. (2021).  A framework for efficient robotic manipulation.  In _Deep RL Workshop NeurIPS 2021_.


[orcid=0000-0002-2291-4028] [1]

# Supervised and Semi-supervised Deep Probabilistic Models for Indoor
Positioning Problems

Weizhu Qian E-mail address<EMAIL_ADDRESS>CIAD, Université Bourgogne
Franche-Comté UTBM, 90010, Belfort, France  Mosel LORIA UMR CNRS 7503,
Université de Lorraine, 54506, Vandœuvre-lès-Nancy, France  Fabrice Lauri
Franck Gechter

###### Abstract

WiFi fingerprint-based indoor localization has been a popular research topic
recently. In this work, we propose two novel deep learning-based models, the
convolutional mixture density recurrent neural network and the variational
autoencoder-based semi-supervised learning model. The convolutional mixture
density recurrent neural network is designed for indoor next location
prediction, in which the advantages of convolutional neural networks,
recurrent neural networks and mixture density networks are combined.
Furthermore, since most of real-world WiFi fingerprint data are not labeled,
we devise the variational autoencoder-based model to compute accurate user
location in a semi-supervised learning manner. Finally, in order to evaluate
the proposed models, we conduct the validation experiments on two real-world
datasets. The final results are compared to other existing methods and verify
the effectiveness of our approaches.

###### keywords:

Variational autoencoders Mixture density networks Semi-supervised learning
WiFi fingerprints Indoor positioning

##  1 Introduction

Location-based services (LBS) are essential for applications such as location-
based advertising, outdoor/indoor navigation, social networking, etc. Studying
human mobility is important for developing LBS applications. Meanwhile, with
the help of significant advances of the smartphone technology in recent
decades, smartphone devices are integrated with various built-in sensors, such
as Global Positioning System (GPS) modules, WiFi modules, cellular modules,
etc. Acquiring the data from such kinds of sensors enables researchers to
study human activities. Therefore, as opposed to some other approaches which
are focusing on using base stations or radars, like RADAR [1] and HORUS [2],
we choose to base our work on the data collected from smartphones.

There are several types of smartphone data that can be used for such research.
For instance, GPS equipment can provide the position information with
latitudes and longitudes directly when smartphone users stay outdoors. The
GPS-based localization methods are favored by many researchers [3], [4].
However, this type of methods are not suitable for indoor positioning tasks
because GPS modules do not work well in indoor environment.

In some cases, we have to use other indirect data to interpret the location
information. Since WiFi connections are now widely used, one widespread
approach is based on the detected WiFi fingerprints of smartphone devices. In
this case, the received signal strength indicator (RSSI) values of WiFi access
points (WAPs) scanned by mobile phones are used to compute the locations of
users instead of using latitudes and longitudes directly. As compared to GPS-
based localization methods, WiFi fingerprints-based localization methods can
work in indoor environment as well.

In our work, we attempt to interpret WiFi fingerprints (RSSI values) into
accurate users location (coordinates). However, this task is not easy to
accomplish. Typically, the RSSI values data is composed of vectors whose
elements represent the unique WAP IDs. To provide the good quality of WiFi
connection, modern public buildings usually are equipped with a relatively
large number of WiFi access points. As a result, this leads to the high
dimensionality problem. Another issue is that RSSI values are not always
stable due to the signal-fading effect and multi-path effect [5]. For example,
a virtual WAP presents itself as multiple access points but in fact it only
has one physical access point. In addition, according to our investigation, we
find that an ordinary deep learning regression model with Euclidean distance-
based loss is not powerful enough to overcome such difficulties.

In this paper, we aim at addressing two WiFi fingerprint-based positioning
issues. This first problem is to predict the next time point user location
with current WiFi fingerprints. The second problem is to recognize user
location by semi-supervised learning. Accordingly, we propose two deep
learning-based models to tackle these two problems respectively. The first
proposed model is called the convolutional mixture density recurrent neural
network (CMDRNN) designed to predict user paths. In the CMDRNN model, to
tackle the high dimensionality issue, we deploy a one-dimensional
convolutional neural network as a sub-structure to detect the features of the
input. Moreover, in order to overcome the instability problem of the data, a
mixture density network sub-structure is incorporated into our model for
sampling the final output. Finally, since our task is a time-series prediction
task, we model the state transition via a recurrent neural network sub-
structure. With such unique design, the CMDRNN model is able to predict user
location with WiFi fingerprints.

As we know, labelling data usually is both time-consuming and labor-consuming,
thus most of real-word data in fact is unlabeled. However, even so, we still
want to make as much use of accessible data as possible. To this end, as the
second proposed approach, we propose a deep learning-based model, the
variational autoencoder-based semi-supervised learning model. In this
approach, we assume that the input (RSSI values) and the target (user
location) are related to the same latent distribution. Therefore, in the
unsupervised learning process, we use the variational autoencoder model to
learn the latent distribution of the input whose information is relatively
abundant. Then, in the supervised learning process, the labeled data is used
to train the predictor. In this way, we can exploit more information of the
dataset than other supervised learning methods.

The main contributions of our work are summarized as follows:

  * •

We devise a novel hybrid deep-learning model (CMDRNN) which allows us to
predict the accurate position of smartphone users based on WiFi fingerprints;

  * •

We devise a variational autoencoder-based semi-
supervised learning model for accurate indoor positioning using WiFi
fingerprints;

  * •

We conduct the evaluation experiments on two real-world datasets and compare
our methods with other existing machine learning and deep learning methods.

The reminder of the paper is organized as follows. Section 2 surveys the
related work. Section 3 describes the problems we will solve in this paper. In
Section 4, the two proposed methods are introduced. Section 5 presents the
validation experiments and the results using the real-world datasets. Finally,
we draw some conclusions and discuss about the potential future work in
Section 6.

##  2 Related Work

In literature, researchers have explored various types of machine learning
techniques, both conventional machine learning and deep learning methods, on
location recognition and prediction with WiFi fingerprints data.

###  2.1 Conventional Machine Learning Methods

In the work of [6], authors compared many traditional machine learning
methods, decision trees (DTs), k-nearest neighbors (k-NNs), naive Bayes (NBs),
neural networks (NNs), etc., for classifying buildings, floors and regions. In
[7], authors clustered the 3D coordinates data by K-means and clustered the
RSSI data by the affinity clustering algorithm, respectively. In particular,
in [8], researchers compared $51$ different distance metrics to investigate
the most suitable distance functions for accurate WiFi-based indoor
localization. Some researchers used Gaussian processes (GPs) [9] to model the
relationship between WiFi signal strength and indoor location [10], [11],
[12]. But GPs are not scalable to large datasets because of the expensive
computational cost. In addition, for representing the time series state
transition, due to the complex relationship between WiFi RSSI values and user
coordinates, neither Kalman filter-based approaches [13] nor hidden Markov
model-based approaches [14] is suitable for our tasks .

###  2.2 Deep Learning Methods

Deep learning methods, such as convolutional neural networks (CNNs) [15],
autoencoders (AEs) [16] and recurrent neural networks (RNNs) [17] also have
been utilized in WiFi-based positioning tasks. For instance, [18] and [19]
proposed the CNN-based models for indoor localization. Generally, a buildings
has many different WiFi access points, consequently the RSSI data in many
situations, can be very high dimensional. For this reason, it is reasonable to
reduce the input dimension before carrying out a regression task or a
classification task. Some deep learning-based dimension-reduction methods like
autoencoders can be the appropriate approaches [20], [21], [22]. For example,
in the research of [21], authors used an autoencoder network to reduce the
data dimension, then used a CNN to proceed accurate user positioning. In [20],
[22], authors used autoencoders to reduce the input dimension before using a
multi-layer perceptron (MLP) to classify the buildings and floors.

In terms of time series prediction, in [5], authors compared different types
of recurrent neural networks including the vanilla RNN, the long short-term
memory (LSTM), the gated recurrent unit (GRU) and the bidirectional LSTM for
accurate RSSI indoor localization. They also employed a weighted filter for
both input and output layers to enhance the sequential modeling accuracy.

###  2.3 Limitations of Conventional Neural Networks

As for traditional neural networks (NNs), they can be regarded as
deterministic models once trained, which can be described as follow:

$y=\mathcal{F}(x;w)$ (1)

where $x$ and $y$ are the input and output of the NN, respectively; $F$
represents the neural network structure and $w$ are the weights of the NN.

Accordingly, the training loss (for instance, mean squared errors) of NNs can
be described as follow:

$Loss=\frac{1}{N}\sum_{n=1}^{N}{(\hat{y_{n}}-y_{n})^{2}}$ (2)

where $N$ is the mini batch size and $\hat{y}$ is the target.

In many situations, a NN model is powerful enough to obtain satisfying
results. However, in some cases, for instance, a high non-Gaussian inverse
problem, traditional neural networks will lead to very poor modeling results
[23]. A good solution to this issue is to seek a framework that can calculate
conditional probability distributions between the input and the target.

Mixture density networks (MDNs) solve this problem by using maximum likelihood
estimation (MLE) [24]. In MDNs, the final output are sampled by mixture
distributions rather than computed directly. One advantage of MDNs is that it
can be applied to an estimation situation in which a large variety lies. For
instance, we can incorporate more mixture of Gaussians to a MDN to enhance its
estimating capacity for more complex distributions. However, as a MLE
approach, the MDN also has obvious disadvantages. First, it needs to set some
hyper-parameters properly (i.e., the number of Gaussians for a MDN),
otherwise, it may not be able to provide desirable results. Moreover, MLE can
be biased when samples are small, thus MDNs are not suitable for semi-
supervised learning. In practice, we find that MDNs also suffer from
computational instability when the mixture number is too large.

To alleviate the disadvantages of MDNs, we can resort to maximum a posterior
(MAP)-based models, in which the prior of model parameters and the likelihood
are both considered. MAP has the regularizing effect which can be used to
prevent overfitting. For instance, Bayesian neural networks (BNNs), which
apply Bayesian inference, have been introduced [25]. Unfortunately, in
practice, we find that BNNs are not flexible enough for very complex
distributions like our cases.

To solve this problem, we use another Bayesian deep learning model,
variational autoencoders (VAEs) [26], in the proposed model so as to introduce
a richer prior information. Meanwhile, since variational autoencoders are
unsupervised deep latent generative models, we can use it to learn latent
representation from the unlabeled data and to devise the semi-supervised
learning model.

##  3 Problem Description

In this work, our purpose is to calculate smartphone user location through
using corresponding WiFi fingerprints. As opposed to some previous work, we
attempt to obtain accurate user location, namely, coordinates (either in
meters or longitudes and latitudes) rather than treat this subject as a
classification task (identifying buildings or floors).The main goals of the
two tasks in our work are summarised as follows:

  * •

In the first task, we aim to predict the next indoor location of the users.
The input is the WiFi RSSI values at the current time points and the target is
the coordinates of the users at the next time points. This task can be
regarded as a high dimensional time-series regression task. In the later
section, we will introduce a hybrid deep learning model to solve this problem;

  * •

In the second task, we aim to compute the location of the users. The input is
the WiFi RSSI values and the target is the coordinates of the users. We do not
treat the data as sequences. This task can be regarded as a high dimensional
regression task. In particular, we wan to investigate semi-supervised learning
methods to solve this problem, therefore we split the input data into the
labeled set and unlabeled set. In the later section, we will introduce a semi-
supervised deep learning model to solve this problem.

Table 1: Descriptions for the two tasks Task | Input | Output | Learning method | Purpose
---|---|---|---|---
Task 1 | RSSI values (sequential) | coordinates (sequential) | supervised | location prediction
Task 2 | RSSI values | coordinates | semi-supervised | location recognition

Table 1 describes the differences between the two tasks. To accomplish these
two tasks, we introduce two novel deep learning-based probabilistic models,
the convolutional mixture density recurrent neural network and the variational
autoencoder-based semi-supervised learning model in this paper.

##  4 Proposed Methods

As explained before, we introduce two proposed deep learning-based models in
this section. The first proposed model is called the convolutional mixture
density recurrent neural network (CMDRNN), which is designed to predict paths
with WiFi fingerprints. The second proposed model is the VAE-based model,
which is designed to proceed semi-supervised learning for WiFi-based
positioning.

###  4.1 Convolutional Mixture Density Recurrent Neural Network

####  4.1.1 1D Convolutional Neural Network

In our first task, the input features are composed of the RSSI values of all
the WiFi access points (WAPs) in the buildings, therefore the input dimension
can be very high. Since the features of the WiFi fingerprint data represent
the different WiFi WAP IDs. The adjacent features suggest that they are close
spatially in the real world. Therefore, their RSSI values are more similar
when the users are approaching compared to the WAPs that are remote to them
(it will be illustrated in the WiFi data samples in the experiments). For this
reason, to deal with the high dimensionality problem, we choose to use a
convolutional neural network [15]. CNNs are powerful tools for detecting
features and widely used for tasks such as image processing, natural language
processing and sensor signal processing. In particular, since the input of our
model are RSSI value vectors, we adopt the 1D convolutional neural network to
extract the properties of the high dimensional input.

####  4.1.2 Recurrent Neural Network

Recurrent neural networks (RNNs) [17] are widely used for natural language
processing (NLP), computer vision and other time series prediction task. In
our model, we employ a RNN model to forecast the user location.

The state transitions of RNNs can be expressed as follow:

$h_{t}=\sigma_{h}(W_{h}{x_{t}}+{U_{h}}h_{t-1}+b_{h})$ (3)

where $x_{t}$ is the input, $h_{t}$ is the hidden state, $\sigma_{h}$ is the
activation function, $W_{h}$ are the weights from the input layer to the
hidden layer, $U_{h}$ are the weights in the hidden layers and $b_{h}$ are the
biases.

The output of a conventional RNN can be described as follow:

$y_{t}=\sigma_{y}(W_{y}{h_{t}}+b_{y})$ (4)

where $y_{t}$ is the output, $\sigma_{y}$ is the activation function, $W_{y}$
are the weights and $b_{y}$ are the output biases.

However, in many situations, RNNs may suffer from the long-term dependency
problem during the learning process. To address this issue, the researchers
proposed a variant of RNNs, called the long short-term memory network (LSTM)
[27]. In a LSTM, each unit has three gates, namely, an input gate, an output
gate and a forget gate. These gates regulate the cell states of the LSTM to
avoid the long-term dependency problem. More recently, the researchers
proposed a variant of RNN, called the gated recurrent unit (GRU) [28], which
has the similar accuracy as LSTMs but less computing cost. We will deploy
these three RNN architectures in the proposed model respectively for
comparisons.

####  4.1.3 Mixture Density Network

A traditional neural network with a mean squared error-based loss function, is
optimized by a gradient-descent based method. Generally, such a model can
perform well on the problems that can be described by a deterministic function
$f(x)$, i.e., each input corresponds to an output of one specific value.
However, for some stochastic problems, one input may has more than one
possible values. Generally, this kind of problems are better to be described
via a conditional distribution $p(y|x)$ than a deterministic function
$y=f(x)$. In such cases, traditional neural networks may not work as expected.

To address this issue, we can replace the original loss function with a
conditional function, for a regression task, the Gaussian distribution can be
a proper choice. Moreover, utilizing mixed Gaussian distributions can improve
the representation capacity of the model. Based on this idea, researcher
proposed mixture density networks (MDNs) model [24]. In contrast with
traditional neural network, the outputs of a MDN are the parameters of a set
of mixed Gaussian distributions and the loss function becomes the conditional
probabilities. Therefore, the optimization process is to minimize the negative
log probability. Hence, the loss function can be described as follow:

$p(y|x)=\sum_{k=1}^{K}{\pi_{k}p(y|x;\theta_{k})}$ (5)

where $x$ is the input, $K$ is the total mixture number, $\pi_{k}$ is the
assignment probability for each model, with
$\sum_{k=1}^{K}\pi_{k}=1,(0\leq\pi_{k}\leq 1)$, and $\theta_{k}$ are the
internal parameters of the base distribution; For Gaussians,
$\theta_{k}=\\{\mu_{k},\sigma_{k}\\}$, $\mu_{k}$ are the means and
$\sigma_{k}$ are the variances.

Accordingly, in our proposed model, the original output layer of the RNN, Eq.
(4), is rewritten as:

$\theta_{t}=\sigma_{\theta}(W_{\theta}{h_{t}}+b_{\theta})$ (6)

where $\theta_{t}$ is the output of the RNN sub-model and also the input of
the MDN sub-model, $\sigma_{\theta}$ is the activation function, $W_{\theta}$
are the weights and $b_{\theta}$ are the biases.

After the training process, we can use the neural network along with the mixed
Gaussian distributions to represent the target distribution.

####  4.1.4 Proposed Model

Knowing the merits of the three aforementioned neural networks, we devise a
novel deep neural network architecture, called the convolutional mixture
density recurrent neural network (CMDRNN). In the CMDRNN model, a 1D CNN is
used to capture the features of the high dimensional input, then the state
transitions of the time series data is modeled by a RNN model, and the output
layer composed of mix Gaussian densities is used to sample the final
prediction. With such a structure, we believe that our model is able to
illustrate complex high dimensional time series data. Fig. 1 shows the whole
structure of the CMDRNN model and Algorithm 1 demonstrates the learning
process of the CMDRNN model.

Figure 1: Convolutional mixture density recurrent neural network. Algorithm 1
Algorithm: the CMDRNN model

1:$x_{t}$ (RSSI Values)

2:$y_{t}$ (Coordinates)

3:

4:while e $<$ max epoch do

5: while i $<$ batch num do

6: $h_{0}$ $\leftarrow$ $Conv1d(x_{t})$ $\triangleright$ convolutional
operation

7: $h_{1}$ $\leftarrow$ max pool $h_{0}$

8: $f_{t}$ $\leftarrow$ flatten $h_{1}$

9: $h_{t}$ $\leftarrow$ $\sigma_{h}(W_{h}*f_{t}+U_{h}*y_{t-1}+b_{h})$
$\triangleright$ update hidden states

10: $\theta_{t}$ $\leftarrow$ $\sigma_{y}(W_{y}*h_{t}+b_{y})$ $\triangleright$
compute RNN model output

11: $\theta_{k}$ $\leftarrow$ $\theta_{t}$ $\triangleright$ assign mixture
density parameters

12: minimize loss function: $-p(y_{t}|x_{t};\theta_{k})$

13: end while

14:end while

15:$y_{t}\sim p(y_{t}|x_{t};\theta_{k})$$\triangleright$ sample final output

16:return $y_{t}$

The uniqueness of our method is that, compared with other existing models in
literature, our model adopts a sequential density estimation approach. Thus,
the learning target of the proposed method is a conditional distribution of
the data rather than a common regressor. By doing so, our model can solve the
complex sequential modeling task in this work.

###  4.2 VAE-based Semi-supervised Learning

Now, we tackle the WiFi-fingerprint location recognition task via semi-
supervised learning. To make the semi-supervised learning scheme applicable
for our case, we need to make two important assumptions first:

  * •

Assume that the input $x$ (RSSI values) and the target $y$ (coordinates) are
related to the same latent variable $z$ (related to user position);

  * •

Assume that a statistic $t(x)$ solely of $x$ is enough to be the sufficient
statistic for $z$, which means $t(x)$ captures all the necessary information
for calculating the parameters in the distribution of $z$.

The first assumption describes the relationship among the input $x$, the
output $y$ and the latent variable $z$. The second assumption explains why we
can infer the distribution of the latent variable $z$ with the input $x$,
which can be regarded as an unsupervised learning process. Since in many real-
world cases, available datasets have more information about the input $x$ and
less information about the target $y$, therefore it is more reliable to infer
the latent distribution of $z$ via $p(z|x)$ rather than via $p(z|y)$.

Let $q(z)$ represent the prior distribution of $z$, then we can approximate
$p(z|x)$ via variational inference. Thus, for the unsupervised learning
process, our goal is to minimize the Kullback–Leibler (KL) divergence
$D_{\text{KL}}$.

$D_{KL}\big{(}p(z|x)||q(z)\big{)}=\iint p(z|x)\log\frac{p(z|x)}{q(z)}dzdx$ (7)

According to the chain rule and the assumptions we made, then the generative
procedure can be formulated as follow:

$\displaystyle p(y,z,x)=p(y|z,x)p(z|x)p(x)$ (8)

Accordingly, the predicting procedure (either deterministic or stochastic) can
be described as follow:

$y\sim p(y|z,x)$ (9)

We can implement Eq. (8) and Eq. (9) via an unsupervised learning process and
a supervised learning process, respectively. Hence, our method consists of two
steps:

  * •

The first step (unsupervised learning): we employ a deep generative model to
obtain the latent distribution $p(z|x)$;

  * •

The second step (supervised learning): we employ a MLP structure to predict
the target $y$ using the learnt latent distribution.

####  4.2.1 Unsupervised Learning Procedure

For the unsupervised learning, we adopt a variational autoencoder as the
generative model to learn the latent distribution. Variational autoencoders
(VAEs) [26] are deep latent generative models which adopt variational
inference. Different to conventional autoencoders or other generative models,
the latent representations of VAEs are continuous probabilistic distributions,
which can be used to represent the real user coordinates.

In VAEs, the prior of the latent variable $z$, $q(z)$ can be regarded as a
standard Gaussian distribution:

$z\sim\mathcal{N}(0,\mathbb{I})$ (10)

In order to obtain the posterior $p(z|x;\phi)$ via the VAE encoder neural
network, $z$ is reparameterized using the following equation:

$z=\mu_{z}+\sigma_{z}\odot\epsilon,\;\epsilon\sim\mathcal{N}(0;\mathbb{I})$
(11)

where $\mu_{z}$ is the mean of $z$, $\sigma_{z}$ is the variance of $z$,
$\odot$ denotes the Hadamard product.

The VAE decoder is used to reconstruct the original input. It can be described
as follow:

$x^{\prime}\sim p(x|z;\theta)$ (12)

where $x^{\prime}$ is the reconstructed input and $\theta$ is the decoder
parameters.

The loss function of the VAE can be written as follow:

$\displaystyle\mathcal{L}(\theta,\phi,D)=\mathop{{}\mathbb{E}}{{}_{z\sim
p(z|x)}}\big{[}-\log p(x|z;\theta)\big{]}$
$\displaystyle+D_{KL}\big{(}p(z|x;\phi)||q(z)\big{)}$ (13)

Once $\mathcal{L}(\theta,\phi,D)$ is minimized, we can have the approximate
posterior $p(z|x;\phi)$ for sampling the latent variable $z$.

####  4.2.2 Deterministic Predictor (M1 Model)

After the unsupervised learning procedure, we can acquire the latent
distribution $z$. The next step is to carry out the supervised learning
procedure to obtain the target $y$. To this end, we devise two predictor
models, one is deterministic, the other is stochastic.

Now we build a deterministic predictor which consists of two predicting steps:

Step $1$: to obtain the means of latent variables.

$\mu_{z}=\mathcal{F}_{\mu}(x;\phi)$ (14)

where $\mathcal{F}_{\mu}(x;\phi)$ can be regarded as the encoder of the VAE.

Step $2$: to obtain the final prediction based on the output of Step $1$.

$y=\mathcal{F}_{y}(\mu_{z};w)$ (15)

where $\mathcal{F}_{y}(\mu;w)$ is a deterministic multi-layer perceptron
model. Consequently, the loss function is:

$\mathcal{L}(D;w)=\frac{1}{N}\sum_{n=1}^{N}{(\hat{y}_{n}-y_{n})^{2}}$ (16)

where $\hat{y}_{n}$ is the labeled target.

Compared to using the original data as the input directly, using the learnt
latent distribution as the input of the predictor can reduce the noisy
information of the original input.

Algorithm 2 Algorithm: M1 model

1:$x_{a}$ (all input), $x_{l}$ (labeled input), $\hat{y}$ (labels)

2:$y$ (predictions)

3:

4:while unsupervised learning do

5: $\mu_{z}$, $\sigma_{z}$ $\leftarrow$ $E_{\phi}(x_{a})$ $\triangleright$
$E_{\phi}(x_{a})$: encoder network

6: $z\sim\mathcal{N}(\mu_{z},\sigma_{z})$ $\triangleright$ sample latent codes

7: $x_{a}^{\prime}$ $\leftarrow$ $p(x|z;\theta)$ $\triangleright$
$x_{a}^{\prime}$: reconstructed input

8: minimize loss function $\mathcal{L}(\theta,\phi,D)$$\triangleright$ Eq.
(4.2.1)

9:end while

10:

11:while supervised learning do

12: $\mu_{z}$ $\leftarrow$ $\mathcal{F}_{\mu}(x_{l};\phi)$ $\triangleright$
get latent codes

13: $y$ $\leftarrow$ $\mathcal{F}_{y}(\mu_{z};w)$ $\triangleright$ get
predictions

14: minimize loss function $\mathcal{L}(D;w)$ $\triangleright$ Eq. (16)

15:end while

16:return $y$

The training process of the M1 model is summarized in Algorithm 2.

####  4.2.3 Stochastic Predictor (M2 Model)

Alternatively, in contrast with the M1 model, we can also devise a stochastic
predictor, the M2 model. To this end, based on Eq. (8), we decompose the joint
distribution as follow:

$p(y,z,x;w,\phi)=p(y|z,x;w)p(z|x;\phi)p(x)$ (17)

where $p(z|x;\phi)$ is the encoder network parameterized by $\phi$ and
$p(y|z;w)$ the predictor network parameterized by $w$; $p(x)$ can be
approximated via empirically drawing the samples from the dataset.

Based on Eq. (17), we can formulate a stochastic prediction model. However,
since Eq. (17) cannot be computed explicitly, we can use Monte Carlo method to
solve it by drawing the samples of $z$ and $y$. To this end, first, we draw
the latent variables $z$ from the VAE encoder:

$z\sim p(z|x;\phi)$ (18)

Then, we draw the predicted values $y$:

$y\sim p(y|z,x;w)$ (19)

Hence, based on Eq. (4.2.1) and Eq. (17), the loss function of the M2 model
can be written as:

$\begin{array}[]{l}\mathcal{L}(D;\theta,\phi,w)=\mathop{{}\mathbb{E}}{{}_{z\sim
p(z|x)}}\big{[}-\log p(y|z,x;w)\big{]}\\\
\;\;\;\;+\mathop{{}\mathbb{E}}{{}_{z\sim p(z|x)}}\big{[}-\log
p(x|z;\theta)\big{]}+D_{KL}\big{(}p(z|x;\phi)||q(z)\big{)}\end{array}$ (20)

In Eq. (20), the first term represents the predictor, the second term
represents the decoder and the last term represents the encoder. The second
term and the last term can be optimized by the unsupervised procedure jointly
in the first step. Afterwards, since $\phi$ and $\theta$ are trained,
according to Eq. (20), here we only need to optimize $p(y|z,x;w)$ in the
second step. Thus, the loss function for training the predictor becomes:

$\mathcal{L}(D;w)=\mathop{{}\mathbb{E}}{{}_{z\sim p(z|x)}}\big{[}-\log
p(y|z,x;w)\big{]}$ (21)

We can use Monte Carlo sampling to solve this loss:

$\mathcal{L}(D;w)\approx-\frac{1}{N}\sum_{n=1}^{N}\log p(y|z,x;w)$ (22)

where $N$ is the mini batch size. Assume that the likelihood function
$p(y|z;w)$ is a Gaussian distribution with noise $\sigma_{y}$ which can be
seen as a hyper-parameter. For the prediction, we draws multiple samples via
the predictor and use their mean values as the final output.

Algorithm 3 Algorithm: M2 Model

1:$x_{a}$ (all input),$x_{l}$ (labeled input),$y_{l}$ (labels)

2:$y$ (predictions)

3:

4:while unsupervised learning do

5: $\mu_{z}$, $\sigma_{z}$ $\leftarrow$ $E_{\phi}(x_{a})$ $\triangleright$
$E_{\phi}(x_{a})$: encoder network

6: $z\sim\mathcal{N}(\mu_{z},\sigma_{z})$ $\triangleright$ sample latent codes

7: $x_{a}^{\prime}$ $\leftarrow$ $p(x|z;\theta)$ $\triangleright$
$x_{a}^{\prime}$: reconstructed input

8: minimize loss function $\mathcal{L}(\theta,\phi,D)$$\triangleright$ Eq.
(4.2.1)

9:end while

10:

11:while supervised learning do

12: $z\sim\mathcal{N}(\mu_{z},\sigma_{z})$ $\triangleright$ sample latent
codes

13: $y\sim p_{y}(z;w)$ $\triangleright$ sample predictions

14: minimize loss function $\mathcal{L}(D;w)$ $\triangleright$ Eq. (22)

15:end while

16:return $y$

Figure 2: VAE-based semi-supervised learning model.

The training process of the M2 model is summarized in Algorithm 3 and the
structure of the VAE-based semi-supervised learning model is illustrated in
Fig. 2.

##  5 Experiments and Results

For the validation, we conduct two groups of experiments to evaluate the two
proposed models, respectively.

###  5.1 Indoor Next Location Prediction

####  5.1.1 Dataset Description

For the validation dataset, we select two paths from the Tampere dataset [29].
As shown in Fig. 3, the input dimension of the Tampere dataset is $489$. The
possible RSSI values of the detected WAPs range from $-110$ dB to $0$ dB and
the RSSI values of undetected WAPs are set to $100$. The units of the target
values are meters. For pre-processing the data, we set the undetected RSSI
values to $0$ for the purpose of computational convenience, which does not
represent the actual signal strength. We only use the coordinates of the users
to shape the paths.

Figure 3: WiFi fingerprint data samples.

####  5.1.2 CMDRNN Implementation Details

The implementation details of our model are illustrated in Table 2. The CNN
sub-network consists of three layers, a convolutional layer, a max pooling
layer and a flatten layer. The RNN sub-network includes a hidden layer with
$200$ neurons. The MDN sub-network has a hidden layer and an output layer. The
mixed Gaussians number of the MDN output layer is $30$, and each mixture has
$5$ parameters, namely, two dimensional means, diagonal variances and
corresponding weights. We use RMSProp [30] as the optimizer.

Table 2: CMDRNN implementation details. Sub-network | Layer | Hyperparameter | Activation function
---|---|---|---
CNN | convolutional layer | filter number: 100; stride: 2 | Sigmoid
CNN | max pooling layer | neuron number: 100 | ReLU
CNN | flatten layer | neuron number: 100 | Sigmoid
RNN | hidden layer | memory length: 5; neuron number: 200 | Sigmoid
MDN | hidden layer | neuron number: 200 | Leaky ReLU
MDN | output layer | 5*mixed Gaussian number (5*30) | -
Optimizer: RMSProp; learning rate: $10^{-3}$

####  5.1.3 Experimental Results

In order to evaluate the effectiveness of our method, we conduct several
experiments to thoroughly compare our CMDRNN model to other deep learning
approaches. Fig. 4 demonstrates the prediction results of the two selected
paths.

(a) Path 1 prediction results.

(b) Path 2 prediction results.

Figure 4: Prediction results of two selected paths.

In the experiments, we test different optimizers and feature detectors and use
learning losses as the metrics to evaluate the performances. In [31], it
reports that RMSProp [30] may have better performance on very non-stationary
tasks than the Adam optimizer [32]. To verify this, we train our algorithm
with RMSProp and Adam, respectively. As it is shown in Fig. 5, the proposed
model can converge to a lower negative log-likelihood using RMSProp than using
Adam. Thus, we choose RMSProp as the optimizer to learn our model.

Figure 5: Training losses using different optimizers.

In terms of feature detectors, since the input is high dimensional, a
sagacious way to deal with this issue is incorporating a sub-network into the
model for dimension reduction or feature detection. Many previous research
adopted autoencoders to reduce dimension, while we argue that the more
appropriate choice for the task in our work is using a one-dimensional CNN. In
order to prove that, we test three different models, one without a feature-
detecting structure, one using an autoencoder and one using 1D CNN (the
proposed model). The autoencoder model with structure {hidden neurons: $256$;
hidden neurons: $128$; code size: $64$; hidden neurons: $128$; hidden neurons:
$256$}. As the results demonstrated in Fig. 6, the proposed model suing the 1D
CNN feature detector can reach lower negative log-likelihood during the
training process.

Figure 6: Training losses using different feature detectors.

We also wan to identify the optimal hyperparameters for the CMDRNN model,
i.e., the number of the mixture Gaussians for the MDN sub-model and the memory
length for the RNN sub-model. From the results illustrated in Fig. 7 and Fig.
8, we can see the optimal mixture number is $30$ and the optimal memory length
is $5$.

Figure 7: Prediction results of different mixture numbers in the MDN (the bars
represent the standard deviations). Figure 8: Prediction results of different
memory lengths in the RNN (the bars represent the standard deviations).

Finally, we conduct several experiments to compare our model with other
existing methods. The purposes of the experiments are indicated as follows:

  * •

Comparing feature detectors: RNN, RNN+MDN, AE + RNN + MDN, CMDRNN;

  * •

Comparing regressors: RNN, CNN+RNN and CMDRNN;

  * •

Comparing RNN variants: CMDRNN, CMDLSTM and CMDGRU.

We use three baseline models, the k-NN, the decision tree and the random
forests (which are not sequential models) [33]. We run the algorithms multiple
times with random initialisation. The results shown in Table 3, we can see
that, compared to the baseline models, our proposed models outperform other
approaches, and especially the CMDGRU model has the best performances.

Table 3: Root mean squared errors of the path prediction results (meter).

(a) Results of the baseline models. Path | k-NN | DT | RF
---|---|---|---
Path 1 | $7.44\pm 0.00$ | $8.78\pm 0.76$ | $7.25\pm 0.25$
Path 2 | $8.02\pm 0.00$ | $20.94\pm 1.52$ | $9.60\pm 0.75$

(b) Results of the sequential prediction models. Path | RNN | CNN+RNN | RNN+MDN | AE+RNN+MDN | CMDRNN | CMDLSTM | CMDGRU
---|---|---|---|---|---|---|---
Path 1 | $29.36\pm 1.61$ | $34.26\pm 3.04$ | $23.86\pm 5.50$ | $11.24\pm 0.86$ | $8.26\pm 1.31$ | $7.38\pm 0.89$ | $6.25\pm 0.80$
Path 2 | $31.61\pm 0.74$ | $36.75\pm 6.17$ | $23.58\pm 2.29$ | $12.01\pm 1.68$ | $10.17\pm 0.72$ | $9.26\pm 0.31$ | $8.67\pm 0.23$

###  5.2 Location Recognition via Semi-supervised Learning

####  5.2.1 Dataset Description

For the validation dataset, we use the UJIIndoorLoc dataset [34] which is
similar to the Tampere dataset. The input dimension of the UJIIndoorLoc
dataset is $520$. The possible RSSI values of the detected WAPs range from
$-100$ dB to $0$ dB and the RSSI values of undetected WAPs are set to $100$.
The coordinates are in longitudes and latitudes. The total instances number
for the experiments is about $20000$.

The units of the target values are meters. For pre-processing the data, we set
the undetected values the into $0$ dB for the purpose of computational
convenience, which does not represent the actual signal strength. We also
remove the duplicate instances. The original target data are longitudes and
latitudes with very large values. They are scaled in the experiments, thus the
predicting results in Table 5 do not have units. We run the algorithms
multiple times with random initialisation.

####  5.2.2 VAE-based Model Implementation Details

The VAE-based model consists of three sub-networks, an encoder, a decoder and
a predictor. The implementation details of the VAE-based semi-supervised
learning model are demonstrated in Table 4.

Table 4: VAE-based model implementation details. Sub-network | Layer | Hyperparameter | Activation function
---|---|---|---
Encoder | hidden layer | neuron number: 512; | ReLU
Encoder | hidden layer | neuron number: 512; latent dimension: 5 | ReLU
Decoder | hidden layer | neuron number: 512 | ReLU
Predictor | hidden layer | neuron number: 512; dropout rate: 0.3 | ReLU
Predictor | hidden layer | neuron number: 512; dropout rate: 0.3 | ReLU
Predictor | hidden layer | neuron number: 512; dropout rate: 0.3 | ReLU
Optimizer: Adam; learning rate: $10^{-3}$

####  5.2.3 Experimental Results

For the experimental set up, we use different portions of labeled data ranging
from $2\%$ to $80\%$. We use the k-NN, the DT and the RF as the baseline
models and use the GP, the MDN with $2$ mixtures, noted as MDN(2), the MDN
with $5$ mixtures, noted as MDN(5), as comparisons.

Fig. 9 demonstrates the distribution of the latent variable $z$. The results
verify our assumption that the latent variable $z$ are related to both the
input $x$ and output $y$.

(a) Latent variables labeled with the building IDs.

(b) Latent variables labeled with the floor IDs.

Figure 9: Latent variables with dimension of $5$, here shows the 2D
projection.

Fig. 10 shows the results obtained by the M2 model with respect to different
amounts of the labeled data. We can see that the accuracy of the testing
results increases as the amount of the labeled data increases in the
supervised learning procedure.

(a) Ground truth. (b) Labeled data: 2%.

(c) Labeled data: 5%. (d) Labeled data: 10%.

(e) Labeled data: 20%. (f) Labeled data: 30%.

(g) Labeled data: 50%. (h) Labeled data: 80%.

Figure 10: Test results of the M2 model. Table 5: Root mean squared errors of
testing results with respect to different portions of labeled data.

(a) Results of the baseline models. Labeled data | k-NN | DT | RF
---|---|---|---
2% | $0.202\pm 3\text{e-}2$ | $0.349\pm 5\text{e-}2$ | $0.234\pm 3\text{e-}2$
5% | $0.177\pm 1\text{e-}2$ | $0.254\pm 2\text{e-}2$ | $0.184\pm 2\text{e-}2$
10% | $0.156\pm 6\text{e-}3$ | $0.201\pm 2\text{e-}2$ | $0.138\pm 6\text{e-}3$
20% | $0.120\pm 4\text{e-}3$ | $0.161\pm 9\text{e-}3$ | $0.112\pm 5\text{e-}4$
30% | $0.104\pm 2\text{e-}3$ | $0.140\pm 4\text{e-}2$ | $0.100\pm 2\text{e-}3$
50% | $0.100\pm 1\text{e-}2$ | $0.126\pm 2\text{e-}3$ | $0.091\pm 2\text{e-}3$
80% | $0.092\pm 7\text{e-}3$ | $0.112\pm 3\text{e-}3$ | $0.087\pm 3\text{e-}3$

(b) Results of the comparison models. Labeled data | GP | MDN(2) | MDN(5) | M1 | M2
---|---|---|---|---|---
2% | $0.565\pm 3\text{e-}2$ | $0.161\pm 5\text{e-}3$ | $0.159\pm 4\text{e-}3$ | $0.175\pm 6\text{e-}3$ | $0.166\pm 8\text{e-}3$
5% | $0.313\pm 2\text{e-}2$ | $0.139\pm 4\text{e-}3$ | $0.143\pm 3\text{e-}3$ | $0.133\pm 3\text{e-}3$ | $0.123\pm 4\text{e-}3$
10% | $0.275\pm 3\text{e-}3$ | $0.120\pm 6\text{e-}3$ | $0.129\pm 1\text{e-}3$ | $0.105\pm 2\text{e-}4$ | $0.106\pm 5\text{e-}3$
20% | $0.262\pm 8\text{e-}4$ | $0.107\pm 6\text{e-}3$ | $0.105\pm 4\text{e-}3$ | $0.093\pm 2\text{e-}3$ | $0.093\pm 2\text{e-}3$
30% | $0.258\pm 2\text{e-}3$ | $0.102\pm 5\text{e-}3$ | $0.105\pm 4\text{e-}3$ | $0.086\pm 3\text{e-}3$ | $0.087\pm 2\text{e-}3$
50% | $0.253\pm 2\text{e-}3$ | $0.101\pm 7\text{e-}3$ | $0.097\pm 4\text{e-}3$ | $0.080\pm 9\text{e-}4$ | $0.083\pm 4\text{e-}3$
80% | $0.253\pm 2\text{e-}3$ | $0.098\pm 7\text{e-}3$ | $0.103\pm 3\text{e-}3$ | $0.077\pm 4\text{e-}3$ | $0.079\pm 3\text{e-}3$

Table 5 shows the results obtained by different methods with respect to
different amounts of the labeled data. From the results, we can see that the
proposed models outperform the baseline model proposed in [33]. Moreover, M1
and M2 can provide satisfying results even when the labeled data are scarce.
The predicting accuracy is improved when the labeled data increases. In
contrast with other existing machine learning and deep learning methods, the
proposed models have better performance. In practice, we also find that the
proposed models, compared to other methods, have such following advantages:

  * •

Compared to the GP, the VAE-based models are less computationally expensive
when the training dataset is large;

  * •

Compared to the MDNs, the VAE-based models are more computationally stable
during the learning process.

##  6 Conclusions and Perspectives

In this paper, we aim to tackle the WiFi fingerprint-based indoor positioning
problem. The first task in our work is predicting next user location with WiFi
fingerprints. By contrast with existing approaches, our solution is a hybrid
deep learning model. The proposed model is composed of three deep neural
networks, a CNN, a RNN and a MDN. This unique deep architecture combines all
the potentials of three deep learning models, which enables us to predict user
location with relatively high accuracy.

The second task in our work is recognizing accurate user location via semi-
supervised learning. To solve this problem, we propose a VAE-based semi-
supervised learning model. Our model employs a VAE model to carry out the
unsupervised learning procedure so as to learn a latent distribution of the
original input. Afterwards, in order to use the learnt distribution to compute
the user coordinates, we devise two different predictors, one is
deterministic, the other is stochastic.

Finally, we test our models on two real-world datasets. From the results, we
can see that the CMDRNN model has better performance than conventional machine
learning methods and conventional RNN-based models for indoor next location
prediction. The VAE-based model outperform other conventional machine learning
and deep learning methods for accurate location recognition. For the future
work, we plan to explore other advanced deep learning methods to develop new
applications.

## References

  * [1] Paramvir Bahl and Venkata N Padmanabhan.  Radar: An in-building rf-based user location and tracking system.  In Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No. 00CH37064), volume 2, pages 775–784. Ieee, 2000.
  * [2] Moustafa Youssef and Ashok Agrawala.  The horus location determination system.  Wireless Networks, 14(3):357–374, 2008.
  * [3] Sung-Bae Cho.  Exploiting machine learning techniques for location recognition and prediction with smartphone logs.  Neurocomputing, 176:98–106, 2016.
  * [4] Chen Yu, Yang Liu, Dezhong Yao, Laurence T Yang, Hai Jin, Hanhua Chen, and Qiang Ding.  Modeling user activity patterns for next-place prediction.  IEEE Systems Journal, 11(2):1060–1071, 2017.
  * [5] Minh Tu Hoang, Brosnan Yuen, Xiaodai Dong, Tao Lu, Robert Westendorp, and Kishore Reddy.  Recurrent neural networks for accurate rssi indoor localization.  IEEE Internet of Things Journal, 6(6):10639–10651, 2019.
  * [6] Sinem Bozkurt, Gulin Elibol, Serkan Gunal, and Ugur Yayan.  A comparative study on machine learning algorithms for indoor positioning.  In 2015 International Symposium on Innovations in Intelligent SysTems and Applications (INISTA), pages 1–8. IEEE, 2015.
  * [7] Andrei Cramariuc, Heikki Huttunen, and Elena Simona Lohan.  Clustering benefits in mobile-centric wifi positioning in multi-floor buildings.  In 2016 International Conference on Localization and GNSS (ICL-GNSS), pages 1–6. IEEE, 2016.
  * [8] Joaquín Torres-Sospedra, Raúl Montoliu, Sergio Trilles, Óscar Belmonte, and Joaquín Huerta.  Comprehensive analysis of distance and similarity measures for wi-fi fingerprinting indoor positioning systems.  Expert Systems with Applications, 42(23):9263–9278, 2015.
  * [9] Carl Edward Rasmussen.  Gaussian processes in machine learning.  In Summer School on Machine Learning, pages 63–71. Springer, 2003\.
  * [10] Brian Ferris, Dieter Fox, and Neil D Lawrence.  Wifi-slam using gaussian process latent variable models.  In IJCAI, volume 7, pages 2480–2485, 2007.
  * [11] Brian Ferris Dirk Hähnel and Dieter Fox.  Gaussian processes for signal strength-based location estimation.  In Proceeding of robotics: science and systems, 2006.
  * [12] Simon Yiu and Kai Yang.  Gaussian process assisted fingerprinting localization.  IEEE Internet of Things Journal, 3(5):683–690, 2015.
  * [13] Tao Yang, Cindy Cappelle, Yassine Ruichek, and Mohammed El Bagdouri.  Online multi-object tracking combining optical flow and compressive tracking in markov decision process.  Journal of Visual Communication and Image Representation, 58:178–186, 2019.
  * [14] Anders Krogh, BjoÈrn Larsson, Gunnar Von Heijne, and Erik LL Sonnhammer.  Predicting transmembrane protein topology with a hidden markov model: application to complete genomes.  Journal of molecular biology, 305(3):567–580, 2001.
  * [15] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [16] Geoffrey E Hinton and Ruslan R Salakhutdinov.  Reducing the dimensionality of data with neural networks.  science, 313(5786):504–507, 2006.
  * [17] Jeffrey L Elman.  Finding structure in time.  Cognitive science, 14(2):179–211, 1990.
  * [18] Mai Ibrahim, Marwan Torki, and Mustafa ElNainay.  Cnn based indoor localization using rss time-series.  In 2018 IEEE Symposium on Computers and Communications (ISCC), pages 01044–01049. IEEE, 2018.
  * [19] Roshan Ayyalasomayajula, Aditya Arun, Chenfeng Wu, Sanatan Sharma, Abhishek Rajkumar Sethi, Deepak Vasisht, and Dinesh Bharadia.  Deep learning based wireless localization for indoor navigation.  In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, pages 1–14, 2020.
  * [20] Michał Nowicki and Jan Wietrzykowski.  Low-effort place recognition with wifi fingerprints using deep learning.  In International Conference Automation, pages 575–584. Springer, 2017.
  * [21] Xudong Song, Xiaochen Fan, Chaocan Xiang, Qianwen Ye, Leyu Liu, Zumin Wang, Xiangjian He, Ning Yang, and Gengfa Fang.  A novel convolutional neural network based indoor localization framework with wifi fingerprinting.  IEEE Access, 7:110698–110709, 2019.
  * [22] Kyeong Soo Kim, Sanghyuk Lee, and Kaizhu Huang.  A scalable deep neural network architecture for multi-building and multi-floor indoor localization based on wi-fi fingerprinting.  Big Data Analytics, 3(1):4, 2018.
  * [23] Christopher M Bishop.  Pattern recognition and machine learning.  Springer Science+ Business Media, 2006.
  * [24] Bishop, Christopher M.  Mixture density networks.  1994\.
  * [25] José Miguel Hernández-Lobato and Ryan Adams.  Probabilistic backpropagation for scalable learning of bayesian neural networks.  In International Conference on Machine Learning, pages 1861–1869, 2015.
  * [26] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In Yoshua Bengio and Yann LeCun, editors, Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2014.
  * [27] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [28] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  In NIPS 2014 Workshop on Deep Learning, December 2014, 2014.
  * [29] Elena Simona Lohan, Joaquín Torres-Sospedra, Helena Leppäkoski, Philipp Richter, Zhe Peng, and Joaquín Huerta.  Wi-fi crowdsourced fingerprinting dataset for indoor positioning.  Data, 2(4):32, 2017.
  * [30] Tijmen Tieleman and Geoffrey Hinton.  Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.  COURSERA: Neural networks for machine learning, 4(2):26–31, 2012\.
  * [31] SC Martin Arjovsky and Leon Bottou.  Wasserstein generative adversarial networks.  In Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, 2017.
  * [32] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations (ICLR), 2015.
  * [33] Jordi Rojo, Germán Martín Mendoza-Silva, Gabriel Ristow Cidral, Jorma Laiapea, Gerardo Parrello, Arnau Simó, Laura Stupin, Deniz Minican, María Farrés, Carmen Corvalán, et al.  Machine learning applied to wi-fi fingerprinting: The experiences of the ubiqum challenge.  In 2019 International Conference on Indoor Positioning and Indoor Navigation (IPIN), pages 1–8. IEEE, 2019.
  * [34] Joaquín Torres-Sospedra, Raúl Montoliu, Adolfo Martínez-Usó, Joan P Avariento, Tomás J Arnau, Mauri Benedito-Bordonau, and Joaquín Huerta.  Ujiindoorloc: A new multi-building and multi-floor database for wlan fingerprint-based indoor localization problems.  In 2014 international conference on indoor positioning and indoor navigation (IPIN), pages 261–270. IEEE, 2014.