# IoT-based Wearables: A comprehensive Survey

Yahuza Bello 1, Emanuel Figetakis 1  1 University of Guelph, Ontario, Canada.

###### Abstract

A substantial amount of growth is being achieved by businesses through IoT-
based services. The emergent of small electronic devices capable of computing,
which are commonly known as wearables in IoT domain has proven to have huge
impact in people’s life. Theses wearables are capable of collecting vital
information about a person’s activities and behaviours regularly. This makes
them suitable for many applications in health monitoring, fitness, sports,
education and some industry related applications. To this end, in this paper,
we aim to provide a general review on IoT-based wearables, the sensors adopted
for several categorized wearables, the communication technologies adopted and
the most widely adopted data processing techniques for wearables. Furthermore,
we present the challenges faced for wide adoption of wearables and the future
research directions.

###### Index Terms:

Internet of things, wearables, communication technologies, data analytic

##  I Introduction

In today’s modern era, several applications are developed with the need of
connectivity to the internet to function effectively. The Internet of Things
(IoT) is the concept of having embedded systems interconnected together and
communicating through the Internet [1]. This provides the required
connectivity to the internet for such applications. The IoT domain has been
thoroughly investigated in both academia and industry and its development
through the years was rapid because of its wide range of applications in
various fields such as health, IoT-enabled sports, IoT-enabled factories,
agriculture, IoT-enabled cities, traffic management, smart supply chain and
smart grid to name a few as depicted in Figure 1.

A substantial amount of growth is being achieved by businesses through IoT-
based services. For example, the biggest economic impact will be from
healthcare and manufacturing applications. The global economy is expected to
generate about $1.1-$2.5 trillion US dollars in growth annually by 2025 as a
result of IoT-based healthcare applications and services, such as mobile
health (mHealth) and telecare services that improves diagnosis, treatment and
other monitoring services. IoT is estimated to generate a total economic
impact amounting to $2.7 trillion - $6.2 trillion annually by 2025 [2]. In
Figure 2, dominant IoT applications (i.e., applications in healthcare,
manufacturing, electricity, urban infrastructure, security, resource
extraction, agriculture, retail and vehicles) are shown with their projected
market share [3].

A typical IoT consists of 6 building blocks as shown in Figure 3. These
buildings blocks are identification, sensing, communication,
computation/processing, services and semantics. By knowing these building
blocks, you can gain a deeper understanding of how IoT works and what it
means. Identifying services and matching them with customer demands is crucial
for IoT. Electronic product codes (EPCs) and ubiquitous codes (uCodes) are two
most adopted identification methods for the IoT systems. Sensors in the IoT
gather data from related objects and send it to data warehouses, databases, or
the cloud. Based on the analysis performed on the collected data, specific
actions are taken according to the services that are required. There are a
variety of IoT sensors, including smart sensors, actuators, and wearable
sensors. We will discuss sensors in the context of wearable devices in section
IV. Using multiple communication technologies, heterogeneous IoT objects can
be connected to provide specific smart services. The different communication
technologies used in the context of wearable devices will be discussed in
detail in section V. In IoT, processors, microcontrollers, System-on-Chips
(SoCs), and Field-Programmable Gate Arrays (FPGAs) comprise the ”brain” and
computational power. Arduino, UDOO, FriendlyARM, Intel Galileo, Raspberry PI,
Gadgeteer, BeagleBone, Cubieboard, Z1, WiSense, Mulle, and T-Mote Sky are some
of the hardware platforms designed to run IoT applications.

The emergent of small electronic devices capable of computing, which are
commonly known as wearables in IoT domain has proven to have huge impact in
peoples life [4]. Theses wearables are capable of collecting vital information
about a person’s activities and behaviours regularly. This makes them suitable
for many applications in health monitoring, fitness, sports, education and
some industry related applications. Wearables are often worn as additional
accessories on individual’s clothing, as implants on certain part of the body,
or even tattooed to the skin. Being part of the IoT ecosystem, wearables are
connected to the Internet in order to gather and transmit vital information.
Additionally, the mobility of people and animals makes wearable devices
increasingly important since they can collect, send, and receive data from the
Internet in real time, and thus help us to make better decisions to improve
our lifestyle.

A wide variety of wearable products such as smart jewelleries, smart
wristbands, smart watches, smart glasses, smart shoes and smart belts are
already available in the markets as illustrated in Figure 4. According to
International Data Corporations (IDC), the global shipments of wearable
products are projected to grow from 66.5 million units as reported in 2019 to
105.3 million units by the end of 2024 [5]. Wearable devices first emerged as
fitness activity trackers, then comes other applications such as Bluetooth
headsets, smartwatches, and web-enabled glasses shortly [6]. Afterwards,
virtual reality headsets and augmented reality headsets were introduced in the
gaming industry. However, health monitoring and medical usage cases are the
most important life-changing applications of IoT-based wearable technologies.

Figure 1: Internet of things Applications [7]

Wearable devices are a rapidly developing field with the potential to open up
new applications. This review is motivated by the potential of wearable
devices to be able to open up new applications in various fields.
Specifically, in this review, we will focus mainly on identifying the
different wearables in the IoT domain, discuss the different sensors adopted
for these wearables and then cover the different data processing techniques
and communication technologies that are widely used for the IoT-based
wearables.

Figure 2: Projected market share of dominant IoT applications by 2025 [3]
Figure 3: IoT components [3]

Although there are several survey papers [4, 8, 9, 10, 11, 12, 13] that
addressed the concept of wearables and the ongoing research efforts, most of
them focused on specific application such as applications in fitness and
health monitoring neglecting the sensors behind each wearable, the various
communication techniques adopted and data processing techniques that are
commonly used for IoT-based wearables. To this end, in this paper, we aim to
provide a general review on IoT-based wearables, the sensors adopted for
several categorized wearables, the communication technologies adopted and the
most widely adopted data processing techniques for wearables.

The rest of the paper is structured as follows: Section II presents the
relevant survey papers published in the literature. Section III introduces
IoT-based wearables and categorized them according to the targeted
application. In section IV, the most commonly used sensors for IoT-based
wearables are discussed. Section V discusses the various communication
technologies adopted for wearables and the various data analytic techniques
used for data processing. Section VI presents the challenges faced for wide
adoption of wearables and the future research directions. Section VII
concludes the paper.

Figure 4: IoT-based Wearable products

##  II Related Works

A number of other research studies have examined wearable technologies in
different ways. For example, the survey paper in [4] provides academics
seeking further research topics with an in-depth understanding of smart
wearables concept. The authors present an analysis of behavioral predictors of
smart wearable adoption and adoption intentions among people. In [8], using
signals and sensors, the authors examines context-based pairing in wearable
devices. The author in [9] examine each type of IoT architecture as well as
different methods of data transfer, data processing, and computing paradigms.
Then the IoT-assisted wearable sensor systems and their various applications
in healthcare as well as the various communication technologies adopted are
presented in the paper. The authors in [10] reviews both scientific papers and
commercial efforts related to wearable health care devices. The paper focuses
on the most important wearable devices that directly measure health status
parameters. An overview of existing literature on intelligent wearables is
presented in [11]. The authors went further to provide a review of the risks
of using intelligent wearables and explain what risks were considered in
previous research. The authors in [12] discuss and explore several
communication and artificial intelligence techniques, which are suitable for
the next generation of wearable devices. These techniques when fully adopted
will enable emergence of innovative services. An overview of recent research
on wearables and IoT technologies used for fitness assessment is provided in
[13].

##  III Wearables in IoT

There are many applications in the field of IoT that can be enhanced by
wearable technology. However, wearable devices will become worth their weight
in gold when they are integrated into a true IoT system. As a result, most
research papers currently published in the literature connect wearable devices
to the Internet in one of two ways. Either the wearable devices send data to
the cloud or to an Internet server for offline processing or some of the data
processing is done locally on the wearable device. Having integrated IoT
platforms and addressing many issues pertaining to data ownership, data
sharing policies, privacy, and safety will fully realize the prevalence of
wearable devices.

In many cases, a mobile network is needed to support these wearable devices.
Data is also a concern when dealing with sensors that collect many readings.
In works [14, 15, 16, 17, 18] the authors create specialized networks that can
handle the needs of wearable devices. These networks also address several
concerns such as security and data storage. In, [19] shows that edge nodes can
also play an important role with IoT devices and data collection. It can
reduce load on the server and save storage space.

Several research works have categorized IoT-based wearables by considering
multiple factors. As part of its ongoing effort to standardize wearable
electronic devices and technologies, Technical Commission (TC) 124 of the IEC
(International Electrotechnical Commission) identifies four types of smart
wearables: accessory wearables, textile/fabric wearables, patchable wearables
and implantable wearables. For details on this categorization, refer to [20].
The IEC Standardization Group (SG) 10 on wearable smart devices also indicated
that these wearable devices can be categorized in accordance with their
location within, on, or near an individual. The categories are near-body
wearables, in-body wearables, on-body wearables and electronic textiles [21].
According to [22], IoT-based wearable devices can be categorized into wrist-
worn devices (such as smart watches and wrist bands), haed-mounted devices
(such as smart eyewear, headset and earbuds), e-textile (such as smart
garments), e-patches (such as sensor patches and e-tattoo) and others (this
category covers that do not fall under the other categories such as smart
jewellery and straps).

Figure 5: Application-based Classification of Wearable Devices

In this paper, we categorized wearable devices according to the targeted
applications in IoT domain similar to the work in [23]. As depicted in Figure
5, IoT-based wearbles can be classified according to the targeted applications
as Health-based wearables, safety-based wearables, sports and activity
recognition wearables and tracking and localization wearables.

TABLE I: Summary of the Categorized Wearables and the relavant related works Wearable categories |  Description |  Related works
---|---|---
Health-based applications |  Wearable IoT devices used to monitor and treat patients remotely, and in some cases for rehabilitation. |  [23, 24, 25, 26, 27, 28, 29]
Safety Applications |  wearables in the safety category are used to provide a safe environment for users. |  [30, 31, 32, 33, 34, 35]
Sports and activity recognition |  This category pertains to applications where wearables are worn during sport activities to track different metrics of user/athlete activity to improve their performance. |  [36, 37]
Tracking and localization |  Tracking people and animals online is the most common use of this category. |  [38, 39]

###  III-A Health-based wearables

In the health sector, Wearable IoT devices are usually used to monitor and
treat patients remotely, and in some cases for rehabilitation. Data about a
patient’s health is collected through sensors, and the wearable device may
perform small analysis before sending the data to the Internet for further
processing. Additionally, these wearables are capable of receiving additional
inputs to aid for further analysis. Often, health-based wearable devices are
typically connected to smartphones for analyzing data and sending it to cloud
computing frameworks like Azure or Amazon Web Services (AWS) for handling,
storing, and processing. The use of mobile health applications can provide
insight about a patient’s health and provide a visual representation of the
analysis of the data. The wearable also can be programmed to respond to
special commands, such as heating up the body or applying shocks, based on the
analyzed data during treatment.

Health-based applications of wearables can be broadly categorized as health
treatment and rehabilitation wearable systems and health monitoring wearable
systems [23]. Several studies have demonstrated the significance of health
treatment and rehabilitation wearable systems.

By using rehabilitation wearables, disabled patients can maintain or improve
their mental or physical abilities. The authors in [24] presents a walker-
based physiotherapy system for monitoring and evaluating movement metrics is
proposed, which sends the data to the cloud for analysis, and displayed on a
mobile application in real time. In [25], a stroke rehabilitation framework
that utilizes smart wearable armbands, machine learning algorithms, and
3-D-printed robot hands to assist stroke patients was proposed. An IoT sensing
device was integrated with textile electrodes to develop the SWA, which
measures, pre-processes, and wirelessly transmits biopotential information.
The work in [26] presents a framework for an m-Health monitoring system based
on a cloud computing platform (Cloud-MHMS) for achieving personalised and
high-quality health monitoring using new technologies, such as mobile networks
and cloud computing. A system using IoT data to support emergency medical
services is presented as a demonstration of how data can be collected,
integrated, and interoperated flexibly in [27]. The authors develop a
Resource-based Data Accessing-IoT (UDA-IoT) method that allow access to IoT
data resources in a universal manner in order to enhance the accessibility of
IoT data resources. The authors in [28] propose a new cloud-based wheelchair
assist system, which supports impaired drivers by providing safe driving
conditions. The authors utilized an embedded mobile System-On-a-Chip (SoC) and
android based mobile application for the proposed system. A gaming-based
rehabilitation system integrating wearable technology and IoT is presented in
[29] in order to help stroke patients who, suffer from upper limb
disabilities.

In the same fashion, health monitoring wearable systems are categorized based
on the kind of sensor adopted as Bio-potential sensors, motion sensors,
environmental sensors and chemical sensors [23]. The different kind of sensors
used in most IoT-based wearables will be discussed in detail in the next
section.

TABLE II: Common Communication Technologies used in IoT-based Wearables Type of Technology |  Range |  Data Rates |  Frequency Band (s) |  Topology
---|---|---|---|---
ZigBee |  10 to 100 meters |  250kbps |  2GHz |  Star, ad hoc, and mesh
WiFi |  10 to 100 meters |  6.75Gbps |  2.4GHz, 5GHz |  Point Hub
Bluetooth |  10 to 100 meters |  2.1Mbps |  2.402GHz to 2.408GHz |  Point to point, point to multi-point
LoRaWan |  15 to 20 Kilometers |  250bps to 5.5kbps |  169 MHz (Asia), 868MHz (Europe) 91MHz (North America) |  Star

###  III-B Safety-based wearables

The wearables in the safety category are used to provide a safe environment
for users. For example, mines can benefit from the use of safety-based
wearable devices that monitor air quality to protect workers and reduce costs
incurred by employers and the workers. Safety-based wearable devices are used
in many applications to detect or prevent falls, especially in elderly people.
Several studies in the literature have investigated and proposed numerous
techniques and algorithms for fall detection.

Using pretested templates for each type of fall and comparing angular
velocities and angles between falls and normal activities, [30] propose a fall
detection framework. In [31], deep learning and activity characteristics are
used to develop a novel method for detecting human falls on furniture using
scene analysis that adopts Region-based Convolutional Neural Network (R-CNN).
A fall detection system incorporated tri-axial accelerometers, gyroscopes, and
magnetometers to obtain fall parameters was proposed by [32]. Based on a
reverse approach, the authors in [33] propose a novel autonomous fall
detection system. The proposed system uses a camera attached to the subject
rather than static sensors at fixed locations. Therefore, monitoring is not
limited to areas where the sensors are located but includes all areas where
the subject travels. [34] presents a novel system that uses a smart camera to
detect falls and classify activities around the waist. An EEG-based BCI
prototype is proposed [35] to detect whether an on-site worker is sleep-
deprived or not in an elegant manner. Modified safety helmets with a
discreetly placed signal acquisition device are worn by workers to ensure
their safety. Table I presents a summary of the categorized wearables and the
relevant related works published in the literature.

##  IV Sensors in IoT-based Wearable

Sensors are a crucial part of IoT-based wearable, they allow for the data
collection of a certain action that can then be analyzed and measured. They
are many different sensors, and each single sensor can have multiple different
applications.

###  IV-A Sensors found in Sports

Wearables that are used in sports to gain metrics and about an athlete’s
performance are categorized under Inertial Measurement Unit (IMU) sensors and
help with the study of motions, kinematics [40] The sensors that are
categorized under IMUs are magnetometer, accelerometer, and gyroscopes.
Magnetometer helps determine orientation of an object which in sports has many
applications, the sensors themselves are small require low energy to power and
can communicate over I2C or SPI [41]. Accelerometers can measure movement in a
given direction, they can be as precise to measure movement such as wrist tilt
[42]. Gyroscopes measure orientation as well as angular velocity, it also has
the ability to maintain orientation as well [43]. Combining the sensors into a
single device can help measure and study kinematics in a sport setting.
Examples of the sensors being used in a device can be found in [44] and [45]
where the sensors are used for their advantages in capturing certain ranges of
motions. The data is then captured and analyzed by their respective data
analysis programs. Sensors in IoT are very violate especially in Wearables,
many sensors have more than one application, for example IMU sensors have
applications in wearables outside of sports. For example, in [46] IMU sensors
are used to create 3D maps, an application that is used outside of the scope
of motion.

###  IV-B Sensors found in Healthcare

When it comes to healthcare there is a long list of sensors that are found in
wearables. Wearables in health care can be used to monitor or treat patients,
which are powered by different types of sensors. Sensors already discussed
[41][42][43] also have use in healthcare since they deal with the study of
motion with the human body [40].

TABLE III: Sensors Found in Healthcare IOT Wearables Sensor |  Product |  Description
---|---|---
Airflow Sensor[47] |  [48] |  Helps correct a patients irregular breathing to supply enough oxygen to the body
Pressure Sensor[49] |  [48][50][51][52] |  Regulates and creates pressure in various applications
Oxygen Sensor[53] |  [48][54] |  Measures a patients oxygen level as well as pulse rate
Electroencephalography(EEG)[55] Sensor |  [56] |  Measures a patients brain activity
Glucose Sensor |  [57] |  Measures a patients blood sugar level
Electrocardiogram(ECG) Sensor [58] |  [59][60][61] |  Measures a patients heart rate

Looking at table III there are many different sensors found in different
wearable devices for healthcare. Many sensors are present in many different
devices, and sometimes are even combined with other sensors from other
categories of wearables. Looking at the airflow sensor, it has a
straightforward purpose within healthcare, to help a patient breathe. They can
be found mostly in ventilators[48] as well as devices for oxygen therapy[50].
A pressure sensor have an unexpectedly large impact on devices, they are used
in ventilators, oxygen therapy, sleep therapy[51], and automation of drug
infusion[52]. Another sensor that is common throughout healthcare is oxygen
sensors, which measure a patients oxygen level. The sensor can be found in
devices that attach to a patient’s finger [54]. EEG sensors are also used in
healthcare, they are used in research for the human brain to help gain a
better understanding and can also be used to help diagnose any issues. Glucose
sensors mostly benefit patients with diabetes, the sensor has become non-
invasive and implemented on IoT devices such as smartwatches that can
automatically send alerts to let the patient know any imbalances are
occurring. The ECG sensor might be the most know sensor in healthcare, it is
present in every hospital and almost all smart watches/ring[59][60][61]
feature an ECG sensor. The ECG measures heart rate and can collect information
about a patient throughout the day to see if any irregularities are occurring.

###  IV-C Sensors found in Safety

Wearables in the area of safety are also finding applications, the scenarios
range from at home to work environment safety. Wearable devices are fitted
with sensors that can check the environment, make sure certain equipment is
being handled properly, and prevent and kind of injury. Taking a look at [62]
the authors used different sensors in a wearable device to determine harmful
environmental conditions. The sensors used in their wearable are temperature,
C02, UV, and CO [63][64][65][66]. When it comes to safety most sensors can be
implemented in wearables, for sports, health, and tracking and localization
they can be added as preventative measures. For example, in sports,
accelerometers can be used to detected impacts which can cause injuries, in
healthcare ECG sensors can be used to detect abnormal heart rates, and in
tracking sensors can be implemented to prevent someone from getting lost.

Figure 6: Example of Safety Wearable with sensors[62]

###  IV-D Sensors found in Tracking/Localization

Werables that are categorized in tracking and localization help with
determining the environment conditions by utilizing sensors as well
determining location [67]. Tracking is a category with a wide area of
applications, it can range from actual geo-locating to tracking a person’s
movements or specific body part. From a medical/sport standpoint sensor like
accelerometers[42] can track movement to help determine the kind of motion,
this also applies to magnetometer and gyroscopes. For determine location via
satellite location sensors are used in wearables [68], this helps with
everything between loss prevention to navigation. For more local mapping that
does not need communication to satellite, like somewhere in a house or small
community, accelerometers are used to measure distance between certain points
to create a map. This comes into play in applications with smart homes, as
well as mobile devices to help predict a user’s habits.

Figure 7: Communication Technologies for IoT-based Wearables [22]

##  V Data Analysis and Communication technologies for IoT-based Wearables

In this section, we will first introduce the various communication
technologies that are widely adopted in IoT-based wearables. Then the various
data analytic techniques in the context of wearables will be discussed.

###  V-A Communication Technologies

Various wireless communication protocols support different communication
ranges in wearables to offer a wide range of applications, as shown in Figure
7. Bluetooth, ZigBee and ANT belong to the short range class, Wi-Fi and
cellular, LoRaWan belongs to the long range class and the ultra short range
class consists of NFC.

####  V-A1 ZigBee

ZigBee technology is built on the IEEE 802.15.4 standard. Resource-constrained
environments and devices with limited power provide the perfect environment
for this kind of technology to thrive. In spite of being standardized in 2003,
it was originally conceived in 1998 [69]. ZigBee has a range of approximately
ten to a hundred meters and consumes an insignificant amount of power. ZigBee
uses the direct sequence spread spectrum (DSSS) technique and adopts the ad
hoc, mesh and star topology. In ZigBee, there are three types of devices: the
coordinator (ZC), the router (ZR), and the end-device (ZED) [70].

####  V-A2 Wi-Fi

A wireless communication standard based on IEEE802.11, wireless fidelity (Wi-
Fi) is one of the most popular wireless technologies. In order to access radio
channels, Wi-Fi uses the CSMA/CA protocol. With a range up to 10 meters and a
high -power consumption, Wi-Fi is a highly efficient wireless network. There
are four types of configurations available with this network: infrastructure,
ad hoc, bridge, and repeater, which has a maximum data transfer rate of
6.75Gbps and 140MHz of channel capacity for a very high price [71].

####  V-A3 Bluetooth

Wireless personal area networks (WPANs) with high security can be formed using
Bluetooth, a proprietary open standard developed for mobile and fixed devices.
Using a master-slave (client-server) model, Bluetooth is a packet-oriented
protocol that is connection-oriented, packet-based. Bluetooth were limited by
the initial specifications’ high-power consumption, long communication set-up
time, and high transmission latency (about 100 ms). Due to these limitations,
Bluetooth Low Energy (BLE) was developed to overcome them. BLE is designed
specifically for applications relating to health, sports, and fitness
applications.

####  V-A4 LoRaWan

IoT networks have become increasingly interested in LoRaWan (low-power wide-
area networks) because it is suitable for LPWAN (low-power wide-area
networks). LoRaWan is very suitable for wireless sensor networks due to its
robustness and range. Frequency Shift Keying (FSK) or Chirp Spread Spectrum
(CSS) are the main modes of communication used in LoRa radios. A node and a
gateway are the two main types of devices in this technology. Gateways are
connected to thousands of nodes at once, each sending and receiving
information from the gateway in a LoRaWan setup.

In the wearable device domain, Bluetooth Classic (standardized but no longer
maintained by IEEE 802.15.1), Bluetooth Smart and Wi-Fi have become the most
common standards [72] used for connectivity to the Internet. A comparative
analysis of the different adopted communication technologies used in IoT-based
wearables in terms of range, data rates, frequency bands and topology is
presented in Table II.

###  V-B Data Analytics

Data analytics has a large application within wearables in IoT, since many new
methods of machine learning are being developed, they can be used within these
applications. In a system there is the device with its many sensors taking
constant measurements it is the analytics that take the measurements and make
them a readable output. IoT-based Wearables are classified in to four
applications, health applications, sports applications, safety applications,
tracking and localization applications. Wearables from each application can
differ slightly in design and include the same sensors, it is how the data is
analyzed that differs from each application. For example, for tracking and
localization wearable can feature an accelerometers to track distance traveled
but a wearable from healthcare application could be using an accelerometers to
track the motion of a patient for physical therapy.

Machine learning can play an important role in data analysis for wearables. It
can help learn a person’s habits and find what is normal for the user. In [73]
it comes in the form of personalized medicine, where a machine learning
algorithm can learn about a patient and then keep checks on the patient for
irregularities outside of their normal. The same type of concept can be used
in all applications of wearables. Another example of data analysis comes from
[74] where a wearable is being used to help diagnose early stages of
Parkinson’s disease. The sensors collect data which is then analyzed and
compared to the early signs of the disease.

Figure 8: System used for collecting data and analyzing data from a sensor
[74]

In many of the systems, the data analysis doesn’t take place on the device
itself. The computation needed cannot be met, there edge nodes and cloud
-based systems are put in place to analyze the data. However, this doesn’t
come without challenges, many forms of data analytics require a large amount
of data also many sensors are collecting data constantly throughout the entire
day. Over time this accumulates into terabytes of data that can fill up
storage drive quickly. Another challenge that must be addressed in privacy,
the data that is being collected about a person’s habits/lifestyle is very
detailed and can be shared between different companies, making privacy an
issue. In [75] an approach is taken with data analytics to unlabeled the data
to help privacy. By doing this they are not disclosing where the data is
coming from and keeping to anonymous, the only competent that is receiving the
labelled data is the algorithm doing the analysis.

Figure 9: System used for privacy data processing [75]

##  VI Challenges and Future Research Directions

There are several challenges in the domain of IoT-based wearables. To fully
utilize the benefits of wearables devices in the community, these challenges
need to be addressed urgently.

###  VI-A Battery-related Issues

Since batteries have limited working times, wearable devices often have a
short sustainable working time, which causes inconvenience to people’s daily
lives. Therefore, when designing wearables, special considerations need to be
taken into account to minimize human interaction and ensure that the batteries
last for many hours without needing to be replaced or recharged. For example,
there are a variety of systems available, such as those that use low power
consumption or techniques that harvest energy such as thermo-electric and
piezo-electric that can be utilized in wearable domain.

###  VI-B Communication-related Issues

Currently, most wearable devices connect to the Internet primarily through
proxy devices such as smartphones or personal computers. For example, most
fitness tracking wearable applications still communicate with clouds through
smartphones. This shows a gap in the adoption of direct communication with the
internet, which hinders the development of some specific delay sensitive
applications in the wearable domain. Potential reasons for this lack of
support may be the absence of secure direct communication devices built into
wearable operating systems or the slower development of third-party
applications for wearables. However, the demand for wearable devices capable
of direct communication with the internet is on the rise [76]. This is a
potential area of research to explore the design of wearable devices with such
capability.

###  VI-C Trust-related Issues for Medical Use Cases

There is lack of trust in the sensitive data produced by consumer wearables in
patients monitoring applications. For example, data related to heart rate,
pulse rate, and other health metrics is sensed by wearables using consumer
hardware. Therefore, physicians are reluctant to use these data for critical
diagnostics because they much rely on the accuracy of the hardware used in
making the wearables as well as the accuracy of using it properly. A critical
challenge would therefore be to improve health-related data accuracy cheaply.

###  VI-D Privacy-related Issues

It is possible that privacy breaches could occur as a result of an exchange of
personal data between wearables and IoT hubs, including vital health signals,
dosage, and location. It is typical for wearable IoT devices to operate in
broadcast mode so that other network nodes can discover them easily. As a
result, unauthorised users can intercept data in the form of eavesdropping
attacks, which leads to privacy violation. A number of questions remain
unanswered concerning the effective protection of users’ privacy. This is an
interesting research problem that is on-going not only in the wearable domain
but in other security-related domains.

##  VII Conclusion

From fitness and sport to health monitoring, wearable devices are becoming
increasingly popular. In this paper, we provided a comprehensive review of the
most important research efforts from the literature in IoT-based wearables. We
categorized the wearables according to their applicable applications.
Additionally, the sensors, communication technologies and data analytic
techniques adopted in the IoT-based wearables is investigated and presented by
surveying multiple papers published in the literature. Additionally. the
challenges as well as the future research directions in IoT-based wearables is
presented. In terms of communication technologies, Bluetooth Classic,
Bluetooth Smart and Wi-Fi are the most common standards adopted for wearables
connectivity to the Internet.

## References

  * [1] L. D. Xu, W. He, and S. Li, “Internet of things in industries: A survey,” _IEEE Transactions on Industrial Informatics_ , vol. 10, no. 4, pp. 2233–2243, 2014.
  * [2] M. Chui, J. Manyika, and M. Miremadi, “Four fundamentals of workplace automation,” _McKinsey Quarterly_ , vol. 29, no. 3, pp. 1–9, 2015.
  * [3] A. Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari, and M. Ayyash, “Internet of things: A survey on enabling technologies, protocols, and applications,” _IEEE Communications Surveys & Tutorials_, vol. 17, no. 4, pp. 2347–2376, 2015.
  * [4] N. Niknejad, W. B. Ismail, A. Mardani, H. Liao, and I. Ghani, “A comprehensive overview of smart wearables: The state of the art literature, recent advances, and future challenges,” _Engineering Applications of Artificial Intelligence_ , vol. 90, p. 103529, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0952197620300348
  * [5] _Forecast unit shipments of wrist-worn wearables worldwide from 2019 to 2024_. [Online]. Available: https://healthtechinsider.com/2020/10/26/idc-forecasts-double-digit-weables-growth-through-2024/
  * [6] D. Nahavandi, R. Alizadehsani, A. Khosravi, and U. R. Acharya, “Application of artificial intelligence in wearable devices: Opportunities and challenges,” _Computer Methods and Programs in Biomedicine_ , vol. 213, p. 106541, 2022\. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0169260721006155
  * [7] _What is the Internet of Things (IoT)_. [Online]. Available: https://www.tibco.com/reference-center/what-is-the-internet-of-things-iot
  * [8] J. Pourbemany, Y. Zhu, and R. Bettati, “A survey of wearable devices pairing based on biometric signals,” 2021.
  * [9] S. D. Mamdiwar, A. R, Z. Shakruwala, U. Chadha, K. Srinivasan, and C.-Y. Chang, “Recent advances on iot-assisted wearable sensor systems for healthcare monitoring,” _Biosensors_ , vol. 11, no. 10, 2021. [Online]. Available: https://www.mdpi.com/2079-6374/11/10/372
  * [10] S. R. Haghi Mostafa, Thurow Kerstin, “Wearable devices in medical internet of things: Scientific research and commercially available devices,” _hir_ , vol. 23, no. 1, pp. 4–15, 2017. [Online]. Available: http://www.e-sciencecentral.org/articles/?scid=1075870
  * [11] Y. Xue, “A review on intelligent wearables: Uses and risks,” _Human Behavior and Emerging Technologies_ , vol. 1, no. 4, pp. 287–294, 2019. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/hbe2.173
  * [12] R. Sanchez-Iborra, “Lpwan and embedded machine learning as enablers for the next generation of wearable devices,” _Sensors_ , vol. 21, no. 15, 2021. [Online]. Available: https://www.mdpi.com/1424-8220/21/15/5218
  * [13] J. Passos, S. I. Lopes, F. M. Clemente, P. M. Moreira, M. Rico-González, P. Bezerra, and L. P. Rodrigues, “Wearables and internet of things (iot) technologies for fitness assessment: A systematic review,” _Sensors_ , vol. 21, no. 16, 2021. [Online]. Available: https://www.mdpi.com/1424-8220/21/16/5418
  * [14] Y. Bello, E. Figetakis, A. Refaey, and P. Spachos, “Continuous integration and continuous delivery framework for sds,” in _2022 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE)_ , 2022, pp. 406–410.
  * [15] Y. Bello, A. R. Hussein, M. Ulema, and J. Koilpillai, “On sustained zero trust conceptualization security for mobile core networks in 5g and beyond,” _IEEE Transactions on Network and Service Management_ , vol. 19, no. 2, pp. 1876–1889, 2022.
  * [16] Y. Bello, A. A. Abdellatif, M. S. Allahham, A. R. Hussein, A. Erbad, A. Mohamed, and M. Guizani, “B5g: Predictive container auto-scaling for cellular evolved packet core,” _IEEE Access_ , vol. 9, pp. 158 204–158 214, 2021.
  * [17] Y. Bello, M. S. Allahham, A. Refaey, A. Erbad, A. Mohamed, and N. Abdennadher, “B5g: Predictive container auto-scaling for cellular evolved packet core,” in _2021 IEEE International Conference on Communications Workshops (ICC Workshops)_ , 2021, pp. 1–7.
  * [18] J. Singh, Y. Bello, A. R. Hussein, A. Erbad, and A. Mohamed, “Hierarchical security paradigm for iot multiaccess edge computing,” _IEEE Internet of Things Journal_ , vol. 8, no. 7, pp. 5794–5805, 2021.
  * [19] E. Figetakis and A. Refaey, “Uav path planning using on-board ultrasound transducer arrays and edge support,” in _2021 IEEE International Conference on Communications Workshops (ICC Workshops)_ , 2021, pp. 1–6.
  * [20] _IEC TC 124 Strategic Business Plan_. [Online]. Available: https://www.iec.ch/public/miscfiles/sbp/124.pdf
  * [21] _IEC TC 124 Strategic Business Plan_. [Online]. Available: https://www.iec.ch/dyn/www/f?p=103:103:0::::FSP_ORG_ID,FSP_LANG_ID:12601,25
  * [22] S. Seneviratne, Y. Hu, T. Nguyen, G. Lan, S. Khalifa, K. Thilakarathna, M. Hassan, and A. Seneviratne, “A survey of wearable devices and challenges,” _IEEE Communications Surveys & Tutorials_, vol. 19, no. 4, pp. 2573–2620, 2017.
  * [23] F. J. Dian, R. Vahidnia, and A. Rahmati, “Wearables and the internet of things (iot), applications, opportunities, and challenges: A survey,” _IEEE Access_ , vol. 8, pp. 69 200–69 211, 2020.
  * [24] C. Nave and O. Postolache, “Smart walker based iot physical rehabilitation system,” in _2018 International Symposium in Sensing and Instrumentation in IoT Era (ISSI)_. IEEE, 2018, pp. 1–6.
  * [25] G. Yang, J. Deng, G. Pang, H. Zhang, J. Li, B. Deng, Z. Pang, J. Xu, M. Jiang, P. Liljeberg _et al._ , “An iot-enabled stroke rehabilitation system based on smart wearable armband and machine learning,” _IEEE journal of translational engineering in health and medicine_ , vol. 6, pp. 1–10, 2018.
  * [26] B. Xu, L. Xu, H. Cai, L. Jiang, Y. Luo, and Y. Gu, “The design of an m-health monitoring system based on a cloud computing platform,” _Enterprise Information Systems_ , vol. 11, no. 1, pp. 17–36, 2017.
  * [27] B. Xu, L. Da Xu, H. Cai, C. Xie, J. Hu, and F. Bu, “Ubiquitous data accessing method in iot-based information system for emergency medical services,” _IEEE Transactions on Industrial informatics_ , vol. 10, no. 2, pp. 1578–1586, 2014.
  * [28] A. Ghorbel, S. Bouguerra, N. B. Amor, and M. Jallouli, “Cloud based mobile application for remote control of intelligent wheelchair,” in _2018 14th International Wireless Communications & Mobile Computing Conference (IWCMC)_. IEEE, 2018, pp. 1249–1254.
  * [29] M. O. Agyeman and A. Al-Mahmood, “Design and implementation of a wearable device for motivating patients with upper and/or lower limb disability via gaming and home rehabilitation,” in _2019 Fourth International Conference on Fog and Mobile Edge Computing (FMEC)_. IEEE, 2019, pp. 247–252.
  * [30] M. Nyan, F. E. Tay, and E. Murugasu, “A wearable system for pre-impact fall detection,” _Journal of biomechanics_ , vol. 41, no. 16, pp. 3475–3481, 2008\.
  * [31] W. Min, H. Cui, H. Rao, Z. Li, and L. Yao, “Detection of human falls on furniture using scene analysis based on deep learning and activity characteristics,” _IEEE Access_ , vol. 6, pp. 9324–9335, 2018.
  * [32] P. Pierleoni, A. Belli, L. Palma, M. Pellegrini, L. Pernini, and S. Valenti, “A high reliability wearable device for elderly fall detection,” _IEEE Sensors Journal_ , vol. 15, no. 8, pp. 4544–4553, 2015.
  * [33] K. Ozcan, S. Velipasalar, and P. K. Varshney, “Autonomous fall detection with wearable cameras by using relative entropy distance measure,” _IEEE Transactions on Human-Machine Systems_ , vol. 47, no. 1, pp. 31–39, 2016.
  * [34] K. Ozcan, A. Mahabalagiri, and S. Velipasalar, “Automatic fall detection and activity classification by a wearable camera,” in _Distributed Embedded Smart Cameras_. Springer, 2014, pp. 151–172.
  * [35] S. R. Dhole, A. Kashyap, A. N. Dangwal, and R. Mohan, “A novel helmet design and implementation for drowsiness and fall detection of workers on-site using eeg and random-forest classifier,” _Procedia Computer Science_ , vol. 151, pp. 947–952, 2019.
  * [36] J.-L. Reyes-Ortiz, L. Oneto, A. Samà, X. Parra, and D. Anguita, “Transition-aware human activity recognition using smartphones,” _Neurocomputing_ , vol. 171, pp. 754–767, 2016.
  * [37] J. Qi, P. Yang, M. Hanneghan, S. Tang, and B. Zhou, “A hybrid hierarchical framework for gym physical activity recognition and measurement using wearable sensors,” _IEEE Internet of Things Journal_ , vol. 6, no. 2, pp. 1384–1393, 2018.
  * [38] R. C. Shit, S. Sharma, D. Puthal, and A. Y. Zomaya, “Location of things (lot): A review and taxonomy of sensors localization in iot infrastructure,” _IEEE Communications Surveys & Tutorials_, vol. 20, no. 3, pp. 2028–2061, 2018.
  * [39] S. Tomic, M. Beko, and R. Dinis, “Rss-based localization in wireless sensor networks using convex relaxation: Noncooperative and cooperative schemes,” _IEEE Transactions on Vehicular Technology_ , vol. 64, no. 5, pp. 2037–2050, 2014.
  * [40] M. Rana and V. Mittal, “Wearable sensors for real-time kinematics analysis in sports: A review,” _IEEE Sensors Journal_ , vol. 21, no. 2, pp. 1187–1207, 2021.
  * [41] Bosch Sensortec, “BMA150-Magnetometer,” https://www.bosch-sensortec.com/products/motion-sensors/magnetometers-bmm150/, accessed: 2022-7-23.
  * [42] ——, “BMA400-Accelerometer,” https://www.bosch-sensortec.com/products/motion-sensors/accelerometers/bma400/, accessed: 2022-7-23.
  * [43] ——, “BMG250-Gyroscope,” https://www.bosch-sensortec.com/products/motion-sensors/gyroscopes-bmg250/, accessed: 2022-7-23.
  * [44] Zepp Labs, “Zepp-Golf,” http://www.zepplabs.com/en-us/golf/zepp-sensor-specifications/, accessed: 2022-7-23.
  * [45] Kinexon, “Kinexon Perform,” https://kinexon.com/solutions/basketball/, accessed: 2022-7-23.
  * [46] C. Li, L. Yu, and S. Fei, “Real-time 3d motion tracking and reconstruction system using camera and imu sensors,” _IEEE Sensors Journal_ , vol. 19, no. 15, pp. 6460–6466, 2019.
  * [47] Honeywell, “AWM3000 series airflow sensors,” https://sps.honeywell.com/ca/en/products/advanced-sensing-technologies/healthcare-sensing/airflow-sensors/amplified/awm3000-series, accessed: 2022-7-23.
  * [48] Hillrom, “Life 2000 Ventilation,” https://www.hillrom.com/en/products/life2000-system/life2000-overview/, accessed: 2022-7-23.
  * [49] NXP, “MPX-5010,” https://www.nxp.com/products/sensors/pressure-sensors/differential-gauge-up-to-10-kpa/0-to-10kpa-differential-and-gauge-integrated-pressure-sensor:MPXx5010, accessed: 2022-7-23.
  * [50] Inogen, “G3-System,” https://www.inogen.com/products/g3-systems/, accessed: 2022-7-23.
  * [51] Philips, “DreamStation-2,” https://www.usa.philips.com/healthcare/product/HCEUX520H15C/dreamstation-2-auto-cpap-advanced-auto-cpap, accessed: 2022-7-23.
  * [52] R. Dudde, T. Vering, G. Piechotta, and R. Hintsche, “Computer-aided continuous drug infusion: setup and test of a mobile closed-loop system for the continuous automated infusion of insulin,” _IEEE Transactions on Information Technology in Biomedicine_ , vol. 10, no. 2, pp. 395–402, 2006.
  * [53] ProtoCentral, “NMAX30100,” https://protocentral.com/product/pulse-pulse-ox-heart-rate-sensor-based-on-max30102/, accessed: 2022-7-23.
  * [54] Insignia, “NS-PSOXMW1,” https://www.insigniaproducts.com/pdp/NS-PSOXMW1/6438023, accessed: 2022-7-23.
  * [55] NeuroSky, “MindWave,” https://www.sparkfun.com/products/14758, accessed: 2022-7-23.
  * [56] Muse, “Muse-2,” https://choosemuse.com/muse-2/, accessed: 2022-7-23.
  * [57] Dexcom, “G6 CGM,” https://ca.store.dexcom.com/en-CA/dexcom-g6/g6-sensor-single/STS-GS-008.html?gclid=EAIaIQobChMI967Ig6qd-QIVZYJbCh0NKASBEAQYASABEgJvRPD_BwE, accessed: 2022-7-23.
  * [58] Mikroe, “ECG-6,” https://www.sparkfun.com/products/19480, accessed: 2022-7-23.
  * [59] Apple, “Apple-Watch,” https://www.apple.com/ca/shop/buy-watch/apple-watch, accessed: 2022-7-23.
  * [60] Fitbit, “Sense-SmartWatch,” https://www.fitbit.com/global/en-ca/products/smartwatches/sense?sku=512SRSG, accessed: 2022-7-23.
  * [61] Oura, “Oura Ring Generation 3-Smart Ring,” https://ouraring.com/product/heritage-silver, accessed: 2022-7-23.
  * [62] F. Wu, C. Rüdiger, J.-M. Redouté, and M. R. Yuce, “We-safe: A wearable iot sensor node for safety applications via lora,” in _2018 IEEE 4th World Forum on Internet of Things (WF-IoT)_ , 2018, pp. 144–148.
  * [63] CozIR, “GC-0012,” https://datasheets.globalspec.com/ds/15/CO2Meter/2E9DD87D-DD7B-4D2A-BCAC-1EAD9AE41EBD, accessed: 2022-7-23.
  * [64] Bosch Sensortec, “BM280,” https://www.bosch-sensortec.com/products/environmental-sensors/humidity-sensors-bme280/, accessed: 2022-7-23.
  * [65] Adafruit, “SI1145,” https://www.adafruit.com/product/1777, accessed: 2022-7-23.
  * [66] Spec Sensors, “968-001,” https://www.digikey.com/en/products/detail/spec-sensors-llc/968-001/6136365, accessed: 2022-7-23.
  * [67] S. Mohanty, A. Tripathy, and B. Das, “An overview of a low energy uwb localization in iot based system,” in _2021 International Symposium of Asian Control Association on Intelligent Robotics and Industrial Automation (IRIA)_ , 2021, pp. 293–296.
  * [68] Adafruit, “PA1616D,” https://www.adafruit.com/product/5186, accessed: 2022-7-23.
  * [69] _ZigBee White Paper_. [Online]. Available: https://www.yumpu.com/en/document/view/65594940/zigbee-white-paper
  * [70] K. Lounis and M. Zulkernine, “Attacks and defenses in short-range wireless technologies for iot,” _IEEE Access_ , vol. 8, pp. 88 892–88 932, 2020\.
  * [71] P. S. Henry and H. Luo, “Wifi: what’s next?” _IEEE Communications Magazine_ , vol. 40, no. 12, pp. 66–72, 2002.
  * [72] R. Gravina and G. Fortino, “Wearable body sensor networks: state-of-the-art and research directions,” _IEEE Sensors Journal_ , vol. 21, no. 11, pp. 12 511–12 522, 2020.
  * [73] F. Ahamed and F. Farid, “Applying internet of things and machine-learning for personalized healthcare: Issues and challenges,” in _2018 International Conference on Machine Learning and Data Engineering (iCMLDE)_ , 2018, pp. 19–21.
  * [74] A. Balakrishnan, J. Medikonda, and P. K. Namboothiri, “Analysis of the effect of muscle fatigue on gait characteristics using data acquired by wearable sensors,” in _2020 IEEE International Conference on Distributed Computing, VLSI, Electrical Circuits and Robotics (DISCOVER)_ , 2020, pp. 137–140.
  * [75] Y.-T. Lee, W.-H. Hsiao, Y.-S. Lin, and S.-C. T. Chou, “Privacy-preserving data analytics in cloud-based smart home with community hierarchy,” _IEEE Transactions on Consumer Electronics_ , vol. 63, no. 2, pp. 200–207, 2017.
  * [76] _Wearable Tech Sales Explored: Just How Many Devices Have Been Sold?_ [Online]. Available: https://www.wareable.com/wearable-tech/wearable-tech-sales-explored-7456


# DeFlow: Decoder of Scene Flow Network in Autonomous Driving

Qingwen Zhang1, Yi Yang1,2, Heng Fang1, Ruoyu Geng3, Patric Jensfelt1 1Authors
are with the Division of Robotics, Perception, and Learning (RPL), KTH Royal
Institute of Technology, Stockholm 114 28, Sweden. (email:
qingwen@kth.se)2Authors are with Research and Development, Scania CV AB,
Södertälje 151 87, Sweden.3Authors are with Robotics Institute, The Hong Kong
University of Science and Technology, Hong Kong SAR, China.

###### Abstract

Scene flow estimation determines a scene’s 3D motion field, by predicting the
motion of points in the scene, especially for aiding tasks in autonomous
driving. Many networks with large-scale point clouds as input use voxelization
to create a pseudo-image for real-time running. However, the voxelization
process often results in the loss of point-specific features. This gives rise
to a challenge in recovering those features for scene flow tasks. Our paper
introduces DeFlow which enables a transition from voxel-based features to
point features using Gated Recurrent Unit (GRU) refinement. To further enhance
scene flow estimation performance, we formulate a novel loss function that
accounts for the data imbalance between static and dynamic points. Evaluations
on the Argoverse 2 scene flow task reveal that DeFlow achieves state-of-the-
art results on large-scale point cloud data, demonstrating that our network
has better performance and efficiency compared to others. The code is open-
sourced at https://github.com/KTH-RPL/deflow.

##  I Introduction

Scene flow estimation, which determines the 3D motion field of a scene, is
essential in the field of autonomous driving. By imitating human behavior when
navigating in complex scenes using motion cues, accurate scene flow
predictions empower autonomous vehicles (AVs) to interpret and navigate in
dynamic environments. Such precise estimations further enhance downstream
tasks in AVs, encompassing detection, segmentation, tracking, and occupancy
flow.

Recent advancements [1, 2, 3] highlight the value of class-agnostic motion
estimations, which are derived directly from point clouds. If satisfactory
performance is guaranteed at point-level, the result of scene flow can be
effortlessly integrated as a prior for subsequent tasks like prediction and
detection [2]. This technique will potentially contribute to elevating the
efficiency and adaptability of autonomous driving systems in dynamic
scenarios.

Figure 1: LiDAR scene flow estimation using our DeFlow method on the Argoverse
2. The predicted scene flow for each point is color-coded based on direction,
with the color wheel anchored in the world frame. (a) Camera view for
visualization purposes only. (b)(c) Estimated LiDAR point clouds’ flow. Varied
colors represent different directions, with more saturated colors indicating
higher velocities. (b) Front view. (c) Bird’s-eye view.

Most methods [4, 5, 6] in object registration scene flow focus on relatively
small-scale point cloud data like synthetic datasets Shapenet [7] and
FlyingThing3D [8]. When they use point cloud data in autonomous driving [9,
10], the points are downsampled to a size of 8,192 points or less. These
methods fail, due to memory overflow on modern driving datasets with the full
number of points as input. Datasets like Argoverse2 [11] and Waymo [9] are
closer to the real autonomous vehicles’ sensor setup, where the number of
points in one frame is around 80k-177k. Recently, some methods [12, 13] have
employed Multi-Layer Perceptrons (MLPs) to optimize proposed self-supervised
objective functions that can successfully run on various sizes of datasets.
However, their runtimes extend from 26 to 35 seconds per frame [14]. In the
field of autonomous driving, real-time capability is important. Consequently,
these optimization-based methods fall short of practicality.

Given the necessity to process and estimate scene flow on full large point
cloud datasets in real-time, FastFlow3D [9] emerges as a practical solution.
An essential strategy to achieve the real-time requirement is voxelization. It
is a popular point cloud processing technique, particularly for detection
tasks [15, 16, 17]. However, there is a distinct difference between detection
and scene flow tasks: the latter necessitates point-level results.
Voxelization-based methods often fail to realize the importance of the decoder
design in the scene flow task, resulting in their inability to differentiate
features among points contained within the same voxel. This is because all
points within a given voxel inherit the same features from the convolutional
network.

Addressing these challenges, we present DeFlow which conducts the Gated
Recurrent Unit (GRU) refinement module to reconstruct the different features
of points inside the same voxel, markedly improving the final results. We
evaluated our method using the Argoverse 2 scene flow task, and it achieved
state-of-the-art results on the online leaderboard, leveraging a training set
of 100k labeled frames. An example is shown in Fig. 1. Our approach is
available open-source at https://github.com/KTH-RPL/deflow. In summary, our
primary contributions include:

  * •

The introduction of a novel real-time network that integrates GRU with
iterative refinement in the decoder design, effectively transitioning from
voxel to point features.

  * •

The proposal of a new loss type optimized for imbalanced data distribution on
static and dynamic points.

  * •

Achieving state-of-the-art results on the large-scale point cloud dataset
Argoverse 2 online leaderboard.

##  II Related work

Introduced by Vedula et al. [18], scene flow estimation captures the 3D point
motion field of a scene. This concept evolved from the two-dimensional optical
flow estimation, a classical topic that predicts apparent motion patterns in
2D images.

Existing methodologies can be broadly categorized into optimization-based and
learning-based approaches. A prominent example within the optimization
category is NSFP [12]. This method utilizes MLPs to refine the flow,
leveraging the chamfer distance as a metric. Such methods are typically not
categorized as learning-based approaches since they do not save learned
weights to infer on other frames. Instead, they employ MLPs to optimize each
frame. The considerable inference time of NSFP makes it impractical for real-
time applications.

Recent 3D learning-based approach [5, 4, 19, 20] can be traced back to 2D
approaches [21, 22, 23]. RAFT [21], a notable work in optical flow is a good
representative. It constructs a multi-scale 4D correlation volume for pixel
pairs by iteratively updating the flow field through a recurrent unit. Based
on it, PV-RAFT [5], DPV-RAFT [24] adapt their frameworks for 3D point cloud
data. However, these methods become computationally challenging for recent
large-scale points due to memory overflow. The issue often happened during the
construction of distance matrix [5, 24] or correlation matrix [4] that grow
exponentially with the number of points.

Figure 2:  Histogram of moving distances in 0.1$\mathrm{~{}s}$ for all dynamic
points across all scenes in the Argoverse 2 validation dataset
(10$\mathrm{~{}Hz}$). The x-axis represents the distance in meters, ranging
from 0.05 to 2.0 meters. The y-axis indicates the number of points for each
distance range. The dynamic points are densely distributed within 0.2 meters.

Fortunately, there are some encoder designs for processing large point clouds
in other point cloud related tasks like detection [15, 16, 17], and
segmentation [25, 26]. PointPillar proposed by Lang et al. [17], stands out as
one of the most favored encoders due to its impressive performance combined
with high efficiency. The approach, which transforms points to voxels,
creating pseudo-images for convolutional networks, is inspired by the 2D
method: FlowNet [27], the pioneering CNN for optical flow estimation, employs
a U-Net autoencoder architecture. In scene flow task, FastFlow3D [9], a
network that can run on large point cloud data (80k - 177k points) in real-
time, also uses PointPillar as its encoder. However, the FastFlow3D decoder
struggles to differentiate features among points within the same voxel, a
limitation attributed to voxelization in PointPillar. Using the PointPillar
approach can improve efficiency but might reduce performance if the design of
the decoder is not thought out well.

It is applicable to adapt the decoder design developed in optical flow to the
ones in the scene flow task. Previous networks used in optical flow like PWC-
Net [28] employ a context network to expand the receptive field size of
outputs, refining flow through dilated convolutions [29]. Moreover, IRR [30]
offers iterative and residual refinement, achieving enhanced accuracy without
enlarging the network. Feihu et al. [31] utilizes the Gated Recurrent Unit
(GRU) for refinement, iterating multiple times to estimate flow. Following the
line of the works in optical flow, we adopt GRU in our method to emphasize the
transition from voxel to point features in the 3D point scene flow task.

Figure 3: DeFlow Architecture. The feature-extracting step, derived from
PointPillars, takes two consecutive point clouds as input and transforms them
into voxels. The encoder utilizes a convolutional U-Net backbone. Our novel
decoder merges the encoder output with the point offset from PointPillars,
employing a GRU for refinement. This process reconstructs the voxel-to-point
information, ultimately producing the flow result.

##  III Problem Statement

Our research tackles the challenge of real-time scene flow estimation in
autonomous driving. Given two sequential input point clouds, $\mathcal{P}_{t}$
and $\mathcal{P}_{t+1}$, captured at times $t$ and $t+1$ respectively, along
with the ego movement as the transformation matrix $\mathbf{T}_{t,t+1}$, the
objective is to predict the motion vector as flow
$\hat{\mathbf{F}}_{t,t+1}(p)=(x,y,z)^{T}$ for each point
$p\in\mathcal{P}_{t}$.

Knowing the frequency of our sensor data collection (10$\mathrm{~{}Hz}$), it
becomes straightforward to interpret the flow as velocity. The overarching
goal is to minimize the End Point Error (EPE) which presents the discrepancy
between the predicted flow and the ground truth flow, as expressed by the
following equation:

$\min\underbrace{\frac{1}{\left\|\mathcal{P}_{t}\right\|}\sum_{p\in\mathcal{P}_{t}}\left\|\hat{\mathbf{F}}(p)-\mathbf{F}_{gt}(p)\right\|_{2}}_{\text{EPE}}.$
(1)

##  IV Approach

Our methodology builds upon the overall pipeline of FastFlow3D [9], which was
designed specifically for large point cloud data. Briefly, it first voxelizes
points and formats them into a bird-eye-view grid feature with a specified
resolution. Through convolutional layers, the network efficiently learns
features within these voxels.

Based on our analysis of Argoverse 2 (as shown in Fig. 2), we observed that
the dynamic points are densely distributed within the range of 0.05 to
0.2$\mathrm{m}$ movement. Consequently, the choice of resolution is important,
as demonstrated in Section V-B. However, a higher resolution also leads to an
increase in computational demands, making it impractical for low-computational
devices. If a coarse resolution is chosen, the design of the decoder becomes
crucial to differentiate point features within the same voxel to achieve
results on par with those of a finer resolution.

In the subsequent sections, we outline our framework as depicted in Fig. 3 and
highlight the areas of focus and improvement.

###  IV-A Input and Output

Leveraging HD maps provided by most autonomous driving datasets [32, 11], or
by deploying ground segmentation techniques like [33], we can easily exclude
ground points from both $\mathcal{P}_{t}$ and $\mathcal{P}_{t+1}$.

The flow $\hat{\mathbf{F}}$ from $\mathcal{P}_{t}$ to $\mathcal{P}_{t+1}$ is
decomposed into two parts as following:

$\hat{\mathbf{F}}=\mathbf{F}_{ego}+\Delta\hat{\mathbf{F}},$ (2)

where $\mathbf{F}_{ego}$ is the flow resulting in ego vehicle’s motion which
can be directly obtained from $\mathbf{T}_{t,t+1}$, and
$\Delta\hat{\mathbf{F}}$ is our network output.

###  IV-B Encoder and Backbone

For point cloud rasterization, we use the dynamic voxelization technique from
PointPillars [16, 17] that improves the framework’s efficiency. We compute
each point’s offset from the pillar center and the cluster offset from the
point to its cluster coordinates. Post voxelization, a linear transformation
aggregates all points within a pillar.

After encoding $\mathcal{P}_{t}$ and $\mathcal{P}_{t+1}$ into grids, we use a
2D convolutional U-Net backbone. Both grids undergo processing through this
shared-weight backbone.

Method | EPE 3-Way ↓ | EPE FD ↓ | EPE BS ↓ | EPE FS ↓ |  | Dynamic
---
IoU ↑
| Dynamic
---
AccRelax ↑
| Dynamic
---
AccStrict ↑
FastNSFP [34] | 0.1657 | 0.3540 | 0.1025 | 0.0406 | 0.0924 | 0.3729 | 0.1958
NSFP w Motion Comp [12] | 0.0685 | 0.1503 | 0.0248 | 0.0302 | 0.3199 | 0.6956 | 0.4537
Zeroflow [14] | Standard | 0.0814 | 0.2109 | 0.0080 | 0.0254 | 0.4791 | 0.4363 | 0.1873
XL | 0.0569 | 0.1440 | 0.0089 | 0.0178 | 0.5224 | 0.6106 | 0.3219
FastFlow3D [9] | 0.0782 | 0.2073 | 0.0020 | 0.0253 | 0.5760 | 0.4407 | 0.1965
DeFlow (Ours) | 0.0534 | 0.1340 | 0.0029 | 0.0232 | 0.6289 | 0.7213 | 0.4483
TABLE I:  Comparisons on Argoverse 2 sensor test set in the online leaderboard
[35]. Our methods achieve state-of-art performance in the scene flow task. The
main improvement happens in the accuracy of flow estimation on dynamic points,
with larger improvements in the Dynamic IoU and Dynamic Accuracy Relaxed. We
bold the best results and underline the second best results.

###  IV-C Decoder

The process of obtaining point-wise flow in FastFlow3D is facilitated by the
unpillar operation. This operation, for each point, retrieves the associated
flow embedding grid cell, appends the point feature, and employs a multi-layer
perceptron to deduce the flow vector. However, as previously highlighted, this
approach is not appropriately designed for the reconstruction of voxel-to-
point features.

As illustrated in Fig. 3, prior to the decoder’s operation, we concatenate the
pillar features of $\mathcal{P}_{t}$ with the output of the U-Net features,
resulting in a format of $\mathbf{M}_{vector}\in\mathbb{R}^{N\times C}$. Here,
$N$ represents the number of points in $\mathcal{P}_{t}$, and $C$ denotes the
number of feature channels post concatenation. We observed that simply
concatenating the $C$ channels common to all points in a voxel with the 3
channels dedicated to point offset led to an imbalance. For the points in the
same voxel, the majority of the channels are identical.

An intuitive solution to this imbalance is to expand the point offset features
to match the dimensionality of $C$. However, this modification worsen the
original performance, proving that a dedicated network design is necessary.

Drawing inspiration from the 2D optical flow techniques that utilize GRU [31],
we propose an alternative method in 3D scene flow depicted in Fig. 3 decoder.
In this approach, we designate $\mathbf{M}_{vector}$ as the first hidden
states, denoted as $\mathbf{H}_{0}$. By employing a linear layer, we expand
the point offsets and set them as input, represented as $\mathbf{x}$. The
relationship between these components is captured by:

$\mathbf{H}_{t}=\mathbf{Z}_{t}\odot\mathbf{H}_{t-1}+\left(1-\mathbf{Z}_{t}\right)\odot\tilde{\mathbf{H}}_{t},$
(3)

where $\mathbf{Z}_{t}$ serves as the update gate. It takes $\mathbf{x}$ as its
input, which subsequently undergoes processing via 1D convolution layers,
utilizing a Sigmoid activation function. The term $\mathbf{H}_{t-1}$
represents the previous hidden state. For the initial instance,
$\mathbf{H}_{0}$ is set to $\mathbf{M}_{vector}$. $\tilde{\mathbf{H}}_{t}$
stands for the candidate hidden state, which is determined by the reset gate
and the model weights.

To maintain a small model without inflating the number of parameters, we
employ multiple iterations within GRU layers. After completing these
iterations, the most recent hidden state $\mathbf{H}_{t}$ is concatenated with
the point offset features. This combined entity then proceeds through MLPs,
resulting in the generation of the final delta flow, denoted as
$\Delta\hat{\mathbf{F}}$ as shown in Fig. 3. A comprehensive ablation study
detailing the intricacies and performance of our decoder design is presented
in Section V-B.

###  IV-D Loss Function

The task of scene flow estimation in autonomous driving scenarios is
inherently challenging due to the dynamic nature of the environment. A
significant portion of the LiDAR points, which reflect static structures such
as buildings or roads, remain stationary. This leads to a label imbalance in
the dataset, with more background static points than others. To address this,
the loss function incorporates a scaling function, denoted as $\sigma(p)$, to
balance the contribution of each point based on its motion characteristics:

$\mathcal{L}=\frac{1}{\left\|\mathcal{P}_{t}\right\|}\sum_{p\in\mathcal{P}_{t}}\sigma(p)\left\|\Delta\hat{\mathbf{F}}(p)-\Delta\mathbf{F}_{gt}(p)\right\|_{2},$
(4)

where $\left\|\mathcal{P}_{t}\right\|$ is the number of points in
$\mathcal{P}_{t}$.

FastFlow3D [9], in their experiments, introduced a scaling approach based on
the distinction between foreground and background points. The difference
between the two is determined by whether a point is contained within the
bounding box of any tracked object.

$\sigma(p)_{t}=\begin{cases}1&\text{ if }p\in\text{ Foreground }\\\ 0.1&\text{
if }p\in\text{ Background }\end{cases}$ (5)

With the advent of self-supervised learning, where labels distinguishing
foreground and background are absent, Zeroflow [14] proposed an alternative
scaling function. This function scales based on the speed (flow) of the
point’s motion:

$\sigma(p)_{s}=\begin{cases}0.1&\text{ if }s(p)<0.4\mathrm{~{}m}/\mathrm{s}\\\
1.0&\text{ if }s(p)>1.0\mathrm{~{}m}/\mathrm{s}\\\ 1.8s-0.8&\text{ o.w.
}\end{cases}$ (6)

Building upon the insights from Zeroflow, we propose a nuanced scaling
approach that takes into account the distribution of dynamic and static point
numbers. This approach divides $\mathcal{P}_{t}$ into three categories based
on their motion speed
$\\{\mathcal{P}_{t/1},\mathcal{P}_{t/2},\mathcal{P}_{t/3}\\}$, as defined in
Eq. 6. The total loss is then the sum of the losses from these three
categories:

$\displaystyle\mathcal{L}_{total}=\sum_{i=1}^{3}\frac{1}{\left\|\mathcal{P}_{t/i}\right\|}\sum_{p\in\mathcal{P}_{t/i}}\left\|\Delta\hat{\mathbf{F}}(p)-\Delta\mathbf{F}_{gt}(p)\right\|_{2}.$
(7)

This comprehensive loss function ensures that the model balances different
types of point motion, providing a robust estimation of scene flow in diverse
scenarios.

# Itr | EPE 3-way ↓ | EPE FD ↓ | EPE BS ↓ | EPE FS ↓
---|---|---|---|---
2 | 0.0528 | 0.1222 | 0.0055 | 0.0308
4 | 0.0516 | 0.1212 | 0.0047 | 0.0289
8 | 0.0517 | 0.1214 | 0.0042 | 0.0295
16 | 0.0532 | 0.1262 | 0.0038 | 0.0297
TABLE II: Ablation study on GRU iteration count. Analyzing the performance impact across varying GRU iteration numbers, with 4 iterations emerging as the optimal configuration for DeFlow. Decoder Design |  Res ($\mathrm{m}$) | EPE 3-Way ↓ | EPE FD ↓ | EPE BS ↓ | EPE FS ↓ | Dynamic IoU ↑ |  GM ($\mathrm{MiB}$) ↓ | FPS ↑
---|---|---|---|---|---|---|---|---
FastFlow3D [9] | 0.4 | 0.1116 | 0.3055 | 0.0037 | 0.0254 | 0.4701 | 2114 | 49.71
0.2 | 0.0852 | 0.2326 | 0.0025 | 0.0206 | 0.5257 | 2874 | 29.17
0.1 | 0.0586 | 0.1463 | 0.0086 | 0.0208 | 0.5238 | 6634 | 11.31
Our w/o GRU | 0.2 | 0.0916 | 0.2499 | 0.0034 | 0.0216 | 0.5224 | 2876 | 28.99
Ours | 0.2 | 0.0564 | 0.1309 | 0.0045 | 0.0337 | 0.4896 | 2878 | 20.49
TABLE III: Decoder design and resolution impact on Argoverse 2 sensor
validation set. The table contrasts results based on different decoder designs
and voxel resolutions. ‘Res’ indicates voxel resolution, ‘GM’ denotes GPU
memory consumption during model execution, and ‘FPS’ (frame per second)
signifies the model’s running speed. The findings reveal that the GRU at a
0.2$\mathrm{~{}m}$ resolution overperformance the original 0.1$\mathrm{~{}m}$
resolution decoder with reduced GPU memory requirements and faster running
speed.

##  V Experiment

In this section, we outline our experimental setup, followed by a series of
ablation studies to understand the contributions of our approach. We then
present quantitative comparisons with state-of-the-art methods on a benchmark
dataset and conclude with qualitative visualization results.

###  V-A Experiment Setup

Dataset: Our approach is evaluated on the large-scale autonomous driving data,
Argoverse 2 [11] which encompasses a variety of sets, including Sensor and
LiDAR. Given that the LiDAR dataset lacks imagery and any other annotations,
our primary focus is on the Sensor dataset. The Sensor dataset encompasses 700
training and 150 validation scenes. Each scene is approximately 15 seconds
long in 10$\mathrm{~{}Hz}$, complete with annotations. Argoverse 2 provides an
online benchmark with 150 testing scenes.

Methods: Our main comparison is with FastFlow3D [9] on the validation dataset.
We reproduce this method based on Zeroflow [14], given that FastFlow3D serves
as its student network and its code is publicly available. To showcase our
method’s superiority, we display results of various baseline methods from the
online leaderboard, including NSFP [12], FastNSF [34], and Zeroflow [14].
NSFP, FastNSF operates on a self-supervised paradigm. Both FastFlow3D and our
Deflow employ supervised learning techniques, leveraging the Sensor dataset.
Zeroflow adopts a semi-supervised strategy, utilizing NSFP for dataset
labeling. While the standard version relies solely on the Sensor dataset, the
XL version incorporates additional data from the LiDAR dataset, amounting to
twice the original size. Furthermore, the XL version sets the resolution to
$0.1\mathrm{~{}m}$ and boosts a model with ten times the parameters of the
standard version.

Implementation Details: In Table I for the Argoverse 2 test dataset, our
DeFlow implementation is as follows: We use four GRU iterations (as shown
Table II), and the model trains for a total of 50 epochs with a batch size of
80. The network optimization leverages the Adam optimizer with a learning rate
set at $2\times 10^{-6}$. The chosen loss function is detailed in Eq. 7. The
resolution is set as $0.2\mathrm{~{}m}$, consquently, the $[512,512]$ grid is
transformed from a $102.4\mathrm{~{}m}\times 102.4\mathrm{~{}m}$ map. Other
benchmarking methods in detail: FastNSFP uses their official public code with
config and FastFlow3D trains in 50 epochs to converge and 64 batch size [9] as
same as their paper’s setup. Others (ZeroFlow and NSFP) are public on the
online leaderboard. For local experiments, aimed at a fair ablation study, all
models are trained with a learning rate of $2\times 10^{-6}$ and the same
batch size of 80. We maintain strict control over the resolution variable to
see the influence of resolution, loss function, and network design on
performance. All local experiments were executed on a desktop powered by an
Intel® Core™ i9-12900KF and equipped with a GeForce RTX 3090 GPU.

Metric: The Argoverse 2 benchmark adopts a unified metric proposed by [13],
which introduces the 3-way Endpoint Error (EPE). EPE, as defined in Eq. 1,
measures the L2 norm of the discrepancy between the predicted and actual flow
vectors, expressed in meters. The 3-way calculates the unweighted average EPE
across three classifications: Foreground Dynamic (FD), Foreground Static (FS),
and Background Static (BS). If the flow of a point $\hat{\mathbf{F}}(p)$
exceeds $0.05\mathrm{~{}m}$, the point is defined as dynamic. Given the
dataset’s 10 Hz collection frequency, this threshold corresponds to a speed of
$0.5\mathrm{~{}m}/\mathrm{s}$. The Dynamic Accuracy Relaxed (Dynamic AccRelax)
metric captures the proportion of dynamic points with an EPE under
$0.1\mathrm{~{}m}$ or a relative error below 10%. In contrast, the Strict
(Dynamic AccStrict) metric requires an EPE under $0.05\mathrm{~{}m}$ or a
relative error of less than 5%.

###  V-B Quantitative Results

In this section, we evaluate the efficacy of our proposed Deflow approach and
compare it with alternative methods, highlighting the effects of different
design choices on performance.

As referenced in Table I and briefly discussed in Section V-A, it’s evident
that our method achieves the state-of-art performance in the Argoverse 2 scene
flow task. The most significant enhancement is observed in the boosted
accuracy of dynamic point flow estimation, with only minimal errors occurring
in static points.

Figure 4: Qualitative results from the validation dataset. The top row displays the ground truth flow, the middle row presents the FastFlow3D result, and the bottom row showcases the DeFlow outcomes. DeFlow estimates closely match the ground truth flow in both speed and angle. As highlighted in the two green circles, our DeFlow method demonstrates better performance in predicting motion angle (indicated by color variations) and speed (represented by color intensity) compared to FastFlow3D. The color wheel has been adjusted to align with the ego vehicle’s forward direction. Loss Type | EPE 3-way | EPE FD | EPE BS | EPE FS
---|---|---|---|---
FastFlow3D Eq. 5 | 0.0852 | 0.2326 | 0.0025 | 0.0206
Zeroflow Eq. 6 | 0.0843 | 0.2112 | 0.0201 | 0.0202
Ours Eq. 7 | 0.0787 | 0.2045 | 0.0041 | 0.0277
TABLE IV: Ablation study on loss types. Our proposed loss function outperforms
others, achieving the best EPE 3-way score by addressing the imbalanced data
distribution between static and dynamic points.

To illustrate the efficacy of our network design and the impact of different
resolution settings, we evaluated a model without the GRU iteration module, as
depicted in Fig. 3. Additionally, we tested both smaller and larger
resolutions in FastFlow3D. As shown in Table III, where all models use the
origin loss function Eq. 5, the results without the GRU module in the network
indicate that merely extending the point offset feature and grid feature
channels using MLPs can adversely affect the accuracy of dynamic point flow.
Conversely, our Deflow GRU framework significantly reduces the EPE 3-way and
enhances accuracy in both relaxed and strict dynamic metrics. When comparing
with FastFlow3D in $0.1\mathrm{~{}m}$ resolution setting, with larger GPU
memory consumption and slower FPS, our DeFlow GRU at $0.2\mathrm{~{}m}$
achieves better performance with efficiency on both GPU memory and computation
speed. Considering both performance and computational resources, our method is
more suitable for lightweight on-board computation in real-time. This outcome
further highlights the effectiveness of our network design.

The design of the loss function plays an important role in training the
network, as discussed in Section IV-D. We conducted an ablation study using
the original FastFlow3D to assess various loss types. Apart from the loss
functions, all other parameters were kept consistent: a training epoch of 50,
a learning rate of $2\times 10^{-6}$ using the Adam optimizer, and a
resolution of $0.2\mathrm{~{}m}$. The loss proposed by us, as presented in
Table IV, emerged as the superior choice, reducing the EPE FD error by 12.1%
compared to FastFlow3D.

###  V-C Qualitative Results

We showcase the qualitative outcomes of our DeFlow on the Argoverse 2
validation set with Fig. 4 serving as an example. From our observation, DeFlow
demonstrates the ability to accurately capture the motion flow in most cases.
When compared to FastFlow3D, it performs better in the prediction of both
speed and motion angle. In certain regions, particularly those that are
blocked or partially occluded, DeFlow occasionally exhibits inaccuracies,
indicating potential areas for further improvement.

##  VI Conclusion

In this paper, we introduce DeFlow, an efficient and high-performance method
for autonomous driving in large-scale point clouds. Our primary contributions
include the introduction of the DeFlow network, which enhances the extraction
and reconstruction of point-voxel-point network features at the point level.
Additionally, we propose a novel loss function to address the challenges of
imbalanced data distribution among points. Our experimental results underscore
the efficacy of our approach.

Future work could be on self-supervised exploration of DeFlow and the fusion
with multi-modality sensors, like cameras and radar. The flow of dynamic
objects is one that we are mainly focused on in scene flow estimation, so
there is a possible solution if we can segment static and dynamic [36] first,
then it can hugely decrease the computation burden for neural optimization-
based approaches.

## Acknowledgement

Thanks to one of ZeroFlow authors: Kyle Vedder who kindly discussed their
results with us and HKUST Ramlab’s member: Jin Wu who gave constructive
comments on this work. This work was partially supported by the Wallenberg AI,
Autonomous Systems and Software Program (WASP) funded by the Knut and Alice
Wallenberg Foundation and Prosense (2020-02963) funded by Vinnova. The
computations were enabled by the supercomputing resource Berzelius provided by
National Supercomputer Centre at Linköping University and the Knut and Alice
Wallenberg Foundation, Sweden.

## References

  * [1] C. Luo, X. Yang, and A. Yuille, “Self-supervised pillar motion learning for autonomous driving,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3183–3192.
  * [2] M. Najibi, J. Ji, Y. Zhou, C. R. Qi, X. Yan, S. Ettinger, and D. Anguelov, “Motion inspired unsupervised perception and prediction in autonomous driving,” in _European Conference on Computer Vision_. Springer, 2022, pp. 424–443.
  * [3] L. Schmid, O. Andersson, A. Sulser, P. Pfreundschuh, and R. Siegwart, “Dynablox: Real-time detection of diverse dynamic objects in complex environments,” _arXiv preprint arXiv:2304.10049_ , 2023.
  * [4] I. Lang, D. Aiger, F. Cole, S. Avidan, and M. Rubinstein, “Scoop: Self-supervised correspondence and optimization-based scene flow,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 5281–5290.
  * [5] Y. Wei, Z. Wang, Y. Rao, J. Lu, and J. Zhou, “PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds,” in _CVPR_ , 2021.
  * [6] Z. Wang, Y. Wei, Y. Rao, J. Zhou, and J. Lu, “3d point-voxel correlation fields for scene flow estimation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [7] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su _et al._ , “Shapenet: An information-rich 3d model repository,” _arXiv preprint arXiv:1512.03012_ , 2015.
  * [8] N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox, “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4040–4048.
  * [9] P. Jund, C. Sweeney, N. Abdo, Z. Chen, and J. Shlens, “Scalable scene flow from point clouds in the real world,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 1589–1596, 2021.
  * [10] M. Menze and A. Geiger, “Object scene flow for autonomous vehicles,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3061–3070.
  * [11] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays, “Argoverse 2: Next generation datasets for self-driving perception and forecasting,” in _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021)_ , 2021.
  * [12] X. Li, J. Kaesemodel Pontes, and S. Lucey, “Neural scene flow prior,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 7838–7851, 2021.
  * [13] N. Chodosh, D. Ramanan, and S. Lucey, “Re-evaluating lidar scene flow for autonomous driving,” _arXiv preprint arXiv:2304.02150_ , 2023.
  * [14] K. Vedder, N. Peri, N. Chodosh, I. Khatri, E. Eaton, D. Jayaraman, Y. Liu, D. Ramanan, and J. Hays, “Zeroflow: Fast zero label scene flow via distillation,” _arXiv preprint arXiv:2305.10424_ , 2023.
  * [15] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 652–660.
  * [16] Y. Zhou, P. Sun, Y. Zhang, D. Anguelov, J. Gao, T. Ouyang, J. Guo, J. Ngiam, and V. Vasudevan, “End-to-end multi-view fusion for 3d object detection in lidar point clouds,” in _Conference on Robot Learning_. PMLR, 2020, pp. 923–932.
  * [17] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 12 697–12 705.
  * [18] S. Vedula, P. Rander, R. Collins, and T. Kanade, “Three-dimensional scene flow,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 27, no. 3, pp. 475–480, 2005.
  * [19] Z. Lu and M. Cheng, “Gma3d: Local-global attention learning to estimate occluded motions of scene flow,” _arXiv preprint arXiv:2210.03296_ , 2022\.
  * [20] W. Cheng and J. H. Ko, “Bi-pointflownet: Bidirectional learning for point cloud based scene flow estimation,” in _European Conference on Computer Vision_. Springer, 2022, pp. 108–124.
  * [21] Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_. Springer, 2020, pp. 402–419.
  * [22] H. Xu, J. Zhang, J. Cai, H. Rezatofighi, and D. Tao, “Gmflow: Learning optical flow via global matching,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 8121–8130.
  * [23] X. Sui, S. Li, X. Geng, Y. Wu, X. Xu, Y. Liu, R. Goh, and H. Zhu, “Craft: Cross-attentional flow transformer for robust optical flow,” in _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_ , 2022, pp. 17 602–17 611.
  * [24] Z. Wang, Y. Wei, Y. Rao, J. Zhou, and J. Lu, “3d point-voxel correlation fields for scene flow estimation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [25] Y. Hou, X. Zhu, Y. Ma, C. C. Loy, and Y. Li, “Point-to-voxel knowledge distillation for lidar semantic segmentation,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8479–8488.
  * [26] J. Cen, P. Yun, S. Zhang, J. Cai, D. Luan, M. Tang, M. Liu, and M. Yu Wang, “Open-world semantic segmentation for lidar point clouds,” in _Computer Vision – ECCV 2022_. Cham: Springer Nature Switzerland, 2022, pp. 318–334.
  * [27] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical flow with convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2758–2766.
  * [28] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, “Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8934–8943.
  * [29] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” _arXiv preprint arXiv:1511.07122_ , 2015.
  * [30] J. Hur and S. Roth, “Iterative residual refinement for joint optical flow and occlusion estimation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 5754–5763.
  * [31] F. Zhang, O. J. Woodford, V. A. Prisacariu, and P. H. Torr, “Separable flow: Learning motion cost volumes for optical flow estimation,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 10 807–10 817.
  * [32] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine _et al._ , “Scalability in perception for autonomous driving: Waymo open dataset,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2446–2454.
  * [33] S. Lee, H. Lim, and H. Myung, “Patchwork++: Fast and robust ground segmentation solving partial under-segmentation using 3D point cloud,” in _Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst._ , 2022, pp. 13 276–13 283.
  * [34] X. Li, J. Zheng, F. Ferroni, J. K. Pontes, and S. Lucey, “Fast neural scene flow,” _arXiv preprint arXiv:2304.09121_ , 2023.
  * [35] A. 2, “Argoverse 2 scene flow online leaderboard,” https://eval.ai/web/challenges/challenge-page/2010/leaderboard/4759, 2023 Sep 14th.
  * [36] Q. Zhang, D. Duberg, R. Geng, M. Jia, L. Wang, and P. Jensfelt, “A dynamic points removal benchmark in point cloud maps,” _arXiv preprint arXiv:2307.07260_ , 2023.