# Virtual Breathalyzer

Ben Nassi , Lior Rokach, Yuval Elovici  Dept. of Software and Information
Systems Engineering
Ben-Gurion University of the Negev, Israel
<EMAIL_ADDRESS>
###### Abstract

Driving under the influence of alcohol is a widespread phenomenon in the US
where it is considered a major cause of fatal accidents. In this research we
present a novel approach and concept for detecting intoxication from motion
differences obtained by the sensors of wearable devices. We formalize the
problem of drunkenness detection as a supervised machine learning task, both
as a binary classification problem (drunk or sober) and a regression problem
(the breath alcohol content level). In order to test our approach, we
collected data from 30 different subjects (patrons at three bars) using Google
Glass and the LG G-watch, Microsoft Band, and Samsung Galaxy S4. We validated
our results against an admissible breathalyzer used by the police. A system
based on this concept, successfully detected intoxication and achieved the
following results: 0.95 AUC and 0.05 FPR, given a fixed TPR of 1.0.
Applications based on our system can be used to analyze the free gait of
drinkers when they walk from the car to the bar and vice-versa, in order to
alert people, or even a connected car and prevent people from driving under
the influence of alcohol.

##  I Introduction

Every 51 minutes a person died in a motor vehicle accident caused by an
alcohol impaired driver in 2013, a tragic statistic that represents more than
30% of all US traffic-related deaths that year [1]. The high rate of fatal
accidents resulting from "driving under the influence" (DUI) reflects the
devastating effects of alcohol consumption on driving (e.g reduced
coordination, difficulty steering, and reduced ability to maintain lane
position and brake appropriately).

DUI is a widespread phenomenon in the US. Yet in 2012, only 1% of the people
that reported episodes of alcohol impaired driving were arrested. Many counter
measures are deployed in order to reduce the influence of driving while
intoxicated. The most well-known strategy employed to catch intoxicated
drivers is the breath alcohol concentration (BrAC) test that measures the
weight of alcohol present within a certain volume of breath [2]. This test is
conducted with the breathalyzer device [3] which uses the driver’s breath as
specimen/sample. The breathalyzer uses infrared or electromechanical tests,
and sometimes even a combination of both, to determine the BrAC within a short
amount of time.

BrAC limits vary by country causing the definition of drunkenness differ
around the world. Table I contains the four most common BrAC thresholds (220,
240, 250, 350 micro-grams of alcohol per one liter of breath) used around the
world. Based on these standards, anyone with a breath alcohol concentration
measured by a breathalyzer above the defined threshold for a given country is
considered to be intoxicated. In the US, the threshold varies widely between
each state.

BRAC Threshold | Countries
---|---
220 | Scotland, Finland, Hong Kong
240 | Israel
250 | Greece, Spain, New Zealand
350 | Singapore, UK, Wales, Trinidad and Tobago
TABLE I: BrAC threshold (micro-grams of alcohol per one liter of breath)
permitted in each country

In this paper, we suggest a new approach for the detection of intoxication
based on wearable technology. It is a known fact that alcohol consumption
causes changes in people’s movements. We hypothesize that these changes can be
measured using wearable device’s sensors in order to detect drunkenness using
a trained machine learning model. The expected contributions of this research
stem from the following points:

  1. 1.

We show that existing widely used wearable devices can be used to identify the
physiological indicators that imply drunkenness (in terms of body movement)
based on free gait instead of traditional ad-hoc sensors that focused on the
breath.

  2. 2.

We formalize the task of the detection of intoxication as a supervised machine
learning task based on body movement measurements derived from wearable
devices. We used an actual breathalyzer (as used by police departments) in
order to label our data and train our models to evaluate our results.

  3. 3.

We analyze and compare the usefulness of each type of sensor and device for
detecting drunkenness (in terms of body movement) based on free gait.

A system based on our approach can prevent people from driving under the
influence with an alert provided after detecting intoxication unobtrusively as
a person leaves a bar (after spending some time drinking) and walks to their
cars. In the era of Internet of Things and connected cars, it might even be
used to prevent people from driving under the influence of alcohol by
triggering a connected car to prevent the ignition of the car when the car
owner is detected as drunk.

##  II Related Work

Although wearable devices have already ubiquitous, there has been no
scientific work conducted in the area of intoxication detection using wearable
devices by analyzing the movement and gait of subjects. However, such analysis
[4, 5, 6, 7, 8] has been used in other areas for various purposes and tasks.

###  II-A Gait Analysis

Gait analysis has been studied for many years, even before the era of
wearables devices. Up to a decade ago, researchers were using ad-hoc sensors
specially designed for research purposes. Mantyjarvi et al. [4] analyzed data
collected from warn accelerometer devices in order to identify subjects by
their gait. Gafurov et al. [5] used a worn accelerometer for authentication
and identification based on the subjects’ gait, while Lu et al. [6] showed
that authentication from gait is also possible from smartphone sensors.
Aminian et al. [7] analyzed accelerometer and gyroscope measurements from ad-
hoc sensors that they designed to be worn on a shoe in order to explore gait.
Xu et al. [9] presented a novel system for gait analysis using smartphones and
three sensors located within shoe insoles to provide remote analysis of the
user’s gait.

###  II-B Wearable Device Products

In 2013, Google introduced Google Glass , and in recent years other companies
have introduced smart watches, wristbands, and fitness trackers. The
prevalence of these devices means that practical research can be carried out
on existing products. In the area of movement analysis and wearable devices
and products, Thomaz et al. [10] used smartwatch motion sensors in order to
detect eating instances. Ranjan et al. [11] analyzed smartwatch sensors during
specific home-based activities (such as turning on a light switch) to identify
subjects based on hand gestures. In the field of emotion detection, Hernandez
et al. [12] analyzed head movement from Google Glass motion sensors in order
to detect stress, fear and calm. Hernandez et al. [13] analyzed smartwatch
motion sensors to estimate heart and breathing rates. Mazilu et al. [14]
analyzed wrist movement to detect gait freezing in Parkinson’s disease using
data sensors of smart watches and wristbands. Gabus et al. [15] and Casilari
et al. [16] used a smartwatch in order to detect falls. Inou et al. [17] used
three accelerometers located on the right wrist, breast, and back hip in order
to recognize nursing activities of nurses in a hospital.

###  II-C Intoxication Detection

Despite of the volumes of related work, there has been a limited amount of
research that addresses the domain of intoxication detection using the
ubiquitous wearable technology. In the area of movement analytics as a method
to detect drunkenness, there are a few works that relied on the fact that the
short-term effects of alcohol on subjects cause impairment to movement, gait,
and balance.

Kao et al. [18], analyzed the accelerometer data collected from the
smartphones of three subjects and compared the step times and gait stretch of
sober and drunk gaits. This research was limited in scope in that it only used
smartphone devices and three subjects. Also, it wasn’t aimed at detecting
whether a person was drunk given data collected from the device; instead the
study compared differences in drunk and sober gait. Arnold et al. [19]
investigated whether a smartphone user’s alcohol intoxication level (how many
drinks they had) can be inferred from their gait. They used time and frequency
domain features extracted from the device’s accelerometer to classify the
number of drinks a subject consumed based on the following ranges: 0-2 drinks
(sober), 3-6 drinks (tipsy), or 6+ drinks (drunk). Their results were not
validated against a real breathalyzer, and the data was only collected from a
mobile phone. Several works have utilized ubiquitous technology to detect
drunkenness based on driving patterns. Dai et al. [20] and Goswami et al. [21]
used mobile phone sensors and pattern recognition techniques to classify drunk
drivers based on driving patterns. Other studies tried to detect drunkenness
using different approaches. Thien et al. [22] and Wilson et al. [23] attempted
to simulate the HGN (Horizontal Gaze Nystagmus) test [24] in order to detect
drunkenness using a camera (i.e., smartphone camera) and computer vision
methods. Hossain et al. [25] used machine learning algorithms to identify
tweets sent under the influence of alcohol (based on text).

##  III Detailed Description of the Research

Our study utilizes machine learning to understand changes in patterns of
movement (movement analytics) that imply intoxication. We use wearable
devices’ sensors to measure the effects of drunkenness on parts of the body.
In this section we describe: (A) the factors to consider with regard to device
selection, (B) the system we developed for the experiment and (C) our approach
for detecting intoxication from wearable devices.

###  III-A Selecting the Wearable Devices

| Criteria/
---
Device
Galaxy S4 | LG-G Watch | Google Glass | Microsoft Band
Type | Smart Phone | Smart watch | Smart Glass | Fitness Tracker
Sensors |  | Accelerometer
---
Linear
Gyroscope
Gravity
Compass
| Accelerometer
---
Linear
Gyroscope
Gravity
Compass
| Accelerometer
---
Linear
Gyroscope
Gravity
Compass
| Accelerometer
---
Gyroscope
| Maximum
---
Sampling
Rate
180Hz | 200Hz | 100Hz | 62Hz
| Software
---
Developer
Kit
Android | Android Wear | Android Wear | Android
Connectivity |  | GSM
---
WiFi
Bluetooth
| WiFi
---
Bluetooth
| WiFi
---
Bluetooth
Bluetooth

TABLE II: Sensors available in each device. Figure 1: Overview of the system
and our approach. Data is collected before and after drinking from the sensors
of the devices and sent to the mobile phone using Bluetooth (S-Before and
S-After measurements). The signals are filtered with SMA (simple moving
average), and 2 sets of features are extracted (F-Before and F-After) from the
signals. A calculation of the difference of the features is applied to extract
a new set of features (F-Difference). The classification/regression model
processes the instance using a learned model and outputs the result.

In order to determine whether intoxication can be detected from changes in
motion using wearable devices, we first have to select the devices to use in
our experiment. There is a wide range of wearable devices available, and in
order to focus our research efforts on a few specific devices, we defined a
list of technical requirements for the devices to be used in our study.

  1. 1.

Applicability - We wanted our research to be as applicable as possible for
future use (i.e., application-oriented in order to develop a product based on
our results). For this reason, we made the decision to utilize the sensors of
existing products as opposed to specifically crafted sensors.

  2. 2.

Built-in motion sensors - Since our research is based on detecting the effects
of drunkenness on subjects’ movements, the wearable device must include motion
sensors.

  3. 3.

Programmability - The wearable device must provide a software developer kit
(SDK) which will allow us to develop the application that will sample the
motion sensors.

  4. 4.

Undependability and Connectivity - The device has the ability to be used and
evaluated separately and as part of connected network.

  5. 5.

Versatility - Since we didn’t know in advance which part of the body would be
the most effective and accurate indicator of drunkenness, we wanted to work
with devices that measure the movements of different parts of the body.

The following devices were selected in accordance with the aforementioned
specifications: Google Glass, the LG G-watch, the Microsoft Band, and the
Samsung Galaxy smartphone.

Smartphones are very common and provide us an excellent infrastructure for our
research. Regarding smartwatches and fitness trackers, according to a survey
conducted by Global-Web Index with 170K participants around the global in
2014, one out of six people already owns either smartwatch or fitness tracker
[26]; they predict that smartwatches will make up over 50% of the wearable
market by 2020 [27]. Although the market share of Google Glass is low (only
800K units were sold in 2014) [28], Google Glass was chosen because: (1)
Business Insider expects unit sales of Glass to climb sharply in the next few
years, to 21 million units in annual sales by the end of 2018[29], and (2)
there is a large increase in smart devices for the head, such as VR glasses,
AR helmets, and smart helmets for motorcycles. We chose Google Glass to
represent the set of smart devices targeted for the head, a type of device
that we believe will become common place in the next years.

Table II presents the motion sensors, SDK and connectivity of each of the
selected devices. The devices selected provide the ability to measure movement
in four parts of the body: the right hand, left hand, head, and rump.

###  III-B Developing the system for the experiment

In order to conduct an experiment to collect data, we used the SDKs to develop
a client application for each of the devices. The client runs as a background
service and waits for start/stop commands in order to start/stop the recording
of all of its available motion sensors (Table II). We paired each of the
wearable devices with the mobile device using Bluetooth communication and
added the functionality of sending start/stop commands to each of the clients
from the mobile device. The data was sampled with the maximum rate supported
by each device and was recorded as a time series in nanoseconds. When the stop
command was received from the mobile phone, each of the clients sent the data
it sampled to the mobile phone using Bluetooth.

Figure 2: From left to right: (a) The x, y, and z axes of the four devices.
(b) The experiment stages.

###  III-C Our Approach

We propose a new approach to detect intoxication (using the platform described
in the previous section) based on changes in movements extracted from two
measurements of simple gait at two times (associated with the two states of
the subject): the first movement sample/measurement is taken before the
subject has begun to drink alcohol, and the second sample is taken after the
subject has finished drinking (our experiment is described in greater details
in the next section).

A subject’s movement sample/measurement consists mainly of a time-series of
values obtained from each of the wearable devices used. Since each wearable
device contains 4-5 motion sensors, each sample consists of almost 20 streams
(time-series) of a sensor’s data.

Given person p and his/her weo samples: s-before (measurement taken before
alcohol consumption) and s-after (measurement taken after alcohol
consumption), we process them as follows:

  1. 1.

Feature Extraction - We extract two sets of features: the f-before set
(extracted from s-before) and the f-after set (extracted from s-after).

  2. 2.

Difference Calculation - We calculate a new set of features called the
f-difference. These features represent the difference (for each feature)
between the f-after and f-before. The difference signifies the effects of the
alcohol consumption on the subject’s movement.

  3. 3.

Labeling - We label the sample of each subject with the result of a
professional breathalyzer (one that is used by law enforcement) taken
immediately after s-after is measured.

  4. 4.

Evaluation - We apply supervised machine learning algorithms on a new set of
data, consisting of the union of the f-differences calculated for each of the
subjects and calculate the error.

By processing the samples following the four steps described above, we are
able to identify the physiological indicators that imply drunkenness (in terms
of body movement) based on free gait. This approach can be used in order to
prevent people from driving under the influence. By obtaining two samples of
an individual’s gait (the first sample obtained on the subject’s way from
their car to a bar, and the second taken as they return to their car from the
bar). Figure 1 provides an overview of our approach.

##  IV Experimental settings

Figure 3: Aerial view of the street where the experiment took a place. The 2
red dots mark 2 out of the 3 bars and white line shows the path that our
subjects walked during the experiment. Figure 4: From left to right. (a)
Breathalyzer samples - The bars represent the results of the subjects’
breathalyzer tests (the amount of micro-grams of alcohol per one liter of
breath). (b) A breakdown of the subjects’ state (sober/drunk) at various BAC
levels.

Our subjects were outfitted with the devices as follows. They each wore Google
Glass on their head, an LG G watch on their left hand, and a Microsoft Band on
their right hand (see Figure 2). The subjects who had a right rear pocket also
carried a mobile device there. Those subjects without a right rear pocket were
only outfitted with three devices (the mobile was not part of their
experiment). Each subject also wore headphones.

###  IV-A The Experiment

The process of obtaining gait samples included the following steps. First,
each subject was instructed to walk with the devices and headphones until
he/she heard a beep in their headphones (for a total of 16 seconds). During
the subject’s gait, the devices’ clients sampled the sensors for eight
seconds, a time period that started on the sixth second of the experiment and
continued until the fourteenth second. The stages of the experiment are
presented in Figure 2.

We decided that using 16 seconds of free gait is the optimal way to conduct
the experiment and obtain the samples for the following reasons: (1) The gait
is probably the best way to ensure that the devices are carried with the
person instead of standing on a table during a conversation. (2) Because we
can automatically turn the model on when the GPS in a device sense a movement,
instead of relying on the subject to remember to turn on the application. (3)
The model will be relevant, even for people whose cars are parked very close
to the bar. (4) Using free gait instead of a specific challenge for our model
will ensure that potential subjects won’t be put off by the nature of the
challenge and allow the model to be used in real life situations.

The experiment was conducted in two sessions. The first session took place
before the subjects had their first drink. The second session took place 15
minutes following the subjects’ last drink, just before they intended to leave
the bar. It was crucial to wait 15 minutes after the last drink, because in
addition to the gait samples, our subjects provided us with breath samples
using our Drager Alcotest 5510 breathalyzer, and 15 minutes is the same amount
of time that police wait before obtaining a suspect’s breath sample. We used
this type of breathalyzer, because it is a professional breathalyzer used by
police departments in different countries around the world. This breathalyzer
outputs results in micro-grams of alcohol per one liter of breath. We labeled
each sample with the breathalyzer result (BrAC).

In order to sample as many people as possible, our experiment took place at
three different bars that offer an "all you can drink" option. We waited for
people to arrive at the bar, and just before they ordered their first drink,
we asked them to participate in our research (participation entailed providing
a gait sample during two brief experimental sessions while wearing wearable
devices, as well as providing a breath sample).

We asked 30 different subjects to participate in our research. Each person was
instructed to walk (while wearing the devices) in any direction they wished
until they heard a beep in the headphones (this was done in order to simulate
the path between the bar and the subject’s car). The subjects were outfitted
with three to four devices(as mentioned previously, depending on whether they
had a right rear pocket). Participants were paid for their participation in
the study (each subject received the equivalent of 13 USD in local currency).
Figure 3 presents an aerial view of two of the three bars and the path that
our subjects took during the experiment.

###  IV-B Data Analysis

Our experiment is based on 60 samples taken from 30 individuals on five nights
at three bars. Table III provides information about subjects. Most of our
participants were in their early 20’s, which according to US National Highway
Traffic Safety Administration (NHTSA) [30] is the group considered to have the
highest risk of causing fatal accidents due to alcohol consumption (in 30% of
the resulting from intoxicated drivers in 2014, the drivers were between the
ages of 21 and 24).

Figure 4 presents the analysis and distribution of the breathalyzer results.
Most of our samples (89%) are labeled with breathalyzer result (the amount of
micrograms of alcohol per one litter of breath) in the 0-400 range. Our data
needed to include samples of both sober and drunk states. This was crucial to
the model creation phase (described later) in order to learn the movement
differences that imply intoxication, as well as the differences that don’t
imply intoxication. The breakdown of the subjects sober/drunk states is also
presented in Figure 4 according to the thresholds presented on Table I. At the
lower thresholds of alcohol concentration (220,240,250) the data is
distributed, such that 27%-35% of the total number of subjects were considered
to be intoxicated. At the highest threshold (350) 16% of the subjects were
considered drunk.

Gender |  | Number of
---
Participants
Age(Year) | Height(CM) | Mass(KG)
Male | 24 (80%) | $24.1\pm 3.6$ | $176.4\pm 9.2$ | $73.1\pm 10.5$
Female | 6 (20%) | $24.5\pm 5.9$ | $168.5\pm 4.5$ | $60\pm 4.5$

TABLE III: Details of the participants in the experiment. Each cell presents
the average and the standard deviation

###  IV-C Fusion of the Sensors

Each subject contributed two breath and gait samples (obtained in two sessions
- before and after drinking). Each gait sample is comprised of sensor readings
(measurements) obtained from three or four different wearable devices
(depending on whether the subject had a right rear pocket for a mobile phone).
Each wearable device contained three to five different motion sensors (as
described in Table 2) that sampled during the experiment. This amounts to
measurements from 15-20 sensors obtained in each session from the various
devices.

Each sensor sample produces a vector with four dimensions: time in nanoseconds
(from the beginning of each experiment) and the values x, y, z that represent
the values for each axis as a real number. The entire set of samples obtained
during each session from a specific sensor completes the signal that is
represented as a time series of values in each of the axes (x,y,z). Figure 2
presents the axes of each device used in the experiment.

##  V Feature Extraction

Given person p with two measurements (s-before and s-after) consisting of the
sensor’s data, with four dimensions (time,x value,y value, and z value) for
each of of the wearable-devices, we consider the time-series of each axis of
each measurement as an independent signal (e.g, the x axis of the smart-watch
accelerometer of measurement s-before). Given two signals s-before and s-after
(measured from the same person and the same axis) from the same wearable-
device, we extract the correlative vectors f-before and f-after, the features
of each signal, from four different types (the entire set of features is
described on Table IV):

  1. 1.

Fast Fourier Transform: FFT was used on each axis of the signals in order to
transform the signal to the frequency domain and extracted the features
described in previous works [4, 13]. Such features may indicate physiological
changes resulting from alcohol consumption that are associated with frequency
of movement (e.g., fewer steps per second as obtained from the mobile phone).

  2. 2.

Statistic features: Again, we extracted statistic features mentioned in
previous works [6, 10, 31, 32], because they may indicate physiological
changes associated with intoxication, such as decreased average acceleration
of hand movements measured by motion sensors on the smart watch or sharp head
movement obtained by the maximum values of Google Glass’ gyroscope.

  3. 3.

Histogram features: We presented the signals as histograms as presented in
previous studies [33, 34]. We extracted histograms from each signal that
represent the values (in integers) between the maximum and minimum value of
each sensor on each axis. Since our data was represented as real numbers, we
round the values of the signals, count them into the histogram, and normalize
them by dividing the length of each signal. We extracted this set of features,
because we thought that they would be helpful to us in detecting differences
in the patterns of movement (and specifically, the distribution of the
movement) resulting from alcohol consumption.

  4. 4.

Known gait features: This type of features has been shown to yield good
results in previous studies [31, 32]

We repeat the feature extraction process for each signal and concatenate the
entire set of features extracted for the measurement of 1 large vector. The
f-before-complete and f-after-complete vectors are vectors that have the same
length extracted from all of the signals of two different measurements
(s-before and s-after). Each index in the vectors is associated with a feature
from the features’ list for a specific signal. The index i in vectors
f-before-complete and f-after-complete represents the same feature being
extracted from the s-before and s-after (e.g., the value of index one in
vectors f-before-complete and f-after-complete is the mean acceleration
measured from the accelerometer of Google Glass in axis x from s-before and
s-after). We consider the entire set of vectors of features extracted from all
of the signals of specific measurement (e.g., all the features’ vectors being
extracted from s-before) as one long vector

We calculate a new set of features called f-difference as follows: for each
one of the features we extract (described in Table IV), we calculate the
difference between its value in f-before-complete and its value in f-after-
complete.

We label these differences with the breathalyzer results (to allow us to use
supervised machine learning algorithms). We consider each f-difference vector
labeled by a breathalyzer result as an instance. We consider the union of
these 30 instances as our data-set, representing the indicators associated
with subjects’ level of intoxication (measured by the breathalyzer).

| Statistics
---
Features
| Mean, Variance, Covariance, Standard Deviation,
---
Skewness, Min, Max, Median, Range, Root mean square,
Zero crossing rate, Mean crossing rate
| FFT
---
Features
| Energy, Top 4 frequency values,
---
Frequency bin that contains the frequency
with maximum magnitude value
| Histogram
---
Features
| Normalized histogram of the
---
values (each value is rounded to the nearest integer)
| Known
---
Gait
Features
| Movement intensity, Normalized signal magnitude area,
---
Eigenvalues of dominant directions, Eigenvalues of
dominant directions, Correlation between acceleration
along gravity and heading directions, Average velocity
along heading direction, Average velocity along gravity
direction, Average rotation angles related to
gravity direction, Dominant frequency, Energy,
Average acceleration energy, Average rotation energy

TABLE IV:  Features types extracted.

##  VI Results

The feature extraction process resulted in 30 labeled instances extracted from
30 users, representing the differences between the extracted features before
and after drinking. We used this data to train supervised machine learning
models using two approaches.

The first approach is to analyze the data as a regression task in which we try
to predict the breathalyzer result given the differences in the subject’s gait
features. We also aim to minimize the RMSE (root mean square error) and MAE
(mean absolute error). We consider a model to provide better results compared
others if its MAE and RMSE are lower.

The second approach is to analyze the data as a classification task, with a
goal of determining whether a person is drunk or sober according to known BrAC
thresholds as measured using a breathalyzer. More precisely, we aim to train a
model that predicts whether a person is intoxicated or not using differences
in the subject’s gait features. We chose to classify our instances as one of
four thresholds 220, 240, 250, 350 (presented in Table 1). We consider an
instance labeled by a breathalyzer result (BrAC) to be sober if its value is
less than the threshold, and drunk if its value exceeds the threshold. In this
approach we try to maximize the AUC (area under the curve) and the model’s
accuracy. In this task, we consider a model to provide better results than
others if it has higher AUC results.

In order to classify/predict an outcome of an instance given the values of the
features, we used ensemble machine learning algorithms based on a decision
tree. A decision tree recursively partitions the independent variables’ space
into subspaces such that each subspace constitutes the basis for a different
prediction. A single decision tree usually has limited prediction performance.
One way to improve the prediction performance is to build a forest which
combines the predictions of several trees into a final prediction. In this
paper, we use decision trees (torest), AdaBoost and GBM (gradient boosted
machine) to build the forest in a stages.

More specifically, we use a CART (classification and regression tree)
algorithm that builds a set of decision trees by selecting features, in order
to divide the training data, by optimizing the entropy in each level. GBM
trains a sequence of trees, where each successive tree aims to predict the
pseudo-residuals of the preceding trees assuming that the loss function is MSE
(mean squared error). This method allows us to combine a large number of
regression trees with a small learning rate. AdaBoost trains a set of weak
learners (decision trees) and combines them into a weighted sum that
represents the final outcome.

Since our data is based on samples from 30 subjects, we can utilize the leave-
one-out protocol, i.e., the learning process is repeated 30 times, and in each
test, 29 subjects are used as a training set and one subject is used as a test
set for evaluating the predictive performance of the method.

###  VI-A Detection of Intoxication Using Classification

The first task in our research is to detect whether a person is intoxicated or
not using supervised machine learning algorithms and in order to do this we
aim to learn the indicators (of intoxication) from the differences to predict
drunkenness. We start by using all four devices as a connected network. We use
movements from the head, hands, and rump obtained by Google Glass, the LG G
watch, Microsoft Band, and Samsung Galaxy S4. The first approach we used was
to handle the task of drunkenness detection as a classification task. The
output of this task is binary (drunk, sober) and varies depending on the
country (based on the BrAC threshold of each country).

We trained three different models: GBC (gradient boosting classifier),
AdaBoost, and decision trees (DT). We evaluated our models using the leave-
one-out protocol against each of the thresholds 220, 240, 250, 350. Table V
shows the AUC score of each model;

BRAC Threshold | AdaBoost | Gradient Boosting Classifier | Decision Trees
---|---|---|---
220 | 0.91 | 0.86 | 0.53
240 | 0.95 | 0.95 | 0.61
250 | 0.56 | 0.56 | 0.56
350 | 0.87 | 0.84 | 0.85

TABLE V:  AUC of classification algorithms (AdaBoost, Decision Trees, and
Gradient Boosting Classifier) using all four of the devices in each threshold.

The GBC and AdaBoost classifiers yielded excellent results with AUCs of 0.95
at a threshold of 240. AdaBoost also yielded a strong result with an AUC of
0.91 at a threshold of 220. In order to determine the accuracy of the models,
we fixed the threshold at 240.

Figure 5 presents the ROC (receiver operating characteristic) curve of the
models at the fixed threshold of 240 (using four devices). The positive class
is drunk, and the negative class is sober.

Figure 5: ROC curve of classification algorithms (AdaBoost, Decision Trees,
and Gradient Boosting Classifier) using all four of the devices in threshold
240.

Despite the high AUC that the GBC yields, there is one drunk instance that was
misclassified as a sober instance (false negative) as can been seen in the
confusion matrix in Table VI. Since this type of mistake can be dangerous (by
accidentally allowing a drunk driver to drive), we wanted to verify our
results with the limitation that each drunk instance in our data-set will be
predicted as drunk. In other words, we fixed the TPR (true positive rate) at
1.0 (the true class is drunk) and checked the damage this limitation caused to
the false positive rate (FPR).

| Drunk | Sober
---|---|---
Predicted as drunk | 8 | 1
Predicted as sober | 1 | 20
TABLE VI: Confusion matrix of the gradient boosting classifier with BrAC
threshold of 240.

Table VII presents the FPR results of the GBC, AdaBoost and Decision Trees
classifiers with a fixed TPR of 1.0 at a BrAC threshold of 240. Since, the GBC
yielded the highest results, we fixed it and evaluated this model using the
leave-one-out approach against four thresholds: 220, 240, 250, 350 (see Table
I). Table VIII shows the FPR of the four thresholds with a fixed TPR of 1.0
using the same data. Since lower values of FPR represent better performance,
it can be seen that the GBC yields excellent results, with 100% detection of
intoxication and a minimum number of mistakes (one sober instance that was
mistakenly classified as drunk - false positive). This demonstrates that this
kind of model can be deployed in order to prevent drunk people from driving
with a high rate of intoxication detection (100% accuracy) and a minimal
amount of misclassification (mistakenly classifying sober individuals as
drunk).

| GBC | AdaBoost | Decision Trees
---|---|---|---
| FPR (with
---
TPR = 1.0)
0.05 | 0.1 | 1.0
TABLE VII: FPR (false positive rate) of the Gradient Boosting Classifier (GBC),AdaBoost and Decition Trees with fixed TPR (true positive rate) of 1.0 in threshold of 240.  | 220 | 240 | 250 | 350
---|---|---|---|---
| FPR (with
---
TPR = 1.0)
0.25 | 0.05 | 0.5 | 1.0
TABLE VIII: FPR (false positive rate) of the Gradient Boosting Classifier with
fixed TPR (true positive rate) of 1.0 in each on each drunkenness threshold.

Since most people don’t carry smart glass, smart watches, smart fitness
trackers, and smart phones all at the same time, we wanted to look at the
results based on subsets of the devices.

Therefore, we conducted the experiment again, with each device separately
(smartphone, glass, smartwatch) and in the following combinations:
smartphone/smartwatch and smartphone/Google Glass. Figure 6 presents the AUC
of each combination at a threshold of 240 using the GBC.

Figure 6: Comparison of AUC for different combinations of devices with GBC and
240 as threshold task.

We can see from the results, that the devices when used on their own, are not
strong enough to effectively predict drunkenness. However, the results of the
smartphone/smartwatch combination (0.94 AUC) were close to those obtained by
using all four devices together (0.95 AUC). These results have great
implications given today’s high use of smartphones (80% usage across the
world). Furthermore, a survey conducted in 2014 [26], indicated that one out
of six people around the world owns fitness tracker or smartwatch (with
optimistic forecasts for increased use in the future), which means that a
model that combines a smartwatch and smartphone could be effectively deployed
in the near future.

###  VI-B Detection of Intoxication Using Regression

The second approach utilized supervised machine learning algorithms for
regression to predict the BrAC. The output of this type of task is numeric and
can be framed as a virtual breathalyzer based on the gait differences derived
from the four devices.

We used the same instances from the first approach (the classification task
with four devices) and trained four different models, three of which are
regression models of the same algorithms used in the classification task: GBR
(gradient boosting regression), ABR (AdaBoost regression), and Regression
Trees (RT), as well as Lasso. We evaluated our models using the leave-one-out
approach.

Table IX presents the MAE and RMSE of each model. This time, the RT resulted
in the best performance, as it yielded a MAE of 83 micro-grams of alcohol per
one liter of breath (the lowest of all of the models). This result represents
a mistake with approximately 19% of the interval of the data, since the
highest label was 430 and the lowest label was 0.

Classifier | MAE | RMSE
---|---|---
Gradient Boosting Regression | 97.2 | 123.3
Regression Trees | 83.1 | 124.7
Lasso | 120.9 | 167.9
AdaBoost Regression | 103 | 144
TABLE IX: MAE (mean absolute error) and RMSE (root mean squared error) of
different regression algorithms.

As was done previously, we wanted to check our ability to predict drunkenness
from a subset of devices. We used the same five combinations of devices
previously utilized in the classification approach: (1) smartphone,(2)
smartwatch, (3) Google Glass, (4) smartphone/smartwatch combination, and (5)
smartphone/Google glass.

Again, we evaluated our models using the leave-one-out approach. Figure 7
presents the MAE (mean absolute error) of each model for each device. The
combination of the watch and the smartphone also yielded the best performance,
as its MAE is minimal across all models.

Figure 7: MAE (mean absolute error) of different regression algorithms (GBR,
RT, ADR) independently from each device.

###  VI-C Feature Robustness

The third task in our research is to detect which set of features yields the
highest performance in the task of detection of drunkenness. In the feature
extraction process we extracted four types of features as described in detail
in Table IV. Since the gait individuals changes as a result of alcohol
consumption, we wanted to identify the best set of indicators to detect
drunkenness (based on body movement patterns) and determine whether using
1)the distribution of the movement (histogram), 2) frequency of the movement
(FFT), 3) statistics or 4) known gait features, is most effective at this
task. in order to do so, we used the 30 instances with all four devices as we
did in the first approach. We evaluated this approach utilizing the classifier
that yielded the highest AUC (the GBC) and a threshold of 240. We classified
each instance using two methods. The first classification method done using a
specific set of features among the sets (histogram, known gait features,
frequency features, statistics). The second classification method done using
all the other sets of features (except the set used in the first method).
Table X presents the AUC of each of the sets for the two classification
methods. The histogram is the most important set of features to include, since
this set of features yields the highest AUC when used on its own, and the
worst AUC is obtained when classification is handled without the histogram
set. the Histogram set of features yields the highest AUC when classifying
only with it, and the worst AUC when classifying without it, It is the most
important set among the other sets. We believe that this is the result of the
effects of alcohol consumption on the distribution of movement across the
parts of the body. Since the frequency set of features yielded an AUC score of
0.53, and the highest AUC score obtained when we classified without this set
of features, we can only infer that either the frequency of the movement of
the body is poor indicator for drunkenness detection, or that we should
attempt to detect the frequency of the body as an indicator for drunkenness
from another device.

Set Of Features | Only this set |  All other sets but this set
---|---|---
Histogram Features | 0.80 | 0.42
Known Gait Features | 0.76 | 0.94
Frequency Features | 0.53 | 0.97
Statistics Features | 0.67 | 0.54
TABLE X: The AUC scores of different set of features

##  VII Discussion

Excellent results can be achieved when combining various wearable devices for
the task of intoxication detection. Even when combining only smartphone a
smartwatch, excellent results are obtained. Since the use of each device on
its own does not result in good prediction, we can infer that (1) the
differences in the movements of a single body part cannot predict
intoxication, and (2) the prediction of intoxication based on movement
differences can be achieved with measurements taken from at least two parts of
the body

We are optimistic that within a few years, with the increased adoption of
wearable devices and the ongoing IoT revolution, our system can be implemented
on subjects that routinely carry a smartwatch along with their smartphone.

Since our system yielded good results predicting intoxication at the
thresholds of 220 and 240, it can currently be deployed in countries that
utilize this threshold range.

##  VIII CONCLUSIONS

In this paper, we describe a novel approach to detect intoxication from motion
differences using wearable devices (smart watch, smart glass, fitness tracker,
smartphone). We conduct an experiment using 30 people across three different
bars in order to evaluate our approach.

Supervised machine learning models were trained and result in an AUC of 0.95
for a BrAC threshold of 240 micrograms of alcohol per one liter of breath
using four devices and an AUC of 0.94 using only a smart phone and smart
watch.

A system based on this approach can be used to alert people from driving under
the influence of alcohol using two simple gait samples (from a car to a bar
and vice-versa) and may also be used to trigger the owner’s connected car to
prevent ignition in cases in which the owner os detected as drunk.

We believe that our results will establish a baseline and also provide an
indication of some of the challenges associated with intoxication detection
based on free gait.

##  IX Future Work

There are numerous opportunities to extend this work. One opportunity is to
add new devices that are not yet on the market to the system in order to
sample more parts of the body such as smart rings and smart shirts.

Another opportunity to extend this research is to compare the results of
personal models/classifiers versus the collaborative model used in this
research. We believe that personal models may result in greater accuracy. In
this case, the process can be simplified by generating different models for
people with the same height, weight, age, and gender.

##  X ACKNOWLEDGMENTS

We would like to thank all of the participants in this study. We also thank A,
B, C, and D for their help collecting samples from the subjects.

## References

  * [1] CDC, “Injury prevention and control: Motor vehicle safety,” 2016, http://www.cdc.gov/motorvehiclesafety/impaired_driving/impaired-drv_factsheet.html.
  * [2] WebMD, “Self-test for breath alcohol,” 2016, http://www.webmd.com/mental-health/addiction/self-test-for-breath-alcohol.
  * [3] Wikipedia, “Breathalyzer,” 2016, https://en.wikipedia.org/wiki/Breathalyzer.
  * [4] J. Mäntyjärvi, M. Lindholm, E. Vildjiounaite, S.-M. Mäkelä, and H. Ailisto, “Identifying users of portable devices from gait pattern with accelerometers,” in _Acoustics, Speech, and Signal Processing, 2005. Proceedings.(ICASSP’05). IEEE International Conference on_ , vol. 2. IEEE, 2005, pp. ii–973.
  * [5] D. Gafurov, E. Snekkenes, and P. Bours, “Gait authentication and identification using wearable accelerometer sensor,” in _Automatic Identification Advanced Technologies, 2007 IEEE Workshop on_. IEEE, 2007, pp. 220–225.
  * [6] H. Lu, J. Huang, T. Saha, and L. Nachman, “Unobtrusive gait verification for mobile phones,” in _Proceedings of the 2014 ACM International Symposium on Wearable Computers_. ACM, 2014, pp. 91–98.
  * [7] K. Aminian, F. Dadashi, B. Mariani, C. Lenoble-Hoskovec, B. Santos-Eggimann, and C. J. Büla, “Gait analysis using shoe-worn inertial sensors: how is foot clearance related to walking speed?” in _Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing_. ACM, 2014, pp. 481–485.
  * [8] S. Murata, M. Suzuki, and K. Fujinami, “A wearable projector-based gait assistance system and its application for elderly people,” in _Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing_. ACM, 2013, pp. 143–152.
  * [9] W. Xu, M.-C. Huang, N. Amini, J. J. Liu, L. He, and M. Sarrafzadeh, “Smart insole: a wearable system for gait analysis,” in _Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments_. ACM, 2012, p. 18.
  * [10] E. Thomaz, I. Essa, and G. D. Abowd, “A practical approach for recognizing eating moments with wrist-mounted inertial sensing,” in _Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing_. ACM, 2015, pp. 1029–1040.
  * [11] J. Ranjan and K. Whitehouse, “Object hallmarks: Identifying object users using wearable wrist sensors,” in _Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing_. ACM, 2015, pp. 51–61.
  * [12] J. Hernandez and R. W. Picard, “Senseglass: Using google glass to sense daily emotions,” in _Proceedings of the adjunct publication of the 27th annual ACM symposium on User interface software and technology_. ACM, 2014, pp. 77–78.
  * [13] J. Hernandez, D. McDuff, and R. W. Picard, “Biowatch: estimation of heart and breathing rates from wrist motions,” in _Proceedings of the 9th International Conference on Pervasive Computing Technologies for Healthcare_. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2015, pp. 169–176.
  * [14] S. Mazilu, U. Blanke, and G. Troster, “Gait, wrist, and sensors: Detecting freezing of gait in parkinson’s disease from wrist movement,” in _Pervasive Computing and Communication Workshops (PerCom Workshops), 2015 IEEE International Conference on_. IEEE, 2015, pp. 579–584.
  * [15] P. Kostopoulos, T. Nunes, K. Salvi, M. Deriaz, and J. Torrent, “Increased fall detection accuracy in an accelerometer-based algorithm considering residual movement,” in _International Conference on Pattern Recognition Applications and Methods_ , 2015.
  * [16] E. Casilari and M. A. Oviedo-Jiménez, “Automatic fall detection system based on the combined use of a smartphone and a smartwatch,” _PLoS one_ , vol. 10, no. 11, p. e0140929, 2015.
  * [17] S. Inoue, N. Ueda, Y. Nohara, and N. Nakashima, “Mobile activity recognition for a whole day: recognizing real nursing activities with big dataset,” in _Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing_. ACM, 2015, pp. 1269–1280.
  * [18] H.-L. C. Kao, B.-J. Ho, A. C. Lin, and H.-H. Chu, “Phone-based gait analysis to detect alcohol usage,” in _Proceedings of the 2012 ACM Conference on Ubiquitous Computing_. ACM, 2012, pp. 661–662.
  * [19] Z. Arnold, D. Larose, and E. Agu, “Smartphone inference of alcohol consumption levels from gait,” in _Healthcare Informatics (ICHI), 2015 International Conference on_. IEEE, 2015, pp. 417–426.
  * [20] J. Dai, J. Teng, X. Bai, Z. Shen, and D. Xuan, “Mobile phone based drunk driving detection,” in _Pervasive Computing Technologies for Healthcare (PervasiveHealth), 2010 4th International Conference on-NO PERMISSIONS_. IEEE, 2010, pp. 1–8.
  * [21] T. D. Goswami, S. R. Zanwar, and Z. U. Hasan, “Android based rush and drunk driver alerting system,” _International Journal of Engineering Research and applications, Page (s)_ , pp. 1–4, 2014.
  * [22] N. H. Thien and T. Muntsinger, “Horizontal gaze nystagmus detection in automotive vehicles.”
  * [23] J. Wilson and A. Avakov, “Image processing methods for mobile horizontal gaze nystagmus sobriety check.”
  * [24] “field-sobriety-test-review,” 2016.
  * [25] N. Hossain, T. Hu, R. Feizi, A. M. White, J. Luo, and H. Kautz, “Inferring fine-grained details on user activities and home location from social media: Detecting drinking-while-tweeting patterns in communities,” _arXiv preprint arXiv:1603.03181_ , 2016.
  * [26] globalwebindex, “Gwi-device-summary,” 2014, http://cdn2.hubspot.net/hub/304927/file-2301369304-pdf/Reports/GWI_Device_Summary_Q3_2014.pdf?submissionGuid=fb7e899d-fda2-4064-b520-cc77d08df121.
  * [27] businessinsider, “smartwatch-report,” 2015, https://intelligence.businessinsider.com/the-smartwatch-report-the-market-for-luxury-wristwatches-retail-distribution-and-apple-watchs-big-opportunity-2015-2.
  * [28] ——, “Bi-googleglass-marketshare,” 2014, https://intelligence.businessinsider.com/google-glass-is-forecast-to-rise-to-mainstream-2013-5?utm_source=House&utm_medium=Edit&utm_term=GG&utm_content=link&utm_campaign=BIIMobile.
  * [29] ——, “Bi-googleglass-forecasts,” 2014, http://www.businessinsider.com/google-glass-11-billion-market-by-2018-2013-5.
  * [30] nhtsa, “Alcohol-impaired driving,” 2015, https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812231.
  * [31] M. Zhang and A. A. Sawchuk, “A feature selection-based framework for human activity recognition using wearable multimodal sensors,” in _Proceedings of the 6th International Conference on Body Area Networks_. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2011, pp. 92–98.
  * [32] ——, “Motion primitive-based human activity recognition using a bag-of-features approach,” in _Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium_. ACM, 2012, pp. 631–640.
  * [33] C. Basaran, H. J. Yoon, H. K. Ra, S. H. Son, T. Park, and J. Ko, “Classifying children with 3d depth cameras for enabling children’s safety applications,” in _Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing_. ACM, 2014, pp. 343–347.
  * [34] I. Hazan and A. Shabtai, “Noise reduction of mobile sensors data in the prediction of demographic attributes,” in _Proceedings of the Second ACM International Conference on Mobile Software Engineering and Systems_ , ser. MOBILESoft ’15. Piscataway, NJ, USA: IEEE Press, 2015, pp. 117–120. [Online]. Available: http://dl.acm.org/citation.cfm?id=2825041.2825062


* [161] Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal.  Improved few-shot visual classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14493–14502, 2020.
  * [162] Dong Hoon Lee and Sae-Young Chung.  Unsupervised embedding adaptation via early-stage feature reconstruction for few-shot classification.  arXiv preprint arXiv:2106.11486, 2021.
  * [163] Haoxiang Wang, Han Zhao, and Bo Li.  Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation.  arXiv preprint arXiv:2106.09017, 2021.
  * [164] Mohamed Afham, Salman Khan, Muhammad Haris Khan, Muzammal Naseer, and Fahad Shahbaz Khan.  Rich semantics improve few-shot learning.  arXiv preprint arXiv:2104.12709, 2021.
  * [165] Mamshad Nayeem Rizve, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah.  Exploring complementary strengths of invariant and equivariant representations for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10836–10846, 2021.
  * [166] Reza Esfandiarpoor, Amy Pu, Mohsen Hajabdollahi, and Stephen H Bach.  Extended few-shot learning: Exploiting existing resources for novel tasks.  arXiv preprint arXiv:2012.07176, 2020.
  * [167] Haoxing Chen, Huaxiong Li, Yaohui Li, and Chunlin Chen.  Multi-scale adaptive task attention network for few-shot learning.  arXiv preprint arXiv:2011.14479, 2020.
  * [168] Bowen Wang, Liangzhi Li, Manisha Verma, Yuta Nakashima, Ryo Kawasaki, and Hajime Nagahara.  Match them up: Visually explainable few-shot image classification.  arXiv preprint arXiv:2011.12527, 2020.
  * [169] Lyes Khacef, Vincent Gripon, and Benoît Miramond.  Gpu-based self-organizing maps for post-labeled few-shot unsupervised learning.  In International Conference on Neural Information Processing, pages 404–416. Springer, 2020.
  * [170] Zhiyu Xue, Lixin Duan, Wen Li, Lin Chen, and Jiebo Luo.  Region comparison network for interpretable few-shot image classification.  arXiv preprint arXiv:2009.03558, 2020.
  * [171] Xian Zhong, Cheng Gu, Wenxin Huang, Lin Li, Shuqin Chen, and Chia-Wen Lin.  Complementing representation deficiency in few-shot image classification: A meta-learning approach.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 2677–2684. IEEE, 2021.
  * [172] Imtiaz Ziko, Jose Dolz, Eric Granger, and Ismail Ben Ayed.  Laplacian regularized few-shot learning.  In International Conference on Machine Learning, pages 11660–11670. PMLR, 2020.
  * [173] Peyman Bateni, Jarred Barber, Jan-Willem van de Meent, and Frank Wood.  Enhancing few-shot image classification with unlabelled examples.  arXiv preprint arXiv:2006.12245, 2020.
  * [174] Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah.  Self-supervised knowledge distillation for few-shot learning.  arXiv preprint arXiv:2006.09785, 2020.
  * [175] Yuqing Hu, Vincent Gripon, and Stéphane Pateux.  Leveraging the feature distribution in transfer-based few-shot learning.  arXiv preprint arXiv:2006.03806, 2020.
  * [176] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi.  Adaptive subspaces for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4136–4145, 2020.
  * [177] Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang.  Boosting few-shot learning with adaptive margin loss.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12576–12584, 2020.
  * [178] Shell Xu Hu, Pablo G Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, Neil D Lawrence, and Andreas Damianou.  Empirical bayes transductive meta-learning with synthetic gradients.  arXiv preprint arXiv:2004.12696, 2020.
  * [179] Yikai Wang, Chengming Xu, Chen Liu, Li Zhang, and Yanwei Fu.  Instance credibility inference for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12836–12845, 2020.
  * [180] Cuong Nguyen, Thanh-Toan Do, and Gustavo Carneiro.  Pac-bayesian meta-learning with implicit prior and posterior.  arXiv preprint arXiv:2003.02455, 2020.
  * [181] Jiechao Guan, Zhiwu Lu, Tao Xiang, and Ji-Rong Wen.  Few-shot learning as domain adaptation: Algorithm and analysis.  arXiv preprint arXiv:2002.02050, 2020.
  * [182] Da Chen, Yuefeng Chen, Yuhong Li, Feng Mao, Yuan He, and Hui Xue.  Self-supervised learning for few-shot image classification.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1745–1749. IEEE, 2021.
  * [183] Jin Xu, Jean-Francois Ton, Hyunjik Kim, Adam Kosiorek, and Yee Whye Teh.  Metafun: Meta-learning with iterative functional updates.  In International Conference on Machine Learning, pages 10617–10627. PMLR, 2020.
  * [184] Jialin Liu, Fei Chao, and Chih-Min Lin.  Task augmentation by rotating for meta-learning.  arXiv preprint arXiv:2003.00804, 2020.
  * [185] Liang Song, Jinlu Liu, and Yongqiang Qin.  Generalized adaptation for few-shot learning.  arXiv preprint arXiv:1911.10807, 2019.
  * [186] Massimiliano Patacchiola, Jack Turner, Elliot J Crowley, Michael O’Boyle, and Amos Storkey.  Bayesian meta-learning for the few-shot setting via deep kernels.  2020\.
  * [187] Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Balasubramanian.  Charting the right manifold: Manifold mixup for few-shot learning.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2218–2227, 2020.
  * [188] Xinzhe Li, Qianru Sun, Yaoyao Liu, Qin Zhou, Shibao Zheng, Tat-Seng Chua, and Bernt Schiele.  Learning to self-train for semi-supervised few-shot classification.  Advances in Neural Information Processing Systems, 32:10276–10286, 2019.
  * [189] Sung Whan Yoon, Jun Seo, and Jaekyun Moon.  Tapnet: Neural network augmented with task-adaptive projection for few-shot learning.  In International Conference on Machine Learning, pages 7115–7123. PMLR, 2019.
  * [190] Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang D Yoo.  Edge-labeling graph neural network for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11–20, 2019.
  * [191] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha.  Few-shot learning via embedding adaptation with set-to-set functions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8808–8817, 2020.
  * [192] Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo.  Revisiting local descriptor based image-to-class measure for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7260–7268, 2019.
  * [193] Eunbyung Park and Junier B Oliva.  Meta-curvature.  arXiv preprint arXiv:1902.03356, 2019.
  * [194] Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang.  Fsce: Few-shot object detection via contrastive proposal encoding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7352–7362, 2021.
  * [195] Chenchen Zhu, Fangyi Chen, Uzair Ahmed, Zhiqiang Shen, and Marios Savvides.  Semantic relation reasoning for shot-stable few-shot object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8782–8791, 2021.
  * [196] Yang Xiao and Renaud Marlet.  Few-shot object detection and viewpoint estimation for objects in the wild.  In European Conference on Computer Vision, pages 192–210. Springer, 2020.
  * [197] Jiaxi Wu, Songtao Liu, Di Huang, and Yunhong Wang.  Multi-scale positive sample refinement for few-shot object detection.  In European Conference on Computer Vision, pages 456–472. Springer, 2020.
  * [198] Xin Wang, Thomas E Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu.  Frustratingly simple few-shot object detection.  arXiv preprint arXiv:2003.06957, 2020.
  * [199] Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin.  Meta r-cnn: Towards general solver for instance-level low-shot learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9577–9586, 2019.
  * [200] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.  Meta-learning to detect rare objects.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9925–9934, 2019.
  * [201] Zhihe Lu, Sen He, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang.  Simpler is better: Few-shot semantic segmentation with classifier weight transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8741–8750, 2021.
  * [202] Gengwei Zhang, Guoliang Kang, Yunchao Wei, and Yi Yang.  Few-shot segmentation via cycle-consistent transformer.  arXiv preprint arXiv:2106.02320, 2021.
  * [203] Juhong Min, Dahyun Kang, and Minsu Cho.  Hypercorrelation squeeze for few-shot segmentation.  arXiv preprint arXiv:2104.01538, 2021.
  * [204] Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye.  Prototype mixture models for few-shot semantic segmentation.  In European Conference on Computer Vision, pages 763–778. Springer, 2020.
  * [205] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia.  Prior guided feature enrichment network for few-shot segmentation.  IEEE Transactions on Pattern Analysis & Machine Intelligence, (01):1–1, 2020.
  * [206] Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He.  Part-aware prototype network for few-shot semantic segmentation.  In European Conference on Computer Vision, pages 142–158. Springer, 2020.
  * [207] Khoi Nguyen and Sinisa Todorovic.  Feature weighting and boosting for few-shot segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 622–631, 2019.
  * [208] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng.  Panet: Few-shot image semantic segmentation with prototype alignment.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9197–9206, 2019.
  * [209] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen.  Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5217–5226, 2019.
  * [210] Dan Andrei Ganea, Bas Boom, and Ronald Poppe.  Incremental few-shot instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1185–1194, 2021.
  * [211] Khoi Nguyen and Sinisa Todorovic.  Fapis: A few-shot anchor-free part-based instance segmenter.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11099–11108, 2021.
  * [212] Zhibo Fan, Jin-Gang Yu, Zhihao Liang, Jiarong Ou, Changxin Gao, Gui-Song Xia, and Yuanqing Li.  Fgn: Fully guided network for few-shot instance segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9172–9181, 2020.
  * [213] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang.  Sg-one: Similarity guidance network for one-shot semantic segmentation.  IEEE transactions on cybernetics, 50(9):3855–3865, 2020.
  * [214] Claudio Michaelis, Ivan Ustyuzhaninov, Matthias Bethge, and Alexander S Ecker.  One-shot instance segmentation.  arXiv preprint arXiv:1811.11507, 2018.
  * [215] Shuo Yang, Lu Liu, and Min Xu.  Free lunch for few-shot learning: Distribution calibration.  arXiv preprint arXiv:2101.06395, 2021.