# An Empirical Study of Low Precision Quantization for TinyML

Shaojie Zhuo Qualcomm AI Research Qualcomm Canada ULC
<EMAIL_ADDRESS>,  Hongyu Chen University of Toronto
<EMAIL_ADDRESS>,  Ramchalam Kinattinkara Ramakrishnan Qualcomm AI
Research Qualcomm Canada ULC<EMAIL_ADDRESS>,  Tommy Chen Qualcomm
AI Research Qualcomm Canada ULC<EMAIL_ADDRESS>,  Chen Feng
Qualcomm AI Research Qualcomm Canada ULC<EMAIL_ADDRESS>,  Yicheng Lin
Qualcomm AI Research Qualcomm Canada ULC<EMAIL_ADDRESS>,  Parker
Zhang Qualcomm AI Research Qualcomm Canada ULC<EMAIL_ADDRESS>and
Liang Shen Qualcomm AI Research Qualcomm Canada ULC<EMAIL_ADDRESS>
(2022)

###### Abstract.

Tiny machine learning (tinyML) has emerged during the past few years aiming to
deploy machine learning models to embedded AI processors with highly
constrained memory and computation capacity. Low precision quantization is an
important model compression technique that can greatly reduce both memory
consumption and computation cost of model inference. In this study, we focus
on post-training quantization (PTQ) algorithms that quantize a model to low-
bit (less than 8-bit) precision with only a small set of calibration data and
benchmark them on different tinyML use cases. To achieve a fair comparison, we
build a simulated quantization framework to investigate recent PTQ algorithms.
Furthermore, we break down those algorithms into essential components and re-
assembled a generic PTQ pipeline. With ablation study on different
alternatives of components in the pipeline, we reveal key design choices when
performing low precision quantization. We hope this work could provide useful
data points and shed lights on the future research of low precision
quantization.

tinyML, neural networks, quantization, benchamrk

††submissionid: 11††copyright: rightsretained††journalyear: 2022

##  1\. Introduction

Embedding AI directly on edge devices is becoming a key to revolutionize the
internet of things (IoT), where billions of tiny devices are leveraged to gain
great productivity and efficiency in areas including consumer, medical,
automotive and industrial. As a result, tiny machine learning (tinyML) has
emerged over the past few years that aims to deploy machine learning
algorithms, especially neural networks, on embedded AI processors such as
micro-controllers or embedded NPUs (e.g. Qualcomm’s eNPU (Qualcomm, 2021) and
ARM’s MicroNPU (Arm, 2021)) with very low power consumption in the level of a
few milli-watts. By performing inferences on the devices, tinyML brings the
advantage of high energy efficiency, fast responsiveness, high privacy and
strong autonomy of edge devices. However, the embedded AI processors are
usually highly resource-constrained with limited memory and computation
capability. A large body of literature has focused on addressing these issues
by making neural networks more efficient, including efficient neural
architecture search (Zoph and Le, 2016; Liu et al., 2019; Cai et al., 2020),
neural networks and system co-design (Lin et al., 2020, 2021) and model
compression such as pruning (Han et al., 2016), knowledge distillation (Hinton
et al., 2015) and quantization (Han et al., 2016; Krishnamoorthi, 2018).

Neural network quantization is one of the most important model compression
techniques. It compresses models by using low-bit precision representation for
weight and activation tensors, instead of the 32-bit (or 16-bit for half-
precision) precision that is commonly used during model training. The memory
consumption and computation of a low precision model are greatly reduced. And
quantization compresses a model without changing its architecture. This is
particularly useful when the model architecture is already optimally designed
for a specific inference system or device. Besides, quantization can be
applied along with other model compression techniques, such as pruning and
knowledge distillation.

Neural networks have been shown to be quite robust to quantization with 8-bit
precision without sacrificing the accuracy(Nagel et al., 2021). Quantization
with lower precision can further reduce the memory consumption and
computation. And ultra-low precision (1 or 2-bit) operations can often be
computed efficiently with bit-wise arithmetic and thus achieving signification
computation acceleration (Zhu et al., 2022). However, due to the large
quantization noise, the benefits of low precision quantization often come at
the cost of significant accuracy degradation. Among approaches to mitigate the
accuracy drop caused by quantization, post-training quantization (PTQ)
directly performs quantization on a pre-trained full precision model to
convert it to a corresponding low precision model. Unlike quantization-aware
training (QAT), PTQ does not require the full training pipeline of the
original model and can be applied with a small amount of calibration data (or
even data-free (Nagel et al., 2019)), thus becoming a popular model
quantization routine.

In this paper, we aim to study and benchmark recent PTQ algorithms on neural
network models designed for tinyML use cases. By doing that, we try to answer
the following questions.

First of all, what is the performance of PTQ algorithms on tinyML models?
Existing work (Nagel et al., 2019, 2020; Li et al., 2021; Esser et al., 2020;
Wang et al., 2020) has shown successful applications of low precision PTQ on
big neural network models for large-scale tasks such as image classification
on ImageNet. However, tinyML models are usually very compact and are designed
for relatively simple tasks. It is unknown whether those models can be
quantized to low precision while preserving their accuracy.

Secondly, what are the design choices making a PTQ algorithm superior to
others? Comparing the performance of different PTQ algorithms helps us to
select a proper quantization algorithm. And further study on their algorithm
design choices can reveal the key reason of an algorithm’s success and provide
insights for further improvement of the algorithm.

Last but not least, what is the trade-off between accuracy and
memory/computation when applying low precision quantization? This is an
important question when deploying a tinyML model. By knowing the sensitivity
of different models to quantization, we are able to seek the best trade-off
between accuracy and memory/computation considering the very limited resources
on tinyML devices.

By conducting extensive experiments with various tinyML models, we find
answers to the previous questions and many insightful observations. To
summarize our contributions:

  * •

We create a quantization simulation framework and an unified PTQ pipeline to
encourage fair comparison of different PTQ algorithms.

  * •

We conduct ablations to reveal the key design choices of the PTQ pipeline
using a set of models with varieties of architectures in tinyML.

  * •

we demonstrate that low precision quantization is useful for tinyML by
compressing a model to greatly save memory and computations while preserving
accuracy.

  * •

We find key gaps of low precision quantization for tinyML and point out the
directions of potential improvements.

We believe that low precision quantization is important to tinyML. And the
observations, challenges and open questions in this study are good data points
to understand the existing PTQ algorithms and the characteristic of low
precision quantization, and shed lights on the direction of future work to
improve the performance.

##  2\. Related work

#### Quantization

We refer to the readers a good overview of quantization basics and algorithms
in (Gholami et al., 2021). Quantization algorithms can be classified into
quantization-aware training (QAT) and post-training quantization (PTQ). QAT
re-trains a model with quantized parameters so that it converges to a point
with a better loss. An important challenge of the backpropagation during re-
training is how to treat the non-differentiable rounding operator in the
quantization function. Straight Through Estimator (STE) (Bengio et al., 2013)
is one of the main techniques used to approximate the gradient. Built on top
of that, several recent papers (Esser et al., 2020; Jain et al., 2020)
proposed to learn the quantization parameters together with the network
parameters and achieve better results. Although STE-based methods enable the
training of quantized networks with gradient-based optimization, the gradient
mismatch between forward and backward passes affects the training
effectiveness, especially for low precision quantization. To alleviate the
gradient mismatch, soft quantizers with sigmoid or tanh function (Gong et al.,
2019) are used to approximate the discrete rounding in both forward and
backward passes. These approaches, however, cause the accuracy drop due to the
discrepancy between the soft rounding during training and the hard rounding at
inference time. Despite the promising results given by QAT methods, they
usually need the whole training pipeline and much more computation. In
contrast, PTQ is attractive in the sense that it requires only a small set of
(unlabeled) calibration data and thus become the main focus of this study. A
fundamental problem in PTQ is finding good quantization parameters for both
weight and activation tensors. It is particular challenging when there is
imbalanced distribution among different channels of a tensor. To resolve that,
cross layer equalization (CLE) (Nagel et al., 2019) and outlier channel
splitting (OCS) (Zhao et al., 2019) are proposed to re-balance the
distribution of weights. Although good performance are achieved for 8-bit
quantization, PTQ suffers from severe performance degradation as precision
goes lower than 8-bit. Recently, a new paradigm makes huge progress for low
precision PTQ. The approach utilizes layerwise optimization to minimize the
error between the output of a quantized layer and that of its corresponding
full-precision layer, with the help of a small set of calibration data. A
series of work leveraging the layer-wise calibration paradigm, namely AdaRound
(Nagel et al., 2020), AdaQuant (Hubara et al., 2020), BitSplit (Wang et al.,
2020), BRECQ (Li et al., 2021), has push the limit of PTQ down to 2 bit. We
thus focus on the study of those methods and compare their performances.

#### Quantization for tinyML

There are works (Krishnamoorthi, 2018; Wu et al., 2020) focusing on the study
of quantization for general applications with big models in the study. In the
context of tinyML, (Novac et al., 2021) proposed a new QAT method for low
precision quantization. (Ghamari et al., 2021; Banbury et al., 2021) focused
on deployment of the 8-bit quantized models and their latency on actual tinyML
devices. By contrast, Our study aims on benchmarking the performance of the
state of the art quantization methods on low precision (less than 8-bit)
quantization for tinyML. We selected several representative applications in
tinyML to study the effects of different algorithm design choices and the
trade-off among accuracy, memory consumption and computational cost.

##  3\. Quantization framework

The performances of existing PTQ algorithms are usually reported based on
separate implementations without a common framework. The implementation
differences (e.g. the way how quantization is simulated) makes it difficult to
do a fair comparison of different algorithms. And existing work usually
focuses on improvement of different aspects in the quantization process
without optimizing the whole pipeline. To this end, we implement a
quantization framework to facilitate the quantization study. Built on top of
that, we propose a unified post-training quantization pipeline, so that
existing PTQ algorithms can be deconstructed and fitted into the pipeline as
improvement of parts of its components. It does not only make fair comparisons
of PTQ algorithms possible, but also has the flexibility to ablate on
different choices of each components in the pipeline and find the optimal one
that generate the best performance. In this section, we introduce our
quantization simulation framework and the proposed unified PTQ pipeline.

Figure 1. Simulated quantization for the forward pass of a convolution layer.

###  3.1. Simulated quantization

We use simulated quantization (Krishnamoorthi, 2018) in our framework. A
tensor is quantized and then de-quantized back to floating precision to
simulate quantization loss while the operations are carried out with floating
point arithmetic. Figure 1 shows an example of the quantized forward pass of a
convolution layer. Both input and weight tensor go through the quantization
function in the quantizer to simulate quantization and then are fed to the
convolution computation in full precision. The bias remains in full precision.
The quantization of other operators is simulated in a similar way.

Figure 2. The unified pipeline for PTQ. It takes a pre-trained full precision
model and a small set of calibration data as input and output a optimized
quantized model through multiple steps.

(a) optimization of the first quantized layer

(b) optimization of the second quantized layer

(c) optimization of the last quantized layer

Figure 3. Layerwise quantization optimization. With a small set of calibration
data, (a) the first quantized layer (in green) is activated and optimized,
while the rest of the quantized layers stay inactive (in gray). The first
quantizated is optimized to minimize the mean squared error(MSE) between its
output and that of the first FP layer; (b) the first quantized layer is
optimized and frozen (in orange) and the second quantized layer is activated
and optimized; (c) The quantized model is optimized layer by layer until the
last one.

We adopt the asymmetric quantization scheme in our framework for quantization
simulation. The asymmetric quantization function is defined by three
quantization parameters: the scale factor $s$, the zero-point $z$ and the bit-
width $b$,

(1)
$\hat{x}=q(x;s,z,b)=s\Big{[}clamp\Big{(}\Big{\lfloor}\frac{x}{s}\Big{\rceil}+z,n,p\Big{)}-z\Big{]}$

where $\hat{x}$ is the quantized approximation of $x$, $\lfloor\cdot\rceil$
indicates the round function and $clamp(\cdot)$ clamps values between $n$ and
$p$, which are determined by the bit-width $b$ and define the range of the
integer grids. As an simplified version of the asymmetric quantization,
symmetric quantization restricts the zero-point to be $0$ to reduce the
additional computation introduced by the zero-point offset. We follow (Nagel
et al., 2021) to use asymmetric quantization for activation tensors and
symmetric quantization for parameters, considering that the distributions of
activation tensors are often asymmetric about zero while those for parameters
are roughly symmetric about zero.

Quantization granularity is another factor affecting the accuracy of a
quantized model. Per-tensor quantization has a single set of quantization
parameter for the whole tensor while per-channel quantization has a separate
set of quantization parameters for each channel of the tensor. Per-channel
quantization usually introduces lower quantization noise due to its finer
granularity. In our framework, we use per-channel quantization for parameters
and per-tensor quantization for activation tensors, considering the readiness
of hardware support and to better preserve the accuracy of a model (Nagel et
al., 2021).

###  3.2. PTQ Pipeline

As illustrated in Figure 2, the proposed unified PTQ pipeline takes a pre-
trained full precision model with a set of calibration data as input and
produces the optimized low precision model through four sequential steps.

Step 1: FP model preprocessing. In this step, a full precision (FP) model is
pre-processed to make it more quantization friendly. One common issue for
quantization is the imbalanced distribution among various channels of the
tensor, which makes it hard to find a suitable quantization parameters for the
tensor. This is especially prevalent in the weights of efficient models with
depth-wise separable convolutions (Nagel et al., 2019). Fine-grained
quantization (e.g., per-channel quantization) is able to alleviate the issue.
Without it, cross-layer range equalization (CLE) (Nagel et al., 2019) and
Outlier channel splitting (OCS) (Zhao et al., 2019) are introduced to re-
balance the distribution of a tensor. Although we use per-channel quantization
for parameters, the activation tensors are quantized using per-tensor
quantization. It is worthwhile to show the impact of FP model preprocessing on
quantization.

Step 2: Add quantizers and initialization. After the preprocessing, we add
quantizers to a model to simulate quantization for both parameters and
activation tensors. To initialize the quantization parameters (the scale $s$
and zero-point $z$) of each quantizer, one common approach is to find the
clipping range $[\alpha,\beta]$ first and then use it to calculate the scale
and zero-point, i.e.,

(2) $s=\frac{\beta-\alpha}{2^{b}-1},z=\lfloor\frac{-\alpha}{s}\rceil+n$

And there are multiple ways to determine the clipping range,

  * •

MinMax initialization sets the clipping range using the min/max of a tensor.
It is simple but usually suffers from outliers.

  * •

MSE initialization finds the range that minimize the mean squared error (MSE)
between the original and quantized tensor using grid search or analytical
approximations
$\alpha^{*},\beta^{*}=\mathop{\arg\min}_{\alpha,\beta}\|x-\hat{x}\|^{2}_{F}$,where
$\hat{x}$ is the quantized version of $x$ and $\|\cdot\|_{F}$ is the Frobenius
norm.

Step 3: Quantization optimization. As shown in Equation 1, the quantization
function maps a floating point value to a low-precision value and then de-
quantize it back to a floating point value. During the process, clipping error
is introduced by the $clamp(\cdot)$ function and rounding error is introduced
by the round function $\lfloor\cdot\rceil$. Quantization optimization is the
process to to seek the right trade-off between the two errors. We adopt a
general layerwise optimization process, as illustrated in Figure 3. A low
precision model is optimized layer by layer to minimized the loss (usually the
mean squared error) between the outputs of the full precision layer and the
quantized layer. As shown in equation 1, multiple variables can potentially be
optimized,

  * •

Optimizing scale and zero-point (opt_qparam). Learnt step-size quantization
(Esser et al., 2020) treats the scale $s$ and zero-point $z$ as trainable
parameter and optimizes them to minimize the layerwise MSE.

  * •

Optimizing weights (opt_weights). AdaQuant (Hubara et al., 2020) extends the
flexibility and optimizes the weights directly to minimize the layerwise MSE.

  * •

Optimizing bits (opt_bits). Instead of working on the floating point, BitSplit
(Wang et al., 2020) performs the optimization on the quantized values directly
and splits the integer into multiple bits and optimizes them to minimize the
layerwise MSE.

  * •

Optimizing rounding (opt_round). AdaRound (Nagel et al., 2020) optimizes the
rounding by introducing a auxiliary parameter for each weight and constraints
it to be either $0$ or $1$ to achieve rounding up or down.

In the qunatization optimization step, instead of layer-wise optimization,
block-wise optimization (Li et al., 2021) is also proposed to capture the
cross-layer dependency within the block and it is clained that block-wise
optimization could be a better bias-variance trade-off choice with limited
calibration data for low precision quantization.

Step 4: LP model post-processing. After quantization optimization, bias tuning
(Hubara et al., 2020) can be applied on the low-precision (LP) model to update
biases in a model to further compensate for quantization loss. The biases are
updated to minimize the MSE of the final outputs between FP model and the LP
model. In theory, the bias tuning can be done during the quantization
optimization, but the rationale to put it in the post-processing stage is
that, by updating a limited part of the model parameters in a end-to-end
fashion, it can provide global guidance to optimize for the final output
without over-fitting.

With the unified pipeline, existing PTQ algorithms can be deconstructed and
fitted into the pipeline. We vary each component in the pipeline to construct
new PTQ algorithms. We do ablations on different options for each component to
reveal the best options and then use them to construct the best PTQ algorithm.

##  4\. Experiments

In this section, we benchmark and ablate the performance of recent PTQ
algorithms on multiple tinyML models. We first describe the tinyML tasks and
models selected for the experiments and then show the experiment results of
ablations on different components of the PTQ pipeline and results of end-to-
end study. In our experiments, for all the models, we report the mean and
standard deviation of the accuracy, calculated using 5 runs with different
initial seeds. And we use 1024 unlabeled data extracted from the original
training data as the calibration dataset.

###  4.1. TinyML tasks and models

We select four use cases for our study: image classification, visual wake
words, Keyword spotting and human activity recognition. The first three are
from the MLPerf Tiny benchmark (Banbury et al., 2021). The selection is based
on the relevance to real world applications, the availability of open-sourced
dataset and models, and the coverage of a wide variety of applications and a
wide variety of model architectures in tinyML. We train each model based on
the recommended setting in (Banbury et al., 2021) to get the full precision
models. A summary of the tasks and models are shown in Table 1.

Use case | Dataset | Model | Float Accuracy | MAC | Num Param. | Peak activation
---|---|---|---|---|---|---
Image Classification | CIFAR10 | Res8 | $86.75$ | $12,591,808$ | $77,706$ | $36,608$
Keyword Spotting | Speech Commands | DS-CNN | $93.03$ | $2,736,832$ | $22,604$ | $36,864$
Visual Wake Words | VWW | MobileNetV1 | $85.10$ | $7,723,776$ | $210,850$ | $32,768$
Human Activity Recognition | UCI-HAR | CNN | $90.36$ | $2,298,368$ | $523,462$ | $8,064$
Table 1. TinyML tasks and models selected for the evaluation of quantization
algorithms. The detailed architecture of each model can be found in the
appendix

###  4.2. Ablations of the PTQ pipeline

|
---|---
|

(a) Initialization: Res8 (CIFAR10) (b) Initialization: DS-CNN(Speech Command)

(c) Pre-processing: Res8 (CIFAR10) (d) Pre-processing: DS-CNN(Speech Command)

Figure 4. Ablation study on FP model preprocessing and quantizer
initialization for both weights and activations. (a) and (b) show the accuracy
after quantizer intialization. (c) and (d) show the accuracy after MSE-bsed
quantizer initialization with and without CLE as the FP model preprocessing.

|
---|---

(a) Res8 (CIFAR10) (b) DS-CNN(Speech Command)

Figure 5. Ablation study on optimization variables during quantization
optimization. Better zoom in to view the results.

FP model preprocessing and quantizer initialization. We consider FP model pre-
processing and quantizer initialization together in this study to see how the
combination of the two affects the accuracy. Figure 4 (a) and (b) show the
comparison of model accuracy with different initialization methods. We need to
ignore the results here for $2W2A$ because the quantized model after
initialization performs as a random classifier caused by large quantization
noise. From the results, we can see that MSE-based quantizer initialization
usually outperforms MinMax-based one. And MSE-based initialization for both
weights and activations performs the best consistently, this is aligned with
previous study on quantizer initialization (Nagel et al., 2021). For FP model
preprocessing, we consider cross layer equalization (CLE) because it is
effective without changing the architecture of the model. Figure 4 (c) and (d)
compare the model accuracy with and without CLE after MSE-based
initialization. We see that there is no accuracy discrepancy for higher
bitwidth. Although we use per-channel quantization for weights, CLE still
helps to improve the accuracy for low precision cases ($3W3A$), while it has
negative impact for the precision in between($4W4A$ or $5W5A$). This is
because activation tensors have per-tensor quantization and CLE also changes
the distribution of activation tensors.

Optimization variable. As discussed in section 3.2, different variables can be
optimized to minimize the layer-wise loss. We investigate the effect of
optimizing different variables. Please note that only the weights are
quantized while activation are in full precision for this study. As shown in
Figure 5, we can see that opt_qparam is clearly less effective compared with
the other three, especially when the bit-width is low. It is because that only
the quantization parameters (zero-points and offsets) are trainable, which
restricted its capability to to compensate for the large quantization noise
introduced by low precision quantization. opt_round and opt_bits usually
outperforms opt_weights. We believe that the STE (Bengio et al., 2013) used in
opt_weights causes biased gradient estimation under low-precision cases that
affects the efficacy of optimization. And the constrained optimization space
in opt_bits and opt_round makes them less prone to over-fitting with limited
calibration data.

|
---|---

(a) Res8 (CIFAR10) (b) DS-CNN(Speech Command)

Figure 6. Ablation study on optimization granularity during quantization
optimization. The solid lines represent the layerwise optimization and the
dotted lines represent the blockwise optimization. Better zoom in to view the
results.

Optimization granularity. We conduct the study on optimization granularity and
the results are shown in Figure 6. The solid lines represent the layerwise
optimization and the dotted lines represent the blockwise optimization.
opt_bits is excluded here because it doesn’t support blockwise optimization by
design. The block is defined as the basic residual block in Res8 and the depth
separable convolution in DS-CNN. We observe that both layerwise optimization
and blockwise optimization have very similar performance when the models are
quantized to $4W4A$ and higher, regardless of the different optimization
variables. For quantization with precision lower than $4W4A$ , blockwise
optimization provides quite significant improvement on the accuracy for Res8
quantized to $3W3A$ and $2W2A$, and for DS-CNN quantized to $3W3A$.

|
---|---

(a) Res8 (CIFAR10) (b) DS-CNN(Speech Command)

Figure 7. Ablation study on bias tuning for LP model post-processing. The
solid lines represent optimization without post-processing and the dotted
lines represent the optimization with post-processing. Better zoom in to view
the results.

LP model Post-processing. We also conduct experiment to show the impact of
bias tuning as the low precision model post-processing. As show in Figure 7,
bias tuning consistently improve the accuracy for all the methods (except for
the case of DS-CNN using opt_qparam). For Res8 using opt_qparam, it has more
than $10\%$ improvement for both $3W3A$ and $2W2A$ quantization. This
empirically shows the effectiveness of end-to-end bias tuning.

###  4.3. Results

Bitwidth | Method | Res8 | DS-CNN | MobileNetv1 | CNN
---|---|---|---|---|---
(W/A) | (CIFAR10) | (KWS) | (VWW) | (UCI-HAR)
32/32 | full precision | 86.75 | 93.03 | 85.10 | 90.36
8/8 | opt_qparam | 86.81±0.06 | 92.63±0.14 | 85.13±0.12 | 90.39±0.17
opt_weights | 86.76±0.04 | 92.55±0.06 | 85.14±0.05 | 90.81±0.24
opt_bits | 86.70±0.10 | 91.93±0.10 | 85.28±0.15 | 90.04±0.13
opt_round | 86.65±0.14 | 91.65±0.20 | 84.98±0.21 | 90.37±0.26
7/7 | opt_qparam | 86.79±0.11 | 92.57±0.10 | 85.18±0.16 | 90.54±0.24
opt_weights | 86.69±0.08 | 92.62±0.14 | 85.15±0.17 | 90.88±0.31
opt_bits | 86.64±0.08 | 91.92±0.15 | 85.16±0.17 | 89.98±0.15
opt_round | 86.70±0.14 | 91.54±0.21 | 84.87±0.11 | 90.37±0.35
6/6 | opt_qparam | 86.67±0.15 | 92.01±0.19 | 84.90±0.14 | 90.24±0.11
opt_weights | 86.71±0.14 | 92.19±0.19 | 85.06±0.10 | 90.86±0.22
opt_bits | 86.63±0.06 | 91.80±0.14 | 85.11±0.10 | 90.03±0.31
opt_round | 86.64±0.06 | 90.56±0.31 | 84.81±0.18 | 90.20±0.23
5/5 | opt_qparam | 86.35±0.03 | 90.86±0.28 | 84.33±0.19 | 90.61±0.41
opt_weights | 86.47±0.12 | 90.82±0.20 | 84.82±0.15 | 91.00±0.47
opt_bits | 86.37±0.15 | 90.42±0.36 | 84.39±0.16 | 90.09±0.21
opt_round | 86.64±0.10 | 90.13±0.93 | 84.21±0.24 | 90.41±0.30
4/4 | opt_qparam | 84.61±0.15 | 83.31±1.02 | 78.40±1.52 | 91.12±0.22
opt_weights | 85.91±0.18 | 88.41±0.25 | 82.96±0.23 | 90.76±0.13
opt_bits | 85.33±0.09 | 87.88±0.36 | 81.72±0.62 | 90.38±0.32
opt_round | 85.34±0.32 | 88.33±0.95 | 81.89±0.50 | 90.57±0.35
3/3 | opt_qparam | 74.30±0.53 | 29.58±4.60 | 56.79±1.43 | 90.57±0.23
opt_weights | 82.62±0.13 | 63.99±2.98 | 76.03±1.36 | 90.83±0.31
opt_bits | 81.81±0.14 | 63.95±2.66 | 74.45±0.71 | 90.32±0.27
opt_round | 83.21±0.41 | 75.22±1.57 | 76.71±1.21 | 91.38±1.15
2/2 | opt_qparam | 25.26±2.2 | 9.16±1.18 | 47.15±0.00 | 82.71±0.76
opt_weights | 62.81±0.53 | 8.83±0.85 | 62.24±0.58 | 90.46±0.54
opt_bits | 63.72±0.47 | 8.77±0.23 | 62.48±0.34 | 86.74±0.59
opt_round | 69.40±0.74 | 9.42±0.51 | 60.58±0.96 | 89.63±0.83
Table 2. Comparison of PTQ pipelines with different optimization variables
while fixing other options the best in the pipeline.

To compare end-to-end performance with both weight and activation quantization
enabled, we fix all the options in the pipeline using the best ones based on
ablation study results, except for the optimization variables. We use
symmetric per-channel quantization for weights and asymmetric per-tensor
quantization for activations. The FP model pre-processing is enabled and MSE-
based initialization is applied on both weight quantizers and activation
quantizers. And we use layerwise optimization with bias tuning enabled as LP
model post-processing. The results are shown in Table 2. For a higher bit-
width ($5W5A$ to $8W8A$), different PTQ pipelines perform similarly with very
small accuracy drop compared to the full precision accuracy. However, for a
lower bit-width, different pipelines show big performance discrepancy. The
pipelines using opt_round usually perform better. However, even for the best
quantization algorithm, there is still big accuracy drop when going to ultra-
low precision. The accuracy for $2W2A$ is $69.40\%$ for CIFAR10, with accuracy
drop as large as $17.5\%$.

---
(a) (b)

Figure 8. Tradoff between accuracy and memory/computation saving. $4W4A$ is a
sweet point having large memory/computation saving with small accuracy drop.
Some models can go lower precision with negligible accuracy drop.

To better show the benefits of low precision quantization, we view the results
on two axes: the accuracy drop of the best PTQ pipeline, and corresponding
computation saving (or memory saving) with different bit-widths. We use bit
operation (BOP) as a system-agnostic proxy to the computation complexity. The
BOP count of a layer $l$ is computed as: $BOP(l)=MACs(l)b_{w}b_{a}$, where
$MACs(l)$ is the multiply-accumulate count of the layer, $b_{w}$ is the bit-
width of the weight and $b_{a}$ is the bit-width of the input. We use peak
memory as the measurement of the memory consumption, which is computed as
$peak\\_memory=total\\_weight*b_{w}+peak\\_activation*b_{a}$. it measures the
maximum memory required to store the weights and activations during
inferences, assuming that previous intermediate activations can be effectively
released if they are not used by the later layers. The results are shown in
Figure 8. We can see that $4W4A$ is a sweet point that has $50\%$ memory
saving and $75\%$ computation saving with small accuracy drop ($<5\%$). This
is mismatching with most inference frameworks that support only 8-bit
quantization (Ghamari et al., 2021). Some models can even tolerant lower
precision quantization with negligible accuracy drop. For example, the CNN
model for UCI-HAR can be quantized to 2W2A with less than $2\%$ accuracy drop.
However, this could be a model/task dependent behavior. For most of the
models, we have to sacrifice the accuracy if going for ultra-low precision
such as 3W3A or 2W2A, and thus facing the accuracy and memory/computation
trade-off. We can clearly see the benefits of low precision quantization in
terms of computation and memory saving (and eventually power saving).

##  5\. Conclusion and Future Directions

In this paper, with a unified quantization pipeline, we benchmark PTQ
algorithms on carefully selected models for tinyML. We conduct ablations to
reveal the key design choices of the pipeline. We show that low precision
quantization with the best PTQ algorithm can have huge memory and computation
savings with negligible accuracy drop, although there is still big accuracy
drop if quantized models to ultra-low precision. Our empirical study provides
useful data points on low precision quantization and points out potential
future directions for improvement on algorithm design and system support for
tinyML.

How to further improve the accuracy for ultra-low precision quantization is an
important direction for future research. Based on our unified PTQ pipeline,
new improvement on different components can further reduce the accuracy
degradation. And mix-precision quantization and quantization aware training
could be potential solutions. Besides, existing embedded AI frameworks, such
as TensorFlow Lite Micro (Google, 2021) and STM32Cube.AI (STMicroelectronics,
2021), can only support computation in floating point or 8-bit integers. The
framework with lower precision quantization support is not widely available.
It first leaves big room to squeeze the model by combining quantization with
other model compression techniques. And it is important to investigate the
best combination in practice. At the meanwhile, developing efficient embedded
AI frameworks with low precision quantization support will be particularly
useful for deployment of tinyML models.

## References

  * (1)
  * Arm (2021) Arm. 2021.  Ethos-U55 Machine Learning Processor.  https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-u55.  last visited 05/03/2022.
  * Banbury et al. (2021) Colby Banbury, Vijay Janapa Reddi, Peter Torelli, Jeremy Holleman, Nat Jeffries, Csaba Kiraly, Pietro Montino, David Kanter, Sebastian Ahmed, Danilo Pau, et al. 2021\.  MLPerf Tiny Benchmark.  _Neurips_ (2021).
  * Bengio et al. (2013) Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013\.  Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation.  _arXiv:1308.3432_ (2013).
  * Cai et al. (2020) Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. 2020.  Once-for-All: Train One Network and Specialize it for Efficient Deployment. In _ICLR_.
  * Esser et al. (2020) Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra Modha. 2020.  Learned Step Size Quantization.  _ICLR_.
  * Ghamari et al. (2021) Sedigh Ghamari, Koray Ozcan, Thu Dinh, Andrey Melnikov, Juan Carvajal, Jan Ernst, and Sek Chai. 2021.  Quantization-Guided Training for Compact TinyML Models.  _TinyML Research Symposium 2021_ (2021).
  * Gholami et al. (2021) Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. 2021\.  A Survey of Quantization Methods for Efficient Neural Network Inference.  _arXiv:2103.13630_ (2021).
  * Gong et al. (2019) Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. 2019\.  Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks.
  * Google (2021) Google. 2021.  TensorFlow Lite for Microcontrollers.  https://www.tensorflow.org/lite/microcontrollers.  last visited 05/03/2022.
  * Han et al. (2016) Song Han, Huizi Mao, and William J. Dally. 2016.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. In _ICLR_.
  * Hinton et al. (2015) Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015\.  Distilling the Knowledge in a Neural Network.  _arXiv:1503.02531_ (2015).
  * Hubara et al. (2020) Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. 2020.  Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming.  _arXiv:2006.10518_ (2020).
  * Jain et al. (2020) Sambhav R. Jain, Albert Gural, Michael Wu, and Chris H. Dick. 2020\.  Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks.  _arXiv:1903.08066_ (2020).
  * Krishnamoorthi (2018) Raghuraman Krishnamoorthi. 2018\.  Quantizing deep convolutional networks for efficient inference: A whitepaper.
  * Li et al. (2021) Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021\.  BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction. In _ICLR_.
  * Lin et al. (2021) Ji Lin, Wei-Ming Chen, Han Cai, Chuang Gan, and Song Han. 2021.  MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning. In _NeurIPS_.
  * Lin et al. (2020) Ji Lin, Wei-Ming Chen, John Cohn, Chuang Gan, and Song Han. 2020.  MCUNet: Tiny Deep Learning on IoT Devices. In _NeurIPS_.
  * Liu et al. (2019) Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019\.  DARTS: Differentiable Architecture Search. In _ICLR_.
  * Nagel et al. (2020) Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020\.  Up or Down? Adaptive Rounding for Post-Training Quantization. In _ICML_.
  * Nagel et al. (2021) Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. 2021.  A White Paper on Neural Network Quantization.  _arXiv:2106.08295_ (2021).
  * Nagel et al. (2019) Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. 2019.  Data-Free Quantization Through Weight Equalization and Bias Correction. In _ICCV_.
  * Novac et al. (2021) Pierre-Emmanuel Novac, Ghouthi Boukli Hacene, Alain Pegatoquet, Benoît Miramond, and Vincent Gripon. 2021\.  Quantization and Deployment of Deep Neural Networks on Microcontrollers.  _Sensors_ (2021).
  * Qualcomm (2021) Qualcomm. 2021.  Hexagon DSP SDK Tools and Resources.  https://developer.qualcomm.com/software/hexagon-dsp-sdk/tools.  last visited 05/03/2022.
  * STMicroelectronics (2021) STMicroelectronics. 2021\.  STM32Cube.AI.  https://www.st.com/en/embedded-software/x-cube-ai.html.  last visited 05/03/2022.
  * Wang et al. (2020) Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. 2020\.  Towards Accurate Post-training Network Quantization via Bit-Split and Stitching. In _ICML_.
  * Wu et al. (2020) Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. 2020.  Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation.  _arXiv:2004.09602_ (2020).
  * Zhao et al. (2019) Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Christopher De Sa, and Zhiru Zhang. 2019.  Improving Neural Network Quantization without Retraining using Outlier Channel Splitting.  _arXiv:1901.09504_ (2019).
  * Zhu et al. (2022) Shien Zhu, Luan H. K. Duong, and Weichen Liu. 2022\.  TAB: Unified and Optimized Ternary, Binary and Mixed-Precision Neural Network Inference on the Edge.  _ACM Transactions on Embedded Computing Systems_ (2022).
  * Zoph and Le (2016) Barret Zoph and Quoc V. Le. 2016.  Neural Architecture Search with Reinforcement Learning.  _CoRR_ (2016).

##  Appendix A Model architectures

Here we show the basic blocks used to define the models and the detailed
architecture of each TinyML model.

|  |
---|---|---

(a) DSConv (b) ResBlock(stride=$1$)

(c) ResBlock(stride=$2$)

Figure 9. The basic blocks used in the model architectures. Input | Operator | #out | stride
---|---|---|---
$32\times 32\times 3$ |  Conv2d, $3\times 3$ | 16 | 1
$32\times 32\times 16$ | ResBlock | 16 | 1
$32\times 32\times 16$ | ResBlock | 32 | 2
$16\times 16\times 32$ | ResBlock | 64 | 2
$8\times 8\times 64$ |  AvgPool, $8\times 8$ | - | 1
$1\times 64$ | Fully Connected | 10 | -
Table 3. Res8 (CIFAR10) architecture definition Input | Operator | #out | stride
---|---|---|---
$10\times 49\times 1$ |  Conv2d, $4\times 10$ | 64 | 2
$5\times 25\times 64$ | DSConv | 64 | 1
$5\times 25\times 64$ | DSConv | 64 | 1
$5\times 25\times 64$ | DSConv | 64 | 1
$5\times 25\times 64$ | DSConv | 64 | 1
$5\times 25\times 64$ | AvgPool, 5x25 | - | 1
$1\times 64$ | Fully Connected | 12 | -
Table 4. DSCNN(Speech Commands) architecture definition Input | Operator | #out | stride
---|---|---|---
$96\times 96\times 3$ |  Conv2d, $3\times 3$ | 8 | 2
$48\times 48\times 8$ | DSConv | 16 | 1
$48\times 48\times 16$ | DSConv | 32 | 2
$24\times 24\times 32$ | DSConv | 32 | 1
$24\times 24\times 32$ | DSConv | 64 | 2
$12\times 12\times 64$ | DSConv | 64 | 1
$12\times 12\times 64$ | DSConv | 128 | 2
$6\times 6\times 128$ | DSConv | 128 | 1
$6\times 6\times 128$ | DSConv | 128 | 1
$6\times 6\times 128$ | DSConv | 128 | 1
$6\times 6\times 128$ | DSConv | 128 | 1
$6\times 6\times 128$ | DSConv | 256 | 2
$3\times 3\times 256$ | DSConv | 256 | 1
$3\times 3\times 256$ |  AvgPool, $3\times 3$ | - | 1
$1\times 256$ | Fully Connected | 2 | -
Table 5. MobileNetv1(VWW) architecture definition Input | Operator | #out | stride
---|---|---|---
$128\times 9$ |  Conv1d, $3$ | 64 | 1
$126\times 64\times 16$ |  Conv1d, $3$ | 64 | 1
$124\times 64$ | MaxPool | - | 2
$62\times 64$ | Flatten | - | -
$1\times 3960$ | Fully Connected | 6 | -
Table 6. CNN(UCI-HAR) architecture definition


# Enhancing Pollinator Conservation towards Agriculture 4.0: Monitoring of
Bees through Object Recognition

Ajay John Alex
Department of Computer Science
Nottingham Trent University
Nottingham
&Chloe M. Barnes
Department of Applied AI and Robotics
ACAIRA
Aston University
Birmingham
&Pedro Machado
Department of Computer Science
Nottingham Trent University
Nottingham
&Isibor Ihianle
Department of Computer Science
Nottingham Trent University
Nottingham
&Gábor Markó
Department of Plant Pathology
Hungarian University of Agriculture
and Life Sciences
Budapest
&Martin Bencsik
Department of Physics
Nottingham Trent University
Nottingham
&Jordan J. Bird
Department of Computer Science
Nottingham Trent University
Nottingham

###### Abstract

In an era of rapid climate change and its adverse effects on food production,
technological intervention to monitor pollinator conservation is of paramount
importance for environmental monitoring and conservation for global food
security. The survival of the human species depends on the conservation of
pollinators. This article explores the use of Computer Vision and Object
Recognition to autonomously track and report bee behaviour from images. A
novel dataset of 9664 images containing bees is extracted from video streams
and annotated with bounding boxes. With training, validation and testing sets
(6722, 1915, and 997 images, respectively), the results of the COCO-based YOLO
model fine-tuning approaches show that YOLOv5m is the most effective approach
in terms of recognition accuracy. However, YOLOv5s was shown to be the most
optimal for real-time bee detection with an average processing and inference
time of 5.1ms per video frame at the cost of slightly lower ability. The
trained model is then packaged within an explainable AI interface, which
converts detection events into timestamped reports and charts, with the aim of
facilitating use by non-technical users such as expert stakeholders from the
apiculture industry towards informing responsible consumption and production.

_K_ eywords Object Recognition $\cdot$ Computer Vision $\cdot$ Agriculture
$\cdot$ Apiculture

##  1 Introduction

The decline of pollinators, particularly bees, has emerged as a critical
concern with adverse effects on global food security. A loss of 1-10% of
biodiversity per decade has also been observed in recent times[1]. Various
species of bee play a key role in agricultural production, supporting a wide
array of crops, including fruits, vegetables, oilseeds and legumes, just to
name a few; Animal pollination supports 30% of global food production[2]. In
the United Kingdom alone, 34% of all pollination is provided by one species,
the European honey bee (Apis mellifera)[3]. Factors such as habitat loss and
rapid climate change have led to a worrying decline in bee populations around
the world, which poses a direct threat to agricultural sustainability.

During these troubling times, the potential of technological intervention
through computational intelligence presents a promising approach to mitigate
these problems. The notion of Agriculture 4.0 is largely based on data-centric
automation[4, 5, 6, 7], which will lead to improvements in agricultural
practices in terms of speed and efficiency through the use of technologies
such as the Internet of Things (IoT) and edge-based processing, Big Data,
Artificial Intelligence (AI) and Machine Learning, as well as Robotics, among
others. Precision agriculture applied to apicultural practices can help us
autonomously monitor bee behaviour and, in the future, could provide a
noninvasive real-time approach to monitor colonies in the long term. This
article introduces a novel application of vision-based computational
intelligence algorithms towards bee identification and tracking from video
streams, which aim to streamline otherwise labour-intensive manual processes.
We show that state-of-the-art recognition algorithms such as those within the
YOLO suite of research can provide real-time and highly precise bee tracking
and that their findings can be distilled into an accessible format for non-
technical stakeholders from industry. As shown in [8] and [9], data
augmentation and hyperparameter optimisation are key considerations when
aiming for approaches that are useful in the real world beyond the laboratory;
the novel combination and application of these state-of-the-art approaches,
along with the release of our dataset for future interdisciplinary research,
aims to facilitate academic research on the conservation of pollinator
populations.

These aforementioned abilities of object recognition algorithms provide timely
benefits to saving time and effort in an autonomous monitoring process and
further enabling larger-scale bee behaviour analysis. As argued by Ngo et
al.[10], real-time imaging plays a critical role in monitoring honeybee
behaviours to assess colony health[11]. This is especially important during a
time of great habitat loss, insecticide use, and rapid climate change. In
[10].

In addition to collecting and annotating a novel large-scale dataset, this
study also features scientific novelty in the evaluation of several data
augmentation and object detection approaches for bee detection. In addition to
the data, all algorithms and models are also released as open source. The
literature review reveals that many object detection experiments taking place
within precision agriculture and biodiversity studies focus primarily on model
ability, while this work also explores the potential for real-time execution.
The results show that YOLOv5m emerges as the most accurate approach overall,
while YOLOv5s exhibits the most promising performance for real-time use in
industry. This study also highlights preliminary results on the significant
reduction in the inference time for bee detection when using keyframe
selection.

Beyond technical achievements, this work encapsulates a broader vision for the
future of sustainable beekeeping and the conservation of bee colonies. In
addition to the trained machine learning models released as open source, an
explainable AI interface is also produced that infers the predictions made by
the model and distils them into meaningful and useful information for key
stakeholders. Therefore, the addition of this module aims to bridge the gap
between complex computational intelligence algorithms and practical apiculture
in the real world. Thus, this approach not only improves the state of the art
in beekeeping practice, but also contributes to the global agendas of
responsible production, climate action, and the preservation of terrestrial
ecosystems. The survival of the human species depends on these natural
ecosystems.

The scientific contributions arising from this comprehensive exploration of
object recognition for bee tracking surround both data and vision models. We
collect, annotate, and release a dataset of 9,664 images which is available
open-source to the research community. In addition, our proposed approach
yields several sets of promising results, including the YOLOv5m model, which
achieves an<EMAIL_ADDRESS>of 85.6% at 8.1ms per frame. We also release an
explainable AI interface prototype to make the model’s findings accessible to
non-technical key stakeholders in order to facilitate real-world application
and contribute to broader goals of sustainable agriculture and environmental
conservation.

The remainder of this article is as follows. A review of the literature is
presented in Section 2, covering the relevant work in apiculture and providing
a technical background on the techniques applied in this work. The methodology
followed in this work is described in Section 3, before the results are
discussed in Section 4. Section 5 then suggests future work arising from the
findings of this study and finally concludes this article.

##  2 Background and Related Work

This section explores relevant state-of-the-art literature in the field.
First, we explore work from an agricultural background before describing the
state-of-the-art in object detection techniques, which are relevant to the
experiments carried out in this article.

###  2.1 Agriculture, Conservation, And Biodiversity

Bees are an important pollinator in the global ecosystem, helping plant
reproduction, and crop quantity and quality. Pollinator services are critical
to food production and security; however, bee populations have recently
declined. This decline presents a substantial danger to agricultural
productivity and the long-term viability of food production[12], which is not
only a major contribution to GDP but sustains the survival of the human
species[13].

This decline in pollinators, particularly bees, has become a critical concern
with adverse effects on global food security. As described in the
introduction, the bee plays a key role in the agricultural production of both
food and industrial crops. Animal pollination supports 30% global food
production[2], and, beyond food, also plays a significant role in industrial
crops. This includes fibres, biofuels, medicinal, material and ornamental
plants[14]. Contributing to the wider discussion of environmental
sustainability, pollinators, including bees, play a significant role in the
cultivation of biofuel crops[15, 16]. That is, by facilitating the pollination
of biofuel crops such as canola and sunflowers, bees directly improve seed
production and thus the availability of these resources, which are critical in
sustainable solutions to energy in the global initiative to move away from
fossil fuels.

Key stakeholders and beneficiaries, such as farmers in the apicultural field,
can implement pollinator-friendly practices that improve agricultural
sustainability. For example, the avoidance of harmful insecticides[17, 18].
Some insecticides can be harmful to bees; symptoms include disruption of
navigation, feeding behaviour, and reproduction and immune systems[19, 20].
These negative effects result in a decrease in colony survival rates and
contribute to population decline[18]. Furthermore, climate change has become
another risk to bee populations globally[21]; availability of flowers, nesting
locations, and general natural dynamics are negatively affected by the
observed rise in temperatures and altered precipitation patterns. Autonomous
monitoring of bees can provide additional information on their response to
these environmental changes, with the benefit of not requiring human presence
throughout the whole data collection process.

This technique contributes to the wider field of environmental monitoring,
which is the systematic analysis of the natural environment[22]. This can
enable researchers to understand the current state of the environment,
forecast future trends over time, and ultimately assess the impact of human
activity on natural systems. Although this article focuses on pollinator
behaviour with a focus on the bee, environmental monitoring also encapsulates,
but is not limited to, air, water, and soil quality, as well as biodiversity
and climate analysis. In the context of this article, monitoring refers to in
situ analysis of bee habitats and behavioural patterns to better understand
their ecological importance and contribute to conservation efforts with
autonomous analysis and data collection.

Technological intervention to monitor the decline in biodiversity has been
promoted in recent times, especially due to the autonomous nature of
artificial intelligence and its ability not only to save time and effort, but
also to perform tasks for which key stakeholders and beneficiaries simply do
not have time to do manually[23, 24]. To this end, researchers suggest that
tracking pollinator behaviour can inform agricultural practices and
potentially protect or even improve current crop yield. This could be, for
example, information to help optimise crop placement, ensure pollination
coverage, and improve the overall efficiency of agricultural systems.

Bee behaviour is often monitored through various modalities of sensor-based
data collection and machine learning, such as in [25], where swarming
behaviour was recognised from vibro-acoustic accelerometer data at an accuracy
greater than 90%. Related work in the field includes [26], which argues that
visual alterations, such as background clipping and approximations, can
improve pollinator detection through data preprocessing. In [10], researchers
suggested the use of techniques that include background subtraction, Kalman
filtering, and the Hungarian algorithm to produce a system that can monitor
the activity that occurs at the entrance of the beehives. The approach
achieved around 93.9% accuracy ($\pm$1.1%) compared to manual counting.
Similar augmentation techniques were proposed in [27], where bees in complex
outdoor environments were tracked with 86.6% accuracy using the YOLOv2 object
detection model.

Environmental monitoring has benefitted from data augmentation as a method to
improve the ability to perform automated recognition. In [28], the authors
propose that the use of GAN-based synthetic data can help improve the
detection of various objects present in forest environments toward forming a
more technologically enhanced environmental monitoring system. Kaur et al.
proposed a data augmentation strategy to better recognise honeybee
disease[29], which used a GAN-based approach to improve classification
accuracy over conventional augmentation methods. In [30], researchers showed
that traditional image augmentation techniques could improve computer vision-
based classification of honeybee subspecies, and Buschbacher et al.[31] also
followed a similar methodology in autonomous recognition of bees.

The concept of autonomous object detection is at the core of several of the
reviewed related works, as well as the main technology behind the approach
proposed by this study. This technique is discussed in the following section.

###  2.2 Object Detection

Several related works to this study make use of Bounding Box Object Detection
(BBOD) for pollinator monitoring. BBOD is a computer vision technique that
predicts the location of a bounding box, or several bounding boxes, around
objects of interest. For example, vision systems within an autonomous vehicle
aim to draw boundaries or completely segment entities from an image to
understand the scene[32], such as the detection of an emergency situation
where a pedestrian has been detected on a high-speed road.

In [33], a model architecture known as You Only Look Once (YOLO) was proposed,
with the aim of implementing real-time BBOD. YOLO is a common deep learning
architecture for BBOD that performs the detection task as a single regression
problem, directly predicting bounding boxes and class probabilities from full
input images in one evaluation. This is unlike many other techniques, where
proposals are first generated and then later classified iteratively. YOLO
operates a grid-based approach, where predictions are made for each grid cell
for bounding box coordinates and two confidence scores (one for the box
containing the object and one for the class label of the object itself). More
details on the differences between model architectures can be found in [34].

Several metrics are important for object detection, which are featured in this
work due to their importance in the state-of-the-art. They include
Intersection over Union (IoU), which is a metric used to evaluate the accuracy
of an object detection algorithm on a particular dataset. IoU is the overlap
between the predicted bounding box and the ground-truth derived from the
expert annotation:

$\text{IoU}=\frac{\text{Area of Overlap}}{\text{Area of Union}}$ (1)

Several different IoU variants, such as GIoU (Generalised IoU) and DIoU
(Distance IoU) metrics, have been proposed in the relevant literature to
improve the accuracy of object detection algorithms[35]. These metrics
consider the size and shape of the predicted and ground-truth bounding boxes,
as well as their location and orientation.

Similarly to other machine learning problems, precision and recall are also
used as part of the evaluation process. Firstly, precision:

$\text{Precision}=\frac{\text{True Positives}}{\text{True
Positives}+\text{False Positives}},$ (2)

which is the accuracy of the positive predictions. In this case, precision is
the proportion of correctly detected bees out of all instances detected as
bees that were incorrect. A True Positive denotes a predicted bounding box
that contains a bee, and a False Positive denotes a predicted bounding box
that does not contain a bee.

Recall is also considered:

$\text{Recall}=\frac{\text{True Positives}}{\text{True Positives}+\text{False
Negatives}}.$ (3)

Recall is a measure of the ability of the model to detect all relevant
instances. That is, the proportion of true bee instances that were detected by
the model out of all instances present in the dataset. False Negative in this
sense thus denotes a bee in an image that did not receive a predicted bounding
box.

Given the trade-off between precision and recall, the mean Average Precision
(mAP) is a useful metric in object detection<EMAIL_ADDRESS>is the mAP calculated at
an IoU threshold of 0.5:

$\text{mAP@0.5}=\frac{1}{N}\sum_{i=1}^{N}\text{AP}_{i}|_{\text{IoU}=0.5},$ (4)

with regards to this work, this means that a prediction is considered a True
Positive if $\text{IoU}\geq 0.5$.

Beyond a single threshold, the mAP@0.5:0.95 metric measures mAP over multiple
thresholds. In this case, $\\{0.5,0.55,0.6,...,0.95\\}$:

$\text{mAP<EMAIL_ADDRESS>(5)

The use of multiple thresholds allows for a more rigorous approach, by
iterating by increasing levels of strictness for overlap, or IoU.

In addition to the prominent metrics for validation, Region of Interest (RoI)
is also an important consideration made by the model. A RoI in the example of
bee detection may be a flower patch or even anthers, or the entrance to a
hive. Given automatic recognition of RoIs, further analysis can be performed
within these areas, and computational resources can be saved by focusing on
the parts of the image most likely to contain a bee. Several prominent works
in the related literature show that effective RoI detection can significantly
reduce the likelihood of false positives from the background[36, 37]. For
example, without RoIs, an object detection algorithm trained on a dataset
prominent in images of bees collecting pollen may mistakenly classify all
flowers as containing bees.

##  3 Method

This section describes the methods used in this work. This includes data
collection and preprocessing (with a link to download the data attributed to
this work), augmentation machine learning approaches, and finally a
description of the method of farmer-facing inference for accessibility.

###  3.1 Data Collection and Preprocessing

Table 1: Information derived from the dataset. Metric | Value
---|---
Total Number of Images | 9,664
Total Number of Bees | 13,402
Average Number of Bees per Image | 1.39
Standard Deviation of Bees per Image | 1.39
Maximum Number of Bees in a Single Image | 11
Minimum Number of Bees in a Single Image | 0
Number of Images with No Bees | 1,436

This research project collected an original dataset of digital images named
“Bee Detection in the Wild" for the purposes of this study. The dataset was
collected from various locations in the United Kingdom and annotated by
several of the authors of this study, with 9664 annotated images that feature
a total of 13402 bounding boxes. On average, there are 1.4 bees per image.
Although most of the images (6272) contain one bee, there are a maximum of 11
bees per image. Information derived from the dataset can be found in Table 1.
Out of the total of 9664 images, 8228 contained at least one instance of a
bee. A considerable majority of the images contained one bee, so both the
average and standard deviation of bees per image were around 1.39.

Figure 1: Presence of the number of bees per image within the dataset.

This can be further observed within Figure 1, which shows the distribution of
the number of bees per image within the dataset. Higher counts of bees were
less frequent, with only one image containing 11 bees, 8 containing 10 bees,
and 20 containing 9 bees.

(a) Distribution of Annotation Sizes

(b) Distribution of Aspect Ratios

Figure 2: Annotation Size and Bounding Box Aspect Ratio Distributions within
the Dataset.

Figure 2 shows the distribution of annotation sizes and aspect ratios for the
entire dataset. Annotation sizes are relatively consistent for most of the
dataset, but it should be noted that the dataset does feature a number of
images that contain larger bounding boxes. This suggests that while the size
of bees is relatively uniform, there are instances of closer views of the bee.
The distribution suggests that there is variability in the appearances of bees
within the dataset, which may lead to an improved robustness of the object
detection algorithm. In terms of bounding box aspect ratios, there exists a
much higher diversity within the dataset. This spread suggests that bees are
captured in various poses and angles, which, again, may lead to improved
robustness of the object detection algorithm, since bees should be detected
irrespective of orientation.

During the annotation stage, annotators checked each image individually for
those that were deemed to be low quality (such as excessive motion blur),
removing them from the dataset. Original images are on average 0.92mp (0.18mp
to 2.59mp). The original median image ratio is 1280x720 px. The images in this
study are resized to 416px squares for YOLO compatibility, but the original
full-resolution images and bounding-box annotations are also released to the
research community.

Figure 3: Examples of annotated (yellow bounding boxes) and preprocessed
images within the dataset, selected at random.

Figure 3 shows examples of annotated images selected from the dataset at
random. The dataset contains examples of a range of bee types, such as Bombus
and Apis mellifera; recognition is based on the holistic bee, and future work
could explore the recognition of individual species. The dataset generated for
this study is available for public download111Dataset avaialable from:
https://www.kaggle.com/datasets/birdy654/bee-detection-in-the-wild.

###  3.2 Machine Learning and Data Augmentation

Following data collection and preprocessing, several state-of-the-art object
detection algorithms are then benchmarked on the Bee Detection in the Wild
dataset. This includes YOLOv5, YOLOv5m (variants of frozen and unfrozen
weights), YOLOv5s, and YOLOv8m[38]222Further details on YOLO can be found at:
https://github.com/ultralytics/ultralytics.

Data augmentation for the approaches makes use of the default recommended
parameters by the original authors. Due to differences in recommended
augmentation strategies, it is important to consider data pre-processing time.
Before learning on the full dataset, an initial preliminary exploration is
performed into the feasibility of data augmentation; 2000 images are selected
at random from the dataset and used to train YOLOv5 both with and without
augmentation strategies for 30 epochs to discern whether the strategy could be
useful for this problem. A preliminary test is chosen due to the constraints
of computational resources, but a more in-depth exploration with the full set
could be performed in the future.

Following the selection of the data preprocessing strategy, each model is
trained for 100 epochs. The aforementioned validation metrics of precision,
recall<EMAIL_ADDRESS>mAP@.5:.95, preprocessing time training time, and inference
speed are measured and compared. Considerations are given to both model
ability in terms of object detection, and also the computational resources
required (i.e. comparison of inference time) given that real-world deployment
of this model could require recognition frequency to be performed in real-
time.

###  3.3 Stakeholder-facing Bee Detection and Time Stamping

Figure 4: Graphical overview of the workflow for bee detection and
timestamping.

An overview of the system can be seen in Figure 4 which describes a prototype
system for bee detection and subsequent timestamping to automate the YOLO
inference process. The interface is web-based and accepts video uploads (such
as those collected from an outdoor camera). The system then extracts keyframes
from the video at $FPS/2$, infers via the selected model, and finally distils
the inference results into a report. The report contains a list of timestamps
(time, bees detected) as well as visualisations in the form of charts. The
system is implemented via Flask (a tool for developing Python-based web
applications) and enables accessibility for key stakeholders and
beneficiaries, such as bee farmers, who therefore do not require the technical
ability to load the model and infer data using Python. The interaction system
is open source and is available from GitHub333The source code for the system
produced by this study is available from:
https://github.com/AjayJohnAlex/Bee_Detection.

##  4 Results and Discussion

This section presents the results for training and validation, as well as the
testing of the object detection models on unseen data. Following that, an
example of the user interface for accessibility from the stakeholder’s
perspective is presented, along with pertinent discussions of all the results
arising from the experiments carried out in this work.

###  4.1 Preliminary Exploration

(a) Bounding box loss (train data)

(b) Bounding box loss (validation data)

(c) Objectiveness loss (train data)

(d) Objectiveness loss (validation data)

Figure 5: Metric comparison of bounding box loss (Figures 5(a) and 5(b)) and
objectiveness loss (Figures 5(c) and 5(d)) for a non-augmented and augmented
subset of training and validation data measured over 30 epochs. Note: y-axis
scales are not comparable.

(a) Precision

(b) Recall

(c<EMAIL_ADDRESS>
(d) mAP@0.5:0.95

Figure 6: Metric comparison of 6(a)) precision, 6(b)) recall, 6(c<EMAIL_ADDRESS>and 6(d)) mAP@0.5:0.95, for a non-augmented and augmented subset of training
and validation data measured over 30 epochs. Note: y-axis scales for Figures
6(a) to 6(c) are not comparable to Figure 6(d).

An example of data augmentation and its effect on training using a subset of
data can be seen in Figures 5 and 6. Validation losses for both the bounding
box (Figure 5(b)) and objectiveness (Figure 5(d)) are lower when augmentation
strategies are implemented. After 30 epochs using non-augmented and augmented
validation data, the bounding-box losses were 0.0415 and 0.0377 respectively
(to 3 S.F.). Similarly, for objectiveness loss, non-augmented and augmented
validation data losses were 0.00677 and 0.00497 respectively (to 3 S.F.).
These results demonstrate the effectiveness of data augmentation during
validation.

Interestingly, this effect cannot be seen during training, where the augmented
losses are higher for the same metrics (Figures 5(a) and 5(c) respectively).
After 30 epochs, training on nonaugmented and augmented data lead to a
bounding-box loss of 0.0122 and 0.0393 respectively, with 0.00282 and 0.00695
reported for objectiveness loss (to 3 S.F.).

###  4.2 Training Results

Table 2: YOLOv5 model training summary Metric | YOLOv5s | YOLOv5m | YOLOv5m-frozen | Base YOLOv5
---|---|---|---|---
Precision | 81.4 | 81.0 | 75.5 | 80.0
Recall | 78.6 | 80.8 | 64.8 | 72.1
<EMAIL_ADDRESS>| 80.9 | 81.5 | 69.0 | 76
mAP@.5:.95 | 34.2 | 35.6 | 27.5 | 30.6
Inference Speed (ms) | 6.3 | 7.6 | 8.3 | 6.9
GPU Training Time (min) | 124 | 163 | 128 | 119

A summary of the training results of the YOLOv5 series of models can be found
in Table 2. It was observed that YOLOv5m has a lower precision than YOLOv5s
(81.0 and 81.4, respectively); however, it experiences higher mAP scores for
both related metrics (81.5 and 35.6 compared to 80.9 and 34.2, respectively).
YOLOv5m with frozen weights has significantly lower recall and mAP values at
64.8, 69.0, and 27.5, respectively. This suggests that the dataset diverges
from COCO-related detection activities. Although the Base YOLOv5 model was the
quickest to train, the YOLOv5s has a much lower training inference time of
6.3ms, with a higher speed suggesting that it is more suitable for real-time
application.

Table 3: Indirect training comparison between YOLOv5 variants with YOLOv8 Model Summary | YOLOv5s | YOLOv5m | YOLOv8
---|---|---|---
Precision | 81.4 | 81.0 | 81.8
Recall | 78.6 | 80.8 | 80.4
<EMAIL_ADDRESS>| 80.9 | 81.5 | 83.3
mAP@.5:.95 | 34.2 | 35.6 | 37.8
Inference Speed (ms) | 6.3 | 7.6 | 4.2
GPU Training Time (min) | 124 | 163 | 214

The results in Table 3 show a comparison between the YOLO models v5 and v8.
Since both follow their default, recommended, and different augmentation
strategies, the comparison is indirect. Although YOLOv8 has a lower recall
score than YOLOv5m, all other metrics outperform it; this includes precision,
both mAP scores, and a lower inference speed. It is also worth noting that
YOLOv8 takes a considerably longer time to train, suggesting a larger initial
investment in computational resources before improving real-time detection
ability.

###  4.3 Testing Results

Table 4: Object recognition ability for the models on the testing data. Inference denotes the total average time taken to fully process an input frame. Model | Precision | Recall |<EMAIL_ADDRESS>| mAP@.5:.95 | Inference (ms)
---|---|---|---|---|---
Base YOLOv5 | 81.5 | 76.6 | 81.4 | 38.3 | 4.8
YOLOv5s | 82.6 | 79.7 | 84.6 | 41 | 5.1
YOLOv5m | 83.1 | 81.4 | 85.6 | 42.2 | 8.1
YOLOv5m-frozen | 77.2 | 67.5 | 75.9 | 34.7 | 8.5
YOLOv8m | 81.9 | 80.3 | 83.3 | 37.8 | 12.5

(a)

(b)

(c)

(d)

Figure 7: Examples of ground-truth bees missed by the YOLOv5m model. Left
images show ground truth data with yellow bounding boxes, right image shows a
lack of model predictions.

The results in Table 4 show the test results for each of the models. Given
these data, it was observed that YOLOv5m has the highest precision at 83.1%,
suggesting that it was the best model to identify relevant objects. This
approach also scored the highest recall, so it missed the fewest bees when
making recognition and classification predictions. At both mAP values, YOLOv5m
also scored the highest. Figure 7 shows examples of mistakes made by the
YOLOv5m model. The results show that, even with a precision of 83.1%, the
model can miss both relatively small bees (Figures 7(a) and 7(c)) and
relatively large bees (Figures 7(b) and 7(d)). To this end, examples of
misclassified data suggest that additional data could be collected in the
future to further improve the robustness of the approach.

Table 5: Breakdown of average processing time for the models on the testing data Model | Average processing time (ms)
---|---
Pre-process | Inference | NMS | Total
Base YOLOv5 | 0.1 | 3.4 | 1.3 | 4.8
YOLOv5s | 0.1 | 3.4 | 1.6 | 5.1
YOLOv5m | 0.1 | 6.7 | 1.3 | 8.1
YOLOv5m-frozen | 0.1 | 6.9 | 1.5 | 8.5
YOLOv8m | 0.8 | 10.3 | 1.4 | 12.5

Real-time object detection, of paramount importance when transferring research
from the laboratory to real-world actionable insights, is the ability to
perform inference at suitable speeds as outlined by the problem it is designed
to solve; for example, a model that may contribute to a critical decision
(such as an autonomous vehicle) should have a low latency time to avoid
causing an emergency. In apiculture, a low inference time for bee detection is
important for real-time monitoring and responses to changes in activity or
health, as well as lower computational complexity making the algorithms more
accessible to those who may not have access to high-end computing resources.
Table 5 shows a more extensive set of results for time-based metrics. In these
experiments, the testing dataset of 996 images was used as input for
prediction. The difference in default and recommended augmentation strategies
can be seen in the pre-processing time, where YOLOv8 takes 0.7 seconds
compared to 0.3. As also observed, YOLOv8m has the highest inference time,
with the lowest being the Yolov5 and YOLOv5s models. Non-Maximum Suppression
(NMS) was relatively similar across all models, with the lowest being 1.3ms
(V5 and V5m) and the highest was 1.6ms (V5s). Overall, the quickest model was
Base YOLOv5 at 4.8ms, closely followed by V5s at 5.1ms. The highest performing
model, YOLOv5m, took longer at 8.1ms per frame.

###  4.4 Stakeholder-facing Interface

As suggested during the literature review, the use of video streams is showing
promise for autonomous monitoring of pollinator behaviour. They are non-
intrusive and can be used to capture critical data such as hive health and
population decline. Given that Python code and machine learning models are
often presented in technical formats, they are therefore inaccessible to a
wider audience. Following the training and validation of the models, this
article proposes an encapsulation of the work in a format that is accessible
to the stakeholder.

Figure 8: Impact of keyframe selection on API Response

Figure 8 shows the impact of keyframe selection on API response time. As
expected, when frames per second are reduced by half, that is, $FPS/2$
keyframes from video streams, the processing time is reduced. Videos of 5, 8,
13, and 20 seconds could be processed in 9, 19, 26, and 38 seconds,
respectively. In the future, keyframe selection strategies could be
benchmarked according to the relevant literature, which is discussed in
Section 5.

Figure 9: A screenshot of the interface for the non-technical interaction with
the inference model. For demonstration purposes, random bee facts are printed
to the screen.

An example of the user interface can be found in Figure 9. The system
encapsulates all model inference and replaces the need to either write Python
code or use the command-line interface by enabling the upload of a video file.
This video is then automatically processed by the object detection model and a
report is generated along with images of the detected bees for viewing and
further analysis. For demonstration purposes, random bee facts are printed
onto the screen, but this can be replaced or removed entirely in the future.

Table 6: An excerpt of an example report showing the event of a bee entering the camera view and being detected by the object detection model.  D | H | M | S |  | Video
---
Time
Time (formatted) | DD_HH_MM_SS | Detected
0 | 0 | 15 | 59 | 959 | 0 days 0 hours 15 mins 59 secs | 00:00:15:59 | 0
0 | 0 | 16 | 0 | 960 | 0 days 0 hours 16 mins 0 secs | 00:00:16:00 | 0
0 | 0 | 16 | 1 | 961 | 0 days 0 hours 16 mins 1 secs | 00:00:16:01 | 0
0 | 0 | 16 | 2 | 962 | 0 days 0 hours 16 mins 2 secs | 00:00:16:02 | 0
0 | 0 | 16 | 3 | 963 | 0 days 0 hours 16 mins 3 secs | 00:00:16:03 | 0
0 | 0 | 16 | 4 | 964 | 0 days 0 hours 16 mins 4 secs | 00:00:16:04 | 0
0 | 0 | 16 | 5 | 965 | 0 days 0 hours 16 mins 5 secs | 00:00:16:05 | 0
0 | 0 | 16 | 6 | 966 | 0 days 0 hours 16 mins 6 secs | 00:00:16:06 | 1
0 | 0 | 16 | 7 | 967 | 0 days 0 hours 16 mins 7 secs | 00:00:16:07 | 1
0 | 0 | 16 | 8 | 968 | 0 days 0 hours 16 mins 8 secs | 00:00:16:08 | 1
0 | 0 | 16 | 9 | 969 | 0 days 0 hours 16 mins 9 secs | 00:00:16:09 | 1
0 | 0 | 16 | 10 | 970 | 0 days 0 hours 16 mins 10 secs | 00:00:16:10 | 1
0 | 0 | 16 | 11 | 971 | 0 days 0 hours 16 mins 11 secs | 00:00:16:11 | 1

Following the submission and processing of a video stream, an example of the
aforementioned report can be found in Table 6. These rows are exemplars
extracted from the CSV file generated, and show how at 16:06, the model
predicted that a bee had entered the frame and thus received a bounding box.
It is important to note that this report was generated solely by following
three steps: 1) clicking the “Choose File" button, 2) choosing a video file,
and 3) clicking the “Detect bees" button. No interaction with code or the
command line interface is required, with the aim of democratising the
technology arising from the scientific contributions of this work.

##  5 Conclusion and Future Work

This study has contributed a significantly large dataset for the research
community and explored the application of object detection algorithms for bee
detection and tracking. The use of these data and/or these algorithms are part
of an endeavour to provide a technological intervention in conservation
management and analysis. It is particularly pertinent to explore these
interventions for pollinators given that food security and the overall
survival of the human species is dependent on their services. The
democratisation of research is also supported by the approaches proposed by
this work, given the open-source nature of all the data, models and software
produced. The findings showed that two particular models benchmarked in this
work are promising, the YOLOv5m model which achieved 85.6&<EMAIL_ADDRESS>with an
inference time of 8.1ms per frame, and the more efficient YOLOv5s at 5.1ms
inference at a slightly lower 84.6% mAP.

Beyond data collection and machine learning implementation, this work also
argued in favour of stakeholder-friendly and accessible application, and
contributed to this in the form of a web application for automated analysis
using the selected model. This meant that operators did not need to use any
computer code to analyse video streams, such as those collected from an
apiary.

A limitation of this work is the preliminary nature of the selection of the
data augmentation strategy, which was restricted to a data subset due to the
availability of computational resources. In the future, multiple augmentation
strategies could be explored using the full dataset to discern a more general
view of which is most effective to use. This, along with combinatorial
optimisation of the augmentation parameters, could lead to a better overall
model. In terms of real-time processing, a relatively simple keyframe
selection strategy of $FPS/2$ was implemented; however, this process could be
improved by making use of a more in-depth keyframe selection strategy
according to the literature[39, 40, 41].

Figure 10: Proposed future pipeline for improved use in industry.

In the future, the application of these models in real-world situations, such
as an apiary, could provide valuable insight into behavioural analysis.
Furthermore, multimodality could improve detection techniques, such as the
fusion of audio collected alongside images. Future work could also explore
multiple classes, such as the detection of individual species, or behaviours,
such as the automatic recognition of pollination events. If used in
conjunction with a flower recognition algorithm, this algorithm could then
enhance the reporting system towards more detailed information about events
that have taken place. Toward the next iteration of this work, Figure 10 shows
an example of how the pipeline could be improved. This includes sending
notifications to stakeholders and beneficiaries (such as an alert to a
population decline) along with the transition away from local to cloud
storage. Additionally, the user interface printed random bee facts for
demonstration purposes; improvements could be made to the interface in future,
such as replacing this block with a live feed of the detection process, that
is, by showing images of some of the bees detected and live graphs that update
as the inference process is performed.

Toward integration reflective of Agriculture 4.0, future implementations of
this framework could consider wireless communication, such as inference on the
edge and wireless communication (5G, 6G, LoraWan etc.). Additionally,
enhancements through Geographic Information System (GIS) and Global
Positioning System (GPS) data could enable the provision of further context
through geographic and location metadata. Furthermore, given long-term data
collection and modelling, findings could be used to infer design decisions
when building digital twins of the relevant ecosystems.

To finally conclude, this work has proposed the use of a novel dataset and
object detection algorithms to facilitate technological intervention in
conservation management, with a focus on pollinator behaviour. Food security,
reacting to climate change, the conservation of the natural world, and
responsible consumption and production are critical issues that support human
survival, and the use of technology to monitor their improvement is possible
given the transfer of interdisciplinary knowledge from the laboratory to the
real world.

##  6 Data Availability Statement

All data collected and subsequent code written is made publicly available for
future work.

The Bee Detection in the Wild, collected and analysed in this study, is
released via the Kaggle data science platform under the MIT license. It can be
downloaded from: https://www.kaggle.com/datasets/birdy654/bee-detection-in-
the-wild

The code for the web interface used to encapsulate the models is available on
Github. It can be downloaded from:
https://github.com/AjayJohnAlex/Bee_Detection

## References

  * [1] Stéphane Kluser, Pascal Peduzzi, et al.  Global pollinator decline: a literature review.  Geneva: UNEP/GRID, 2007.
  * [2] Shaden AM Khalifa, Esraa H Elshafiey, Aya A Shetaia, Aida A Abd El-Wahed, Ahmed F Algethami, Syed G Musharraf, Mohamed F AlAjmi, Chao Zhao, Saad HD Masry, Mohamed M Abdel-Daim, et al.  Overview of bee pollination and its economic value for crop production.  Insects, 12(8):688, 2021.
  * [3] Tom D Breeze, Alison P Bailey, Kelvin G Balcombe, and Simon G Potts.  Pollination services in the uk: How important are honeybees?  Agriculture, Ecosystems & Environment, 142(3-4):137–143, 2011.
  * [4] Yiannis Ampatzidis, Victor Partel, and Lucas Costa.  Agroview: Cloud-based application to process, analyze and visualize uav-collected data for precision agriculture applications utilizing artificial intelligence.  Computers and Electronics in Agriculture, 174:105457, 2020.
  * [5] Jun Liu, Lei Shu, Xu Lu, and Ye Liu.  Survey of intelligent agricultural iot based on 5g.  Electronics, 12(10):2336, 2023.
  * [6] Spyros Fountas, Borja Espejo-García, Aikaterini Kasimati, Marilena Gemtou, Hercules Panoutsopoulos, and Evangelos Anastasiou.  Agriculture 5.0: Cutting-edge technologies, trends, and challenges.  IT Professional, 26(1):40–47, 2024.
  * [7] Rabiya Abbasi, Pablo Martinez, and Rafiq Ahmad.  The digitization of agricultural industry–a systematic literature review on agriculture 4.0.  Smart Agricultural Technology, 2:100042, 2022.
  * [8] Pushkar Bhuse, Bhavesh Singh, and Purva Raut.  Effect of data augmentation on the accuracy of convolutional neural networks.  In Information and Communication Technology for Competitive Strategies (ICTCS 2020) ICT: Applications and Social Interfaces, pages 337–348. Springer, 2022.
  * [9] Iza Sazanita Isa, Mohamed Syazwan Asyraf Rosli, Umi Kalsom Yusof, Mohd Ikmal Fitri Maruzuki, and Siti Noraini Sulaiman.  Optimizing the hyperparameter tuning of yolov5 for underwater detection.  IEEE Access, 10:52818–52831, 2022.
  * [10] Thi Nha Ngo, Kung-Chin Wu, En-Cheng Yang, and Ta-Te Lin.  A real-time imaging system for multiple honey bee tracking and activity monitoring.  Computers and Electronics in Agriculture, 163:104841, 2019.
  * [11] Jacobus C Biesmeijer, Stuart PM Roberts, Menno Reemer, R Ohlemuller, Mike Edwards, Tom Peeters, AP Schaffers, Simon G Potts, RJMC Kleukers, CD Thomas, et al.  Parallel declines in pollinators and insect-pollinated plants in britain and the netherlands.  Science, 313(5785):351–354, 2006.
  * [12] Marcelo A Aizen, Lucas A Garibaldi, Saul A Cunningham, and Alexandra M Klein.  Long-term global trends in crop yield and production reveal no current pollination shortage but increasing pollinator dependency.  Current biology, 18(20):1572–1575, 2008.
  * [13] Alexandra-Maria Klein, Bernard E Vaissière, James H Cane, Ingolf Steffan-Dewenter, Saul A Cunningham, Claire Kremen, and Teja Tscharntke.  Importance of pollinators in changing landscapes for world crops.  Proceedings of the royal society B: biological sciences, 274(1608):303–313, 2007.
  * [14] Grigoriy Komlatskiy and Tatyana Makarova.  Pollination by bees in industrial crop production.  In BIO Web of Conferences, volume 66, page 12001. EDP Sciences, 2023.
  * [15] Mary A Gardiner, Julianna K Tuell, Rufus Isaacs, Jason Gibbs, John S Ascher, and Douglas A Landis.  Implications of three biofuel crops for beneficial arthropods in agricultural landscapes.  BioEnergy Research, 3:6–19, 2010.
  * [16] María José Romero and José Javier G Quezada-Euán.  Pollinators in biofuel agricultural systems: the diversity and performance of bees (hymenoptera: Apoidea) on jatropha curcas in mexico.  Apidologie, 44:419–429, 2013.
  * [17] Claire Kremen, Neal M Williams, Marcelo A Aizen, Barbara Gemmill-Herren, Gretchen LeBuhn, Robert Minckley, Laurence Packer, Simon G Potts, T’ai Roulston, Ingolf Steffan-Dewenter, et al.  Pollination and other ecosystem services produced by mobile organisms: a conceptual framework for the effects of land-use change.  Ecology letters, 10(4):299–314, 2007.
  * [18] Francisco Sánchez-Bayo, Dave Goulson, Francesco Pennacchio, Francesco Nazzi, Koichi Goka, and Nicolas Desneux.  Are bee diseases linked to pesticides?—a brief review.  Environment international, 89:7–11, 2016.
  * [19] Carl A Johansen, Daniel F Mayer, Jack D Eves, and Christopher W Kious.  Pesticides and bees.  Environmental Entomology, 12(5):1513–1518, 1983.
  * [20] Vrushali Vijaykumar Pashte and Chidanand Shivshankar Patil.  Toxicity and poisoning symptoms of selected insecticides to honey bees (apis mellifera mellifera l.).  Archives of Biological Sciences, 70(1):5–12, 2018.
  * [21] Simon G Potts, Hien T Ngo, Jacobus C Biesmeijer, Thomas D Breeze, Lynn V Dicks, Lucas A Garibaldi, Rosemary Hill, Josef Settele, and Adam Vanbergen.  The assessment report of the intergovernmental science-policy platform on biodiversity and ecosystem services on pollinators, pollination and food production.  2016\.
  * [22] Anuj Kumar, Hiesik Kim, and Gerhard P Hancke.  Environmental monitoring systems: A review.  IEEE Sensors Journal, 13(4):1329–1339, 2012.
  * [23] Malika Nisal Ratnayake, Adrian G Dyer, and Alan Dorin.  Towards computer vision and deep learning facilitated pollination monitoring for agriculture.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2921–2930, 2021.
  * [24] Thomas Stark, Valentin Ştefan, Michael Wurm, Robin Spanier, Hannes Taubenböck, and Tiffany M Knight.  Yolo object detection models can locate and classify broad groups of flower-visiting arthropods in images.  Scientific Reports, 13(1):16364, 2023.
  * [25] Michael-Thomas Ramsey, Martin Bencsik, Michael Ian Newton, Maritza Reyes, Maryline Pioz, Didier Crauser, Noa Simon Delso, and Yves Le Conte.  The prediction of swarming in honeybee colonies using vibrational spectra.  Scientific reports, 10(1):9798, 2020.
  * [26] Baptiste Magnier, Gaëtan Ekszterowicz, Joseph Laurent, Matthias Rival, and François Pfister.  Bee hive traffic monitoring by tracking bee flight paths.  In 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, January 27-29, 2018, in Funchal, Madeira, Portugal, pages 563–571, 2018.
  * [27] Malika Nisal Ratnayake, Adrian G Dyer, and Alan Dorin.  Tracking individual honeybees among wildflower clusters with computer vision-facilitated pollinator monitoring.  Plos one, 16(2):e0239504, 2021.
  * [28] Dominik Bittner, Joao Filipe Ferreira, M Eduarda Andrada, Jordan J Bird, and David Portugal.  Generating synthetic multispectral images for semantic segmentation in forestry applications.  In ICRA 2022 Workshop in Innovation in Forestry Robotics: Research and Industry Adoption, 2022.
  * [29] Manjit Kaur, Iman Ardekani, Hamid Sharifzadeh, and Soheil Varastehpour.  A cnn-based identification of honeybees’ infection using augmentation.  In 2022 International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME), pages 1–6. IEEE, 2022.
  * [30] Dario De Nart, Cecilia Costa, Gennaro Di Prisco, and Emanuele Carpana.  Image recognition using convolutional neural networks for classification of honey bee subspecies.  Apidologie, 53(1):5, 2022.
  * [31] Keanu Buschbacher, Dirk Ahrens, Marianne Espeland, and Volker Steinhage.  Image-based species identification of wild bees using convolutional neural networks.  Ecological Informatics, 55:101017, 2020.
  * [32] Yali Amit, Pedro Felzenszwalb, and Ross Girshick.  Object detection.  In Computer Vision: A Reference Guide, pages 875–883. Springer, 2021.
  * [33] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [34] Peiyuan Jiang, Daji Ergu, Fangyao Liu, Ying Cai, and Bo Ma.  A review of yolo algorithm developments.  Procedia Computer Science, 199:1066–1073, 2022.
  * [35] Zhaohui Zheng, Ping Wang, Wei Liu, Jinze Li, Rongguang Ye, and Dongwei Ren.  Distance-iou loss: Faster and better learning for bounding box regression.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 12993–13000, 2020.
  * [36] Chenchao Xiang, Zhou Yu, Suguo Zhu, Jun Yu, and Xiaokang Yang.  End-to-end visual grounding via region proposal networks and bilinear pooling.  IET Computer Vision, 13(2):131–138, 2019.
  * [37] Daniel Cores, Manuel Mucientes, and Víctor M Brea.  Roi feature propagation for video object detection.  In ECAI 2020, pages 2680–2687. IOS Press, 2020.
  * [38] Ultralytics.  YOLOv5: A state-of-the-art real-time object detection system.  https://docs.ultralytics.com, 2021.  Accessed: Feb 2024.
  * [39] BS Rashmi and HS Nagendraswamy.  Effective video shot boundary detection and keyframe selection using soft computing techniques.  International Journal of Computer Vision and Image Processing (IJCVIP), 8(2):27–48, 2018.
  * [40] Cheng Huang and Hongmei Wang.  A novel key-frames selection framework for comprehensive video summarization.  IEEE Transactions on Circuits and Systems for Video Technology, 30(2):577–589, 2019.
  * [41] Rukiye Savran Kızıltepe, John Q Gan, and Juan José Escobar.  A novel keyframe extraction method for video classification using deep neural networks.  Neural Computing and Applications, 35(34):24513–24524, 2023.