Grape detection, segmentation and tracking using deep neural networks and three-dimensional association
Thiago T. Santos Leonardo L. de Souza Andreza A. dos Santos Sandra Avila
cs.CV





1]Thiago T. Santoscor1
thiago.santos@embrapa.br

2]Leonardo L. de Souza
leonardolimasza@gmail.com

2]Andreza A. dos Santos
andi.apsantos@gmail.com

2]Sandra Avila
sandra@ic.unicamp.br

[cor1]Corresponding author
  
[1]
  Embrapa Agricultural Informatics,
  Av. André Tosello 209,
  Campinas SP, 13083-886, Brazil


[2]
  Institute of Computing, University of Campinas,
  Av. Albert Einstein 1251,
  Campinas SP, 13083-852, Brazil




Agricultural applications such as yield prediction, precision agriculture
and automated harvesting need systems able to infer the crop state
from low-cost sensing devices. Proximal sensing using affordable
cameras combined with computer vision has seen a promising
alternative, strengthened after the advent of convolutional neural
networks (CNNs) as an alternative for challenging pattern recognition
problems in natural images. Considering fruit growing monitoring
and automation, a fundamental problem is the detection, segmentation
and counting of individual fruits in orchards. Here we show that for
wine grapes, a crop presenting large variability in shape, color, size and
compactness, grape clusters can be successfully detected, segmented and
tracked using state-of-the-art CNNs. In a test set containing 408 grape
clusters from images taken on a trellis-system based vineyard, we have
reached an F_1-score up to 0.91 for instance segmentation, a fine
separation of each cluster from other structures in the image  that
allows a more accurate assessment of fruit size and shape. We have
also shown as clusters can be identified and tracked along video sequences recording orchard
rows. We also present a public dataset containing grape clusters
properly annotated in 300 images and a novel annotation methodology for 
segmentation of complex objects in
 natural images. The presented pipeline  for annotation,
training, evaluation and tracking of agricultural patterns in images
can be replicated for different crops and production systems. It
can be employed in the development of sensing components for several
agricultural and environmental applications. 


  fruit detection yield prediction computer vision deep learning



fancy






§ INTRODUCTION


Automation in agriculture is particularly hard when compared to
industrial automation due to field conditions and the uncertainty
regarding plant structure and outdoor environment. That creates a need   
for systems able to monitor structures as plants and fruits in a
fine-grained level <cit.>. Proper detection and
localization for such structures are critical components for monitoring, 
robotics and autonomous systems for agriculture <cit.>. 

Accurate fruit detection and localization are essential
for several applications. Fruit counting and yield estimation are the
more immediate ones. Precision agriculture applications, accounting
for management of inter and intra-field variability, can be derived if
detection data is properly localized in space. Fruit detection can also be a
preliminary step for disease and nutrient deficiency monitoring
<cit.> and a crucial component on actuation, for example, automated
spraying and harvesting may be an important application considering the
declining in agricultural labor force <cit.>. Beyond farms,
fruit detection can be employed in field phenotyping, aiding plant
research and breeding programs <cit.>.

Off-the-shelf RGB cameras and computer vision can provide
affordable and versatile solutions for fruit
detection. State-of-the-art computer vision systems based on deep
convolutional neural networks <cit.> can deal with
variations in pose, shape, illumination and large inter-class variability
<cit.>, essential features needed for robust
recognition of complex objects in outdoor environments. Recent researches
<cit.> have shown that the Faster R-CNN
(region-based convolutional neural network) architecture
<cit.> can produce accurate 
results for a large set of fruits, including peppers, melons, oranges,
apples, mangoes, avocados, strawberries and almonds. Detection results
can be integrated by data association approaches, by employing object
tracking or mapping, to perform fruit counting for rows in the crop
field <cit.>.

These previous detection systems identify individual objects by rectangular
bounding boxes, as seen in Figure <ref>. Such
boxes, if well fitted to the fruits boundaries, could
provide estimations of fruit shape and space occupancy for fruits
presenting a regular shape as oranges and apples (circular
shape). However, for grape clusters, rectangular boxes would not
properly adjust to the berries. A step further beyond object detection
is instance segmentation <cit.>: the fruit/non-fruit pixel
classification combined with instance assignment
(Figure <ref>). Instance segmentation 
can properly identify berries pixels in the detection box,
providing finer fruit characterization. Also, occlusions by leaves,
branches, trunks and even other clusters can be properly addressed by
instance segmentation, aiding on robotic manipulation and other
automation tasks.

To approach fruit instance segmentation as a supervised machine
learning problem, we need datasets that capture the variations
observed in the field. Wine grapes present large variations in shape,
size, color and structure, even for the same grape variety,
contrasting to citrus and apples. Also, the dataset have to provide
masks for the individual clusters, isolating grapes from
background pixels and from occluding objects. We also need a neural
network architecture able to simultaneously perform object detection
and pixel classification. Thus, the present work introduces the
following contributions:  


  
  * a new methodology for image annotation that employs interactive
    image segmentation <cit.> to generate object masks,
    identifying background and occluding foreground pixels;
  
  * a new public dataset[Available at
      <doi:10.5281/zenodo.3361736> <cit.>.]
    for grape detection and instance segmentation, 
    comprising images, bounding boxes and masks – this dataset is
    composed by images of five different grape varieties taken on
    field (Figure <ref>); 
  
  * an evaluation of two deep learning detection architectures for
    grape detection:  Mask R-CNN <cit.>, a
    convolutional framework for instance segmentation that is simple
    to train and generalizes well <cit.>, and YOLO
  <cit.>, a single-stage network 
    that can detect objects without a previous region-proposal stage 
    <cit.> – such evaluation allows a comparison between
    instance segmentation and box-based object detection approaches;
  
  * a fruit counting methodology that employs three-dimensional
    association to integrate and localize the detection results in
    space, avoiding multiple counting, addressing occlusions and
    accumulating evidence from different images to confirm detections.  






§ RELATED WORK



As seen in computer vision applications on other fields, classic machine learning
and pattern recognition have been replaced by modern deep learning techniques, which can
address the enormous variability in object appearance, as shortly described
in the following sections.



 §.§ Earlier works: feature engineering and machine learning


Earlier works in fruit detection employed the classic feature
  engineering approach: human-designed descriptors based on color,
geometric and texture features. Using such features, machine learning
techniques such as Bayesian classifiers, support vector machines and
clustering were applied to perform fruit detection and
classification. <cit.> presented an
extensive review of the works employing this approach. <cit.> presented one of the
earliest works to employ image processing for grape detection. They used color
thresholding to detect “grape pixels” in images showing mature Cabernet Sauvignon clusters in vines. A white screen was placed
behind the canopy to create a regular background.

<cit.> presented a computer vision methodology
intended for realistic field operation without background
control. Their multi-stage method employed Loy and Zelinsky's Radial
Symmetry Transform <cit.> to find berry candidates, further filtered
by a K-nearest neighbors classifier using color and texture
features. In the last step, neighboring berries were grouped in
clusters, eliminating isolated berries (likely
false-positives). For a set of 2,973 berries, their system reached 
63.7% for recall and 98.0% for precision overall (the set included
berries of three grape varieties). The authors performed linear regression
for the berries count found for individual vines, finding a 0.74 correlation score
for crop weight.

In a further work, <cit.> added a data
association component based on visual odometry <cit.>
to avoid double-counting and to estimate the spatial distribution of
yield. They proposed a new berry detector for a particular flash-based
setting for night imaging developed by them and evaluated other
image features for berry classification: SIFT <cit.> and
FREAK <cit.>. <cit.> stated that segmentation of berries
clusters (grape clusters) is challenging because of occlusion and touching clusters; after some experiments with 3-D modeling, the authors
chose to perform yield estimation using berry counting. 
They performed controlled imaging and reported variations in results
possibly caused by illumination and imaging differences.



 §.§ Deep learning-based works


Earlier works present issues that foreshadow the advantages and
power of convolutional neural networks (CNNs). These networks learn
effective representations for a given machine learning task, replacing
feature engineering <cit.>. Systematically, deep learning
approaches are being 
adopted in fields presenting image-based perceptual problems, and
agricultural applications are no exception <cit.>.   

CNN's invariance to local translation give vision systems
robustness in situations where a feature's presence is 
more important than its exact location
<cit.>. As an example, 
<cit.> reported that variations in the berry candidate
location by detection affected their berry 
classification. CNNs are also able to encode variance regarding pose,
color and illumination, if the training data presents sufficient
examples of such variation, which relieves the need for controlled
imaging, illumination and camera settings. The first attempts employed 
CNNs to perform pixel classification, followed by additional steps to segment
individual fruits <cit.>. Further, these earlier
approaches were replaced by end-to-end object detection
<cit.>  based on the popular Faster R-CNN
architecture <cit.>.

<cit.> employed transfer learning, using a
VGG16 network <cit.> pre-trained using  ImageNet <cit.> 
(VGG16 is the perceptual backbone in the Faster R-CNN architecture). They
reached F_1-scores up to 0.83  in tests on sweet pepper and rock melon,
using a dataset of images captured in a greenhouse, and presented similar
performance for smaller datasets of strawberry, apple, avocado, mango
and orange images retrieved from Google Images Search. The authors also
fused RGB and Near Infrared (NIR) data in four-channel arrays, showing that the CNN
paradigm can easily benefit from multi-spectral imaging.

<cit.> also employed the Faster
R-CNN architecture for fruit detection. They produced datasets from
images captured in orchards by a robotic ground vehicle for apples and
mangoes, and a dataset for almonds, also in orchards, but using a
hand-held DSLR camera (digital single-lens reflex). Employing image augmentation
strategies on training, the authors reached F_1-scores up to 0.90 for mangoes and apples 
and 0.77 for almonds. A surprising result reported by the authors is that
transfer learning between farms (same crop) or between orchards of
different crops showed little advantage compared to ImageNet transfer
learning. Bargoti and Underwood state such result increase the body
of evidence showing ImageNet features applicability for a broad range
of tasks.

In Faster R-CNN, detection is performed in two stages. The first stage
uses a region proposal network, an attention mechanism
developed as an alternative to the earlier sliding window based
approaches. In the second stage, bounding box regression and object
classification are performed. Faster R-CNN is fairly recognized as a
successful architecture for object detection, but it is not the only
meta-architecture <cit.> able to reach
state-of-the-art results. Another group of architectures is the
single shot detector (SSD) meta-architecture <cit.>, single feed-forward convolutional networks able to 
predict classes and bounding boxes in a single stage. The YOLO
(You Only Look Once) networks, proposed by Redmon and Farhadi
<cit.>, are examples of the SSD family. 

Grape clusters present larger variability on size, shape and 
compactness compared to other fruits like peppers, apples or mangoes 
<cit.>. A focus on berry detection, such as in
<cit.>, can be seen as a way to
circumvent grape cluster variability, performing yield prediction
over berry counting, consequently bypassing the grape cluster segmentation
problem. CNNs can learn representations of complex 
visual patterns <cit.>, so are an interesting
alternative for grape cluster detection. However, object detection using
bounding boxes could be insufficient for yield prediction
applications, considering the enormous variability in grape clusters'
shapes and compactness. On the other hand, semantic segmentation (the
classification of pixels as fruit or background) could also be
inadequate, considering the severe occlusion between fruits observed in
orchards <cit.>. Instance segmentation
(Figure <ref>), the combined task of object
detection (where are the grape clusters?) and pixel classification
(this pixel belongs to which cluster?), is an alternative machine learning task
formulation for yield prediction and automated harvesting applications.

Mask R-CNN <cit.> is a derivation of Faster R-CNN
able to perform instance segmentation, jointly
optimizing region proposal, bounding box regression and semantic pixel
segmentation. However, differently of object detection in which
rectangular bounding boxes annotations are sufficient for training,
instance segmentation needs image pixels to be properly attributed to
an instance or to the background in the training dataset for
supervised learning. In Section <ref>, we describe a methodology for fruit
instance segmentation based on Mask R-CNN, including a
novel instance annotation tool for objects of complex shape. We
compare YOLO and Mask R-CNN results on wine grape cluster detection,
and we evaluate Mask R-CNN results on cluster instance segmentation.  

Fruit detection in single images can be the perceptual step in a fruit
counting system, but without some sort of integration of the information
produced for the orchard, accurate prediction of yield is not
possible. <cit.>, extending the work in
<cit.>, integrated the fruit
detection results in image sequences (video frames) performing
object tracking. Employing the bounding box centers as
observations, the authors implemented an object tracker based on the
Kanade-Lucas-Tomasi algorithm (optical flow), Kalman filters and the
Hungarian Assignment algorithm, tracking fruits in video frame
sequences. To address issues caused by missing detections and
occlusions, they performed structure-from-motion,
recovering three-dimensional information using the box centers and their
inter-frame correspondence. Associating fruit locations in 3-D and the
CNN detection in 2-D frames, <cit.> integrated data from a
camera moving along a mango orchard row, avoiding double-counting from
the same fruit observed in different frames, addressing occlusions and
localizing yield information in space.  Similarly, we propose a simple but effective spatial
  registration step for fruit tracking and counting, also employing
3-D association from structure-from-motion data.      




§ MATERIALS AND METHODS



The proposed methodology introduces a new public dataset for image-based 
grape detection, including a novel method for interactive mask annotation for 
instance segmentation (Section <ref>). Three neural networks are
trained and evaluated for fruit detection: Mask R-CNN <cit.>, 
YOLOv2 <cit.> and YOLOv3 <cit.> (Section <ref>). 
Evaluation measures for semantic segmentation, object detection, and instance 
segmentation variants are presented in Section <ref>. Section <ref> 
presents our approach for spatial integration.



 §.§ The dataset



The Embrapa Wine Grape Instance Segmentation Dataset (WGISD) is
composed by 300 RGB images showing 4,432
grape clusters from five different grape varieties, as summarized in
Table <ref>. All images were captured from a single
winery, that employs dual pruning: one for shaping (after
previous year harvest) and one for production, resulting in canopies
of lower density. No pruning, defoliation or any intervention in the
plants was performed specifically for the dataset construction: the
images capture a real, trellis system-based wine grape production. The
camera captures the images in a frontal pose, that means the camera
principal axis is approximately perpendicular to the wires of the
trellis system and the plants rows. As seen in
Table <ref>, the Syrah images were taken in a
different field visit, one year earlier, and consist a smaller set if
compared to the other four varieties. <ref> presents a
detailed description for the dataset, following the guidelines
proposed by <cit.> for dataset characterization, and
including information about cameras, field location, pre-processing
and file formats. The WGISD is publicly available <cit.>
under the CC BY-NC 4.0 (Attribution-NonCommercial 4.0 International)
license. 



To be employed on supervised instance segmentation training, WGISD
has to provide a set of masks that properly segment grape
clusters. WGISD provides binary masks for 2,020 clusters from the
4,432 total, as seen in Table <ref>. Mask
annotation for instance segmentation 
is a laborious task that requires custom tools to allow the annotation
of hundreds of images in 
limited time. The VGG Image Annotator (VIA) <cit.> is a
popular tool used by the computer vision community. It allows users to
mark objects of interest using rectangles, circles, ellipses or
polygons. In an interesting attempt to automatize annotation,
<cit.> proposed an interactive 
tool that uses a neural network (Polygon-RNN++) to predict the next
vertex in polygonal annotations.

In WGISD construction, the accurate annotation of complex objects
in natural scenes using polygonal shapes proved to be extremely
laborious, even when employing the vertex prediction facilities from
Polygon-RNN++. To relieve the annotation process, we have created an
annotation tool based on interactive image segmentation by graph
matching, as proposed by <cit.>. This
method starts from an over-segmentation, produced by the watershed
algorithm <cit.>, to create an
attributed relational graph (ARG) representation for the image –
G_i. Then, the user can freely mark the image using
scribbles. Such marks are used to create a model graph
G_m, a labeled ARG. Exploiting the spatial relations among ARGs
vertices, a match is computed between the model graph G_m and the
input image graph G_i, allowing the propagation of labels from G_m
to G_i.

Figure <ref> shows an example of grape annotation. The
dataset was previously annotated for object detection using standard
rectangular bounding boxes (see <ref> for
details). The instance annotation tool uses the bounding boxes as
inputs, displaying each grape cluster for an interactive image
segmentation procedure by graph matching (Figure <ref> (a)). An 
annotator can draw scribbles, freely
marking pixels that should be considered part of the grape cluster and
pixels that are part of the background or occluding foreground
objects (Figure <ref> (b)). The graph matching-based
algorithm uses the scribbles to produce a segmentation,
propagating the labels from the model to the input image
(Figure <ref> (c)). The tool allows the scribble marking and graph matching
steps to be repeated by the user until a reasonable annotation is
achieved. Finally, the grape pixels are stored as masks for supervised
instance segmentation learning. Readers interested in a detailed
description of the graph matching algorithm should refer to
<cit.>.





 §.§ The perceptual step: CNN architectures



Mask R-CNN <cit.> is a consolidation of a long sequence
of works developed by He, Dollár, Girshick and colleagues. This
network is essentially the combination of a Faster R-CNN object detector
<cit.> and a fully convolutional network (FCN)
<cit.> for semantic segmentation, providing a complete,
end-to-end, instance segmentation solution. The Faster R-CNN is also a
combination of two architectures: a region proposal network
(RPN) and an object detector, the Fast R-CNN <cit.>. RPN
works as an attention mechanism, finding anchors in the 
feature space, rectangular boxes that can contain objects of
interest (Figure <ref>). The Fast R-CNN is composed of a
softmax object classifier and a per-class bounding box regressor
(Figure <ref>). The Mask R-CNN employs as feature extractor
a feature pyramid network (FPN) <cit.>, an
architecture able to create semantic feature maps for objects at
multiple scales, built over a ResNet <cit.>. 

Another approach to object detection is to predict the locations and 
the objects' class in a single step, in order to avoid a previous region proposal procedure. 
<cit.> refer to this approach as single shot detector meta-architecture, and the YOLO networks proposed by 
<cit.> are prominent members of this family.
In the YOLO networks, the image is split into a fixed grid of S × S cells. 
A cell is responsible for performing a detection if an object center is over it. Each cell
is associated to B boxes, composed by 5 values representing the object center (c_x, c_y),
the object width and height and a confidence score that represents the model
confidence that the box contains an object and also the accuracy of the box boundaries regarding
the object. The box also includes C conditional class probabilities, one to each
class of objects. Consider, for example, a 7 × 7 grid of cells (S = 7), where
each cell predicts B = 2 boxes for 20 different classes of object (C = 20). The YOLO
network will produce a 7 × 7 × 30 output tensor. This means a B · 5 + C vector
for each one of the 49 cells. The training step tries to minimize a loss function defined over
such a tensor, performing detection and classification in a single
step. The YOLOv2 and YOLOv3 networks have a few differences, mainly
regarding their feature extraction convolution part. YOLOv3 
presents a deeper convolutional network that incorporate some
state-of-the-art techniques such as residual networks <cit.>,
skip connections and multi-scaling (similar to FPNs). YOLOv3
classification is based in multi-label classification instead of the
softmax employed by YOLOv2, allowing the former able to deal with
multi-class problems. 







 §.§ Training



Table <ref> shows the splitting between
training/validation and test sets. The images were assigned randomly
to each set in an 80-20% proportion. It is important to note that
Cabertnet Franc and Sauvignon Blanc varieties presented a
higher number of clusters per image. In Section <ref>, we
will show that although the differences in the numbers of images and
clusters, the results are very similar for all five grape varieties.
For instance segmentation, a set of 110 images presenting masks is
available for training. We have split it into an 88 images training
set (1,307 clusters) and a validation set composed of 22 images (305 
clusters).



We employed image augmentation to mitigate overfitting
<cit.>, adopting transformations of the original images
that could simulate field conditions: differences in lighting, camera
focus, noise or dirty lenses. We applied Gaussian blur, contrast
normalization, additive Gaussian noise and pixel
dropouts[Similar to “pepper” noise – see  
  documentation for details <cit.>.] using the  library
<cit.>. These augmentations were randomly selected and ordered
in such a way that different transformations were applied for each
source image. We have applied 20 random augmentations for each image,
as shown in Figure <ref>, which produced a rich set of
variations, potentially reflecting real field conditions.



We employed the Keras/TensorFlow-based implementation for Mask R-CNN
developed by Matterport, Inc., publicly available at GitHub
<cit.>. The network was initialized using the weights
previously computed for the COCO Dataset <cit.>. No layer was
frozen during training, so all weights could be updated by the
training on the grapes dataset. Due to GPU memory limitations, to
allow multiple images per batch, input images are resized to 1024
× 1024 × 3 tensors, preserving aspect ratio by applying
zero padding as needed.
Two feature extraction architectures were evaluated:
ResNet-101 and the shallower ResNet-50 <cit.>. For the YOLO networks, we employed 
Darknet, the original implementation developed by <cit.>, initialized using
pre-trained weights from ImageNet <cit.>. In our single-class grape 
detection case, C = 1. Training was performed on a computer
containing a single NVIDIA TITAN Xp GPU (12 GB memory), an Intel
i7-x990 CPU and 48 GB RAM, and running Ubuntu 18.04 LTS. For
Mask R-CNN, training was performed in approximately 10 hours (100
epochs, around 6 minutes per epoch). YOLO training (v2 and v3) spent four days 
using the same hardware.



 §.§ Evaluation



The WGISD dataset allows evaluations for the semantic segmentation,
object detection and instance segmentation problems. This work will
present results using the standard metrics of precision (P),
recall (R), and their harmonic mean (F_1), as usual in the
information retrieval literature:

    P = N_tp/N_tp + N_fp,


    R = N_tp/N_tp + N_fn,  and


    F_1 = 2 · P · R/P + R.

These measurements depend on the
number of true positives (N_tp), false negatives (N_fn)
and false positives (N_fp), which need to be properly defined for
each type of problem:

  
  * For semantic segmentation, we are considering just one class
  (grape) and pixel classification. In this case, we employ the
  masked images in the test set for evaluation, where the grape pixels are
  properly marked (27 images, 408 grape clusters). N_tp^seman is the
  number of  pixels correctly classified  as grape pixels according to
  the ground truth, N_fn^seman the number of grape pixels incorrectly
  classified as non-grape pixels, and N_fp^seman the number of
  non-grape pixels wrongly reported as grape ones by the classifier. Such three measures allow
  the computation of P_seman and R_seman, respectively precision and
  recall, for the semantic segmentation problem.

  * In object detection, each grape cluster instance is
  localized by a rectangular bounding box. A hit or a miss is
  defined by a one-to-one correspondence to ground truth instances,
  obeying an intersection over union (IoU) threshold computed
  using the rectangular areas and their intersections. N_tp^box is the
  number of correctly predicted instances (bounding boxes) and N_fn^box
  and N_fp^box are similarly defined. These three
  measures give the values for P_box and R_box, respectively precision and
  recall, for the object detection problem and the evaluation is
  performed for the entire test set (58 images, 837 grape clusters).       
 
  * Instance segmentation follows the same instance-based
   logic as in object detection, but IoU is computed using the areas
   and intersections of the masks instead of rectangular bounding
   boxes. Again, we are limited to the masked images in the test set: 27
   images containing 408 clusters. The measures are P_inst and R_inst for
   instance segmentation precision and recall, respectively. 

Mask R-CNN results can be evaluated for the three problems, but the
YOLO-based results are just evaluated regarding object detection. 



 §.§ Spatial registration: 3-D association



Structure-from-Motion (SfM) <cit.> is a fundamental achievement in
computer vision and a core component in modern photogrammetry. It solves
the camera pose and scene geometry estimation simultaneously,
employing only image matching and bundle adjustment <cit.>, and
finding three-dimensional structure by the motion of a single
camera around the scene (or from a set of independent cameras). In a
previous work <cit.>, we showed that SfM can
recover vine structure from image sequences on vineyards. It is an
interesting alternative to integrate image data from different camera poses
registering the same structures in space. Similarly to Liu et
  al. <cit.>, we use 3-D data from the COLMAP SfM
software <cit.> to perform spatial registration, integrating the fruit
instance segmentation data produced by the perceptual CNN-based step. 

Consider the directed graph G = (V, E), where V is a set of nodes u_i,j
 representing the j-th instance found by the neural network in the i-th
frame. Consider the set of 𝒳 = {𝐗_k}_k=1..M of
M three-dimensional points 𝐗_k found by the SfM 
procedure. We create an oriented edge (u_i,j, v_i',j') ∈ E,
considering i < i',  if there is a 3-D point 𝐗_k that
projects to the instance j in frame i and to instance j' in frame
i'. In other words, there is a link between instances from two
different frames if there is a three-dimensional point whose
2-D projections are contained in the masks associated to 
these instances, evidence they could be observing the same object
in the 3-D world.

Figure <ref> (a) shows the graph G. Each
node represents an instance (a grape cluster) and a column of nodes
represents all instances found in a frame i. Neighboring columns
represent successive frames i and i', i < i'. G is a
directed graph, with oriented edges indicating instance association
between frames. Each edge has a weight w[u_i,j, v_i',j'] that
indicates the total number of three-dimensional points that links the two
instances, accumulating the evidence that associates instance j in
frame i to the instance j' in i'.



The structure of G is affected by occlusions: if changes in camera
pose make two or more clusters to occlude each other, then two or more edges
will be incident to a node u_i,j. Similarly, when the camera movement
reveals two or more occluded clusters, two or more edges will flow
from the same node. We filter the edges in E in such 
a way that, for each node, there is up to one incident edge and up to
one departing edge. The filtering strategy is simple: the
heaviest (maximum weight w) edge is kept. The intuition
behind this strategy is it would favor the occluding grape
cluster while the occluded one is tracked by an edge spanning many
frames – that means (u_i,j, v_i',j') where i' > i +
1. These edges spanning many frames also help with the relocalization
of grapes occluded by other objects in the scene (leaves, trunks,
etc.) missed by the neural network in some frames. 

After edge filtering, nodes are sorted by their frame index i and,
for each node u_i,j, we find the longest path in G using
depth-first search on edges, corresponding to the track of one grape
cluster along the frame sequence. Too short paths (we use a threshold
of 5 edges) are filtered out, an attempt to remove false positives
from the perceptual stage by integrating evidence from multiple
frames. Figure <ref> (b) illustrates the final
set of longest and disjoints paths, where different colors
discriminate different tracks (different grape clusters). The number
of paths is an estimation of the total number of grape clusters in the
entire image sequence.





§ RESULTS



The validation set was employed to select the best models for further evaluation
on the test set. For the Mask R-CNN, the ResNet 101 feature extraction backbone
produced the best results. Table <ref> presents the evaluation of 
predictions produced by Mask R-CNN for instance segmentation, considering the masked test
set (408 clusters in the ground truth) and confidence threshold of 0.9
for the grape class[In the experiments, Mask R-CNN did not exhibit great
  sensibility to the confidence threshold. We have tested 0.5, 0.7,
  0.9 and 0.95, all presenting very similar results, and F_1 variations
  inferior to  0.005.]. The table shows the precision and recall measures
for seven different values of IoU, from 30% to 90%. The corresponding values 
for F_1 score and average precision[The AP
  summarizes the shape of the precision/recall curve, and it is
  defined as the mean precision at a set of equally spaced
  recall levels. See <cit.> for details.] (AP) as defined in Pascal VOC Challenge
<cit.> are also presented.  Considering the diversity in
clusters sizes and shapes, IoU is specially important: higher values
indicate better grape berries coverage, a desirable property for yield
prediction applications. Lower IoU values indicate poorer berry
coverage or disagreement in clusters segmentation between prediction
and the ground truth, that means divergence in the berries' assignment
to clusters.      











































Figure <ref> shows five examples of instance
segmentation results produced by the Mask R-CNN. It illustrates the
network capability to learn shape, compactness and color
variability, and discriminate occluding foreground as branches and
trunks. Inter-variety color variation (Chardonnay/Sauvignon
  Blanc vs. Cabernet/Syrah) and intra-variety color variation (Syrah and Cabernet maturity) are properly modeled by the
  network, as well as shape, size and elongation (Chardonnay vs. Cabernet,
  for example). The confidence level is also expressive: even
  considering that the confidence threshold is 0.9, most of the instances
  present levels equal or close to 1.0. Values lower than 0.99 can be
  observed in cases of severe occlusion, like the leftmost grape cluster
  in the Syrah example.



Grape cluster segmentation is challenging, even to the human
annotators: occlusions and the absence of 3-D input or on-site
annotation make the dataset error-prone regarding the correct
segmentation of large agglomerations of
clusters. Figure <ref> shows a 
case where segmentation divergence produces false negatives and false
positives in the evaluation, although the almost correct detection of
the grape berries. Note that for the clusters on the center in
Figure <ref>, the prediction looks like a more
reasonable segmentation: two 
clusters are more plausible than one big and bifurcated cluster
proposed by the ground-truth, but we would need in-field inspection
(or a 3-D model) to check the correct detection. The other divergences
in the figure illustrate the difficulty of the segmentation task: the
clusters proposed by the prediction and the ground-truth look equally
plausible, and again only an in-field checking could provide the right
answer. Difficulties on successful cluster segmentation were also
reported by <cit.>. As noted before, segmentation divergence can also
deteriorate IoU. Fortunately, we will see further in this work than
data from a moving camera and 3-D association can relief such
occlusion issues by properly integrating images from multiple poses.



R_inst and P_inst can suffer from erroneous segmentation, but what
about semantic segmentation? As can be seen in
Figure <ref>, despite cluster segmentation errors,
at the berry level most of the grape pixels look properly
detected. To evaluate the detection of grape pixels, we use the
measures R_seman and P_seman, recall and precision for the semantic
segmentation variation of the problem. Table <ref> 
shows the overall result for
semantic segmentation on the entire masked set (last line), but also
the results found for each one of the 27 images. The table groups the
masked test set by the different varieties, allowing a comparison
across different grape types. The overall F_1 score for semantic
segmentation is 0.89 and no single variety has exhibited a remarkably
different score. This is also an evidence that berries assignment to
individual clusters (cluster segmentation) is the main factor
affecting IoU.



Table <ref> presents the results for
object detection produced by the three networks, considering the entire test set of 837
clusters in 58 images. It is worth remembering that the models were trained using the
masked training set, composed of 88 images (1,848 after
augmentation), but the results in Table <ref>
show the evaluation for the entire “boxed” test set (considering
intersection over union for the rectangular bounding boxes produced by
Mask R-CNN). The recall values in the table show the YOLO networks
lose more clusters if compared to the Mask R-CNN network, specially
YOLOv3. Figure <ref> shows some examples of
object detection for the three networks. In the figure we can see more
false negatives (lost clusters) in the YOLOv3's results. Also in
Table <ref>, as we require higher values of IoU
(what means better adjusted bounding boxes), YOLO networks show worse
results compared to Mask R-CNN.        






To evaluate the spatial registration method and the potential of the
entire methodology to address fruit counting, we employed a video
sequence captured on field. Multiple counting is avoided by the
tracking produced by the 3-D assignment: the number of individual
tracks should correspond to the number of observed clusters in the
video sequence, as seen previously in
Figure <ref> (b). The video sequence was captured by a
smartphone camera 
in full-HD (1,920 × 1,080 pixels) while a service vehicle moved
along a row of vines. The keyframes of the MPEG video sequence were
extracted and the first 500 keyframes were employed in this
evaluation. Employing keyframes from the MPEG stream is useful
because (i) these frames present fewer compression artifacts than other frames
in the video sequence, (ii) the number of images (frames) is reduced,
and (iii) there is still sufficient overlap between frames to perform the
feature correspondence needed by structure-from-motion and to provide multiple
views for each grape cluster. Mask R-CNN inference was performed for
each keyframe and the found mask stored. COLMAP was employed to create
a sparse 3-D model by SfM. Finally, the spatial registration proposed
on Section <ref> was employed, matching the clusters
along the frame sequence
(Figure <ref>). The results for the entire
frame sequence can be seen in an available
video[<https://youtu.be/1Hji3GS4mm4>. Note the video is
  edited to a 4 frames/second rate to allow the viewer follow the
  tracks more easily.].     






§ DISCUSSION



The Mask R-CNN network presented superior results as compared to the YOLO networks. 
Considering IoU values equal or superior to 0.5, the advantage of Mask R-CNN 
becomes more salient: even considering a 70% IoU, the F_1 score is 
impressive. As a reference, <cit.> reported 
0.828 and 0.848 F_1 scores for sweet peppers and rock melons respectively
at 0.4 IoU using Faster R-CNN while <cit.>
reported a 0.90 F_1 for apples and mangoes considering a 0.2 IoU, also
employing Faster R-CNN. However, readers should keep in mind that it is just
a reference, not a direct comparison or benchmark considering the different
crops and datasets.

The use of three different scales by YOLOv3 could not be an advantage
over YOLOv2 considering the almost constant distance between the camera and
the vineyard row. In the same way, Mask R-CNN's use of FPN could be reconsidered. 
Agronomical constraints could be explored: how big a group of berries 
should be to be considered a cluster? In other words, the operational and 
agronomical  context should be explored to define the scales of interest.
YOLOv3 employs multi-label classification, useful for problems presenting 
non-mutually exclusive object classes. However, considering our single class fruit detection
problems, this would not be an advantage of YOLOv3 compared to YOLOv2. Considering
that the YOLOv3 is deeper and, as consequence, prone to overfitting, it could need
more data to reach and surpass the results of YOLOv2, as observed in 
Table <ref>. Although the better results
produced by Mask R-CNN, it is important to note that rectangular bounding box
annotation for object detection is faster to be produced, what means
bigger datasets could be available to train networks like YOLOv2 and
YOLOv3 – in this work, we have constrained YOLO training to the same
dataset available to Mask R-CNN.

What could we be say about the generalization of these models?
In other scenarios, like different developmental stages, crop
production systems or camera poses, they would be able to detect grape
clusters properly? 
Figure <ref> shows the results produced by the
Mask R-CNN for images collected from the Internet[The four
  images in Figure <ref> are public-domain or Creative
Commons-licensed pictures. Their authors are McKay Savage
(<https://www.flickr.com/photos/56796376@N00/4034779039>), Hahn
Family Wines
(<https://commons.wikimedia.org/wiki/File:Hand_harvesting_wine_grape.jpg>),
Peakpx (<http://www.peakpx.com/564748/green-grape-vine>) and Circe
Denyer
(<https://www.publicdomainpictures.net/en/view-image.php?image=298996>.]). No
parameter 
tuning or other adaptions was
employed. Figures <ref> (a) and
<ref> (b) show  mature crops in
camera poses very different from the images in WGISD, the latter showing
a worker and a tool – elements that are not present in the
dataset. The network was able to detect of a considerable
part of the instances. Figure <ref> (c) shows an
example where false positives appeared in parts of textured leaves,
but most of the berries were detected and the predicted clusters looks
plausible. Finally, Figure <ref> (d) shows clusters in
an earlier developmental stage. Apparently, the network is splitting
the clusters in regions of higher compactness, probably because the
training set in WGISD provides examples of more compact clusters. Such
qualitative results indicate the model present good generalization and
accurate results could be produced by tuning and transfer learning.   

The presented CNN-based detectors can be integrated in larger systems
that, employing a data association strategy, will be able 
to integrate the detections and perform localized fruit
counting on site. A moving camera, combined to fruit tracking, is a
powerful way to deal with occlusions due the integration of
different camera poses. Integrating video information  can also
relieve localized errors in fruit detection in a few frames.
As shown, an ordinary 1080p RGB camera can produce
input for accurate results, being an affordable approach to fruit
counting and orchard inspection. Such vision systems can be easily
integrated in tractors, implements, service vehicles, robots and UAVs,
possibly employing high performance processing units (GPUs and TPUs)
with low energy consumption or even edge computing
<cit.>.

Notwithstanding, while our spatial integration is employing a
computational-intensive process such as structure-from-motion, other
implementations could use SLAM algorithms (simultaneous localization
and mapping), the real-time formulation 
of SfM. <cit.> avoided the
computationally-intensive process of feature detection and matching in
SfM by employing the fruits' centers found by Faster R-CNN and Kalman Filter
tracking for inter-frame association. In other words, the fruits
became the landmarks for the SfM procedure (implemented in
COLMAP). However, it is unclear what happens if no fruits are
available in a segment of the video sequence. A fast SLAM algorithm such as
ORB-SLAM <cit.> or SVO <cit.>, not relying
on any specific landmark, could be a more robust alternative.






§ CONCLUSION



Computer vision's current maturity level can produce impressive
and robust results in photogrammetry and perceptual tasks, even in 
challenging outdoor environments such as agricultural orchards. Combining
structure-from-motion (or its real-time version: SLAM) and
convolutional neural networks, advanced monitoring and robotics
applications can be developed for agriculture and livestock.

This work presents a methodology for grape detection, tracking and
counting in vineyards employing a single off-the-shelf 1080p
camera. We have reached F_1 scores superior to 0.9 for instance detection
in wine grapes, a challenging crop that presents enormous variability
in shape, size, color and compactness. We also showed that 3-D models
produced by structure-from-motion or SLAM can be employed to track
fruits, avoiding double counts and increasing tolerance to errors
in detection.  The same methodology could be used successfully for other
crops produced in trellis-like systems such as apples, peaches and
berries. Adaptions of the methodology can be  developed for fruits
grown in trees presenting bigger canopies, like citrus and mangoes –
yield could be estimated from regression from the visible fruit
counts.

Further research could consider more integration between the
photogrammetry and perception modules, looking for more
sophisticated scene understanding systems able to robustly cope
with occlusions and other sources of errors.    



§ ACKNOWLEDGMENTS



This work was supported by the Brazilian Agricultural Research Corporation
(Embrapa) under grant 01.14.09.001.05.04 and by CNPq PIBIC Program
(grants 161165/2017-6 and 125044/2018-6). S. Avila is partially funded by
Google LARA 2018 & 2019, FAPESP (2017/16246-0) and FAEPEX (3125/17). We thank
Amy Tabb who provided helpful comments on the first version of the
manuscript. We also thank to Luís H. Bassoi and Luciano V. Koenigkan
for their generous support and help. Special thanks to Guaspari Winery
for allowing image data collection.  



model5-namesauthoryear







§ EMBRAPA WINE GRAPE INSTANCE SEGMENTATION DATASET – EMBRAPA
  WGISD



This section presents a detailed description of the dataset, a datasheet for the dataset 
as proposed by <cit.>.



 §.§ Motivation for Dataset Creation





  §.§.§ Why was the dataset created?



Embrapa WGISD (Wine Grape Instance Segmentation Dataset) was created
to provide images and annotation to study object detection and
instance segmentation for image-based monitoring and field robotics
in viticulture. It provides instances from five different grape
varieties taken from the field. These instances show variance in grape
pose, illumination and focus, including genetic and phenological
variations such as shape, color and compactness. 



  §.§.§ What (other) tasks could the dataset be used for?



Possible uses include relaxations of the instance segmentation
problem: classification (Is a grape in the image?), semantic
segmentation (What are the “grape pixels” in the image?), and object
detection (Where are the grapes in the image?). The WGISD can also be used
in grape variety identification.






  §.§.§ Who funded the creation of the dataset?



The building of the WGISD dataset was supported by the Embrapa SEG
Project 01.14.09.001.05.04, Image-based metrology for Precision
Agriculture and Phenotyping, and the CNPq PIBIC Program (grants 161165/2017-6 and 
125044/2018-6).



 §.§ Dataset Composition





  §.§.§ What are the instances?



Each instance consists of an RGB image  and an annotation describing
grape clusters locations as bounding boxes. A subset of the instances
also contains binary masks identifying the pixels belonging to each
grape cluster. Each image presents at least one grape cluster. Some grape
clusters can appear far at the background and should be ignored. 



  §.§.§ Are relationships between instances made explicit in the data?



File names prefixes identify the variety observed in the instance
(Table <ref>). 



















  §.§.§ How many instances of each type are there?



The dataset consists of 300 images containing 4,432 grape clusters
identified by bounding boxes. A subset of 137 images also contains
binary masks identifying the pixels of each cluster. It means that
from the 4,432 clusters, 2,020 of them present binary masks for
instance segmentation, as summarized in
Table <ref>. 



  §.§.§ What data does each instance consist of?



Each instance contains an 8-bit RGB image and a text file
containing one bounding box description per line. These text files
follow the “YOLO format” <cit.>:



class is an integer defining the object class – the dataset presents
only the grape class that is numbered 0, so every line starts with 
this “class zero” indicator. The center of the bounding box is the point
(c_x, c_y), represented as float values because this format normalizes
the coordinates by the image dimensions. To get the absolute position,
use  (2048 · c_x, 1365 · c_y). The bounding box dimensions are given by
W and H, also normalized by the image size. 

The instances presenting mask data for instance segmentation contain
files presenting the  extension. These files are compressed
archives for NumPy n-dimensional arrays <cit.>. Each array is a 
H × W × n_clusters three-dimensional array
where n_clusters  is the number of grape clusters observed in the image. After assigning the NumPy array to a variable , the mask for the i-th grape cluster can be found in
. The i-th mask corresponds to the i-th line in the bounding
boxes file.

The dataset also includes the original image files, presenting
the full original resolution. The normalized annotation
for bounding boxes allows easy identification of clusters in the
original images, but the mask data will need to be properly 
rescaled if users wish to work on the original full resolution. 



  §.§.§ Is everything included or does the data rely on external resources?



Everything is included in the dataset.



  §.§.§ Are there recommended data splits or evaluation measures?



The dataset comes with specified train/test splits. The splits are
found in lists stored as text files. There are also lists referring
only to instances presenting binary masks. 














Standard measures from the information retrieval and computer vision
literature should be employed: precision and recall, F_1 score and
average precision as seen in <cit.> and <cit.>.



  §.§.§ What experiments were initially run on this dataset?



To the present date, this work describes the first experiments run on
this dataset.



 §.§ Data Collection Process





  §.§.§ How was the data collected?


Images were captured at the vineyards of Guaspari Winery, located at
Espírito Santo do Pinhal, São Paulo, Brazil (Lat -22.181018, Lon
-46.741618). The winery staff performs dual pruning: one for shaping
(after previous year harvest) and one for production, resulting in
canopies of lower density.  The image capture was realized in April
2017 for Syrah and in April 2018 for the other varieties (see
Table <ref>).  No pruning, defoliation or any
intervention in the plants was performed specifically for the dataset
construction: the images capture a real, trellis system-based wine
grape production. The camera captures the images in a frontal pose,
that means the camera principal axis is approximately perpendicular to
the wires of the trellis system and the plants rows.

A Canon™   EOS REBEL T3i DSLR camera and a
Motorola™  Z2 Play smartphone were used to 
capture the images. The cameras were located between the vines lines,
facing the vines at distances around 1-2 meters. The EOS REBEL T3i
camera captured 240 images, including all Syrah pictures. The
Z2 smartphone grabbed 60 images covering all varieties except
Syrah. The REBEL images were scaled to 2048 × 1365 pixels and
the Z2 images to 2048 × 1536 pixels (see Section <ref>).
More data about the capture process can be found in the
Exif data found in the original image files, included in the dataset. 



  §.§.§ Who was involved in the data collection process?


The authors of this paper. T. T. Santos, A. A. Santos and
S. Avila captured the images in field. T. T. Santos, L. L. de Souza 
and S. Avila performed the annotation.



  §.§.§ How was the data associated with each instance acquired?


The rectangular bounding boxes identifying the grape clusters were
annotated using the 
tool[<https://github.com/tzutalin/labelImg>]. The clusters
can be under severe occlusion by leaves, trunks or other
clusters. Considering the absence of 3-D data and on-site annotation,
the clusters locations had to be defined using only a single-view image,
so some clusters could be incorrectly delimited.

A subset of the bounding boxes was selected for mask annotation, using
a novel tool developed by the authors and presented in this work. This
interactive tool lets the annotator mark grape and background pixels
using scribbles, and a graph matching algorithm developed by
<cit.> is employed to perform image segmentation to every
pixel in the bounding box, producing a binary mask representing
grape/background classification.



 §.§ Data Preprocessing




  §.§.§ What preprocessing/cleaning was done?



The following steps were taken to process the data:


  
  * Bounding boxes were annotated for each image using the
     tool.
  
  * Images were resized to W = 2048 pixels. This
    resolution proved to be practical for mask annotation, a
    convenient balance between grape detail and time spent by the
    graph-based segmentation algorithm.
  
  * A randomly selected subset of images was employed for mask
    annotation using the interactive tool based on graph matching.
  
  * All binaries masks were inspected, in search of pixels
    attributed to more than one grape cluster. The annotator assigned
    the disputed pixels to the most likely cluster.
  
  * The bounding boxes were fitted to the masks, which provided a
    fine-tuning of grape cluster locations. 




  §.§.§ Was the “raw” data saved in addition to the
  preprocessed data?


The original resolution images, containing the Exif data provided by
the cameras, is available in the dataset.



 §.§ Dataset Distribution




  §.§.§ How is the dataset distributed?


The dataset is available at Zenodo <cit.>.



  §.§.§ When will the dataset be released/first distributed?


The dataset was released in July 2019.



  §.§.§ What license (if any) is it distributed under?


The data is released under Creative Commons BY-NC 4.0
(Attribution-NonCommercial 4.0 International license). There is a
request to cite the corresponding paper if the dataset is used. For
commercial use, contact the Embrapa Agricultural Informatics business
office at .



  §.§.§ Are there any fees or access/export restrictions?


There are no fees or restrictions. For commercial use, contact Embrapa Agricultural 
Informatics business office at .



 §.§ Dataset Maintenance




  §.§.§ Who is supporting/hosting/maintaining the dataset?


The dataset is hosted at Embrapa Agricultural Informatics and all
comments or requests can be sent to Thiago T. Santos at
 (maintainer).



  §.§.§ Will the dataset be updated?


There are no scheduled updates. In case of further updates, releases will be
properly tagged at GitHub.





  §.§.§ If others want to extend/augment/build on this dataset,
  is there a mechanism for them to do so?


Contributors should contact the maintainer by e-mail.



  §.§.§ No warranty


The maintainers and their institutions are exempt from any liability, judicial or extrajudicial, for any losses or damages arising from the use of the data contained in the image database.

