††footnotetext: Copyright © 2020 for this paper by its authors. Use permitted
under Creative Commons License Attribution 4.0 International (CC BY 4.0). CLEF
2020, 22-25 September 2020, Thessaloniki, Greece.11institutetext:  SenticLab,
Iasi, Romania
22institutetext: Faculty of Computer Science, ”Alexandru Ioan Cuza” University
of Iasi, Romania
<EMAIL_ADDRESS>
# Revealing Lung Affections from CTs.
A Comparative Analysis of Various Deep Learning Approaches for Dealing with
Volumetric Data

Radu Miron 1122 Cosmin Moisii 1122 Mihaela Elena Breaban 1122

###### Abstract

The paper presents and comparatively analyses several deep learning approaches
to automatically detect tuberculosis related lesions in lung CTs, in the
context of the ImageClef 2020 Tuberculosis task. Three classes of methods,
different with respect to the way the volumetric data is given as input to
neural network-based classifiers are discussed and evaluated. All these come
with a rich experimental analysis comprising a variety of neural network
architectures, various segmentation algorithms and data augmentation schemes.
The reported work belongs to the SenticLab.UAIC team, which obtained the best
results in the competition.

##  1 Introduction

Medical imaging technologies like Computer Tomography (CT) and Magnetic
Resonance (MR) produce high volumes of data in the form of volumetric images.
The richness of information they provide is essential to correct diagnosis but
brings at the same time new challenges, both for manual/human and
automatic/machine processing: these are not only about the size of the
produced data but also about the complexity of the diagnosis process itself.
With respect to automated diagnosis, the volumetric images, which can be seen
both as matrices of pixels/voxels or series of 2D images (usually called
slices), produced high effervescence in the deep learning research community,
triggering a variety of new architectures and approaches.

The current paper makes use of deep learning to automatically detect
tuberculosis and related affections in lung CTs, in the context of the
ImageClef Tuberculosis task [1, 2]. We investigate three types of approaches,
different with respect to the way the volumetric data is given as input to
neural network-based classifiers. One type, popular among the participants in
the previous year competition [3], is based on reducing the volumetric image
to a small set of 2D projections. Obviously, this approach consistently
reduces the size of the data to be processed by the classifier but inherently
may lose important information. The second type exploits the whole data matrix
by using 3D convolutions or by fusing the information from the slices. The
third type, which was ranked as the winner of the 2020 evaluation session,
consists in moving the decision layer from the whole volume of data to the
slice level. All these three different approaches come with a variety of
neural network architectures, various segmentation algorithms and data
augmentation schemes. The work reported stays behind the SenticLab.UAIC team,
obtaining the best results in the
competition111https://www.imageclef.org/2020/medical/tuberculosis/ [1].

The paper is structured as follows. Section 2 describes the challenge and the
dataset. Section 3 describes the approaches developed based on reducing the
volumetric image to 2D projections, starting with the previous year winning
approach reported in [4], which we further enhanced to address the 2020 tasks.
Section 4 presents the approaches we used to exploit the whole volumetric
information. Section 5 describes the architectures used to process the
information at slice level and the heuristics used to produce the diagnosis
report at the CT level. Because of the large number of approaches we
evaluated, some of them abandoned earlier (not submitted in the competition)
due to poor results on our local validation data, we report and discuss
performance results immediately after each method description. Section 6
summarises the results for the best approaches that were evaluated on the
blind test set in the competition and discusses comparatively the performance
of the three classes of methods. Section 7 concludes the paper.

##  2 ImageClef Tuberculosis: tasks, data, evaluation

The challenge in the 2020 ImageClef Tuberculosis competition is the automatic
detection of tuberculosis and related lesion types in CTs. The CT report to be
generated must contain 3 binary labels for each lung, indicating the presence
of TB lesions in general, the presence of pleurisy and caverns in particular.

The training dataset consists of 283 CTs. All CTs present at least one lung
affected, 19 have pleurisy and 126 caverns. Because we split each CT into
left/right lungs, this translates to 566 inputs, 444 affected, 21 with
pleurisy and 145 with caverns.

The task is therefore a multi-binary classification problem, with three target
labels per lung. For each target label the AUC is computed and the ranking is
done on a test set, first by computing the average AUC and then by the minimum
AUC over the 3 target labels.

We split the data into train/validation in the same fashion as [4] setting
apart every 4th input into the validation set and use this configuration
throughout the competition.

##  3 Squeezing Volumetric Data: 2D Projections

The 3D matrix representing the volumetric image can be reduced to simpler 2D
representations by traversing it in each of its three dimensions and computing
statistics on numeric vectors. In the case of lungs CTs, a segmentation
algorithm is firstly applied to detect the lungs and eliminate the other parts
in the CT. Further, we used the method proposed in [4], where the mean, the
maximum and the standard deviation is computed on each direction, generating
three 2D matrices which can be interpreted as an RGB image (a single 2D image
with 3 channels). All the processing steps described in [4] are kept: mask
erosion, increasing the voxels intensity in the CT by 1024HU, dividing the
mean values and standard deviations values (red and blue channels) by their
maximum, dividing the maximum values (green channel) by 1500. At the end, for
each lung we have a set of three 2D RGB images, each image corresponding to
one of the three dimensions of the 3D matrix.

###  3.1 The Impact of Segmentation

Figure 1: Projections dataset creation flow using 4 segmentation variants (in
order: growth-based, registration-based, unetLTRCLobes and unetR231). Although
the 4 types of projections resulted look only slightly different, the
difference in classification scores is significant.

The first step behind all our approaches is image segmentation, with the aim
of identifying and isolating each lung in the volumetric image. Because the
performance of further processing is greatly influenced by the quality of
segmentation (especially in the case of the 2D projection approach where the
projections take into account the entire volume), we tested several
segmentation methods. The organizers provided for all patients two versions of
automatically extracted masks of the lungs: one which relies only on
anatomical assumptions [5], and one based on non-rigid registration [6].
Additionally, we used _U-net(R231)_ and _U-net(LTRCLobes)_ which were pre-
trained for lung segmentation on large and diverse datasets [7]
222https://github.com/JoHof/lungmask. Our experiments show that the first
technique based on anatomical assumptions behaves much like region growing not
being able to catch holes or necrotic tissue in lungs, the second technique
manages to capture necrotic tissue while the ones based on U-net include
airpockets, tumors and effusions. The flow of the dataset creation together
with some projections corresponding to several segmentation techniques can be
seen in fig. 1.

Feeding a VGG neural network [8] with 2D projections obtained on the segmented
volumetric image, the average AUC scores obtained on our validation set
indicate the registration based method to give the best performance
(AUC=0.693) followed at small distance by _U-net(R231)_ (AUC=0.674) and _U-
net(LTRCLobes)_ (AUC=0.668), but consistently surpassing the anatomy-based
method (AUC=0.580).

Consequently, all our further experiments use the segmentation provided by
non-rigid registration [6].

###  3.2 Data Augmentation

The images go through a series of augmentations, each with a certain
probability of being applied, including: horizontal and vertical flipping,
small degrees of rotations, blurring, added gaussian noise, distortions,
random cropping, and changing different values of hue, saturation or
brightness. We used the Albumentations 333https://github.com/albumentations-
team/albumentations library for most of these augmentations.

###  3.3 The 2D Approach with Preprocessing (_PreProcProj_)

In an effort to improve over the last year result, we used the pre-processing
provided in [9], with the aim to eliminate the small vessels from the
projection, thus making the affected area more obvious. We adopted all the
pre-processing steps that the authors mention, except the regional maxima
calculation. Figure 2 illustrates the difference between projections with and
without further pre-processing.

Figure 2: Comparison between projection without(left) and with pre-
processing(right)

For training we chose AlexNet[10]. The input consists of the three projections
of a volume, on each axis. After extracting features from each projection with
AlexNet, we concatenate all the features, feed them into a linear layer and
predict probabilities for a lung to have affections, caverns, pleurisy or be
healthy. With this approach we scored an AUC of 0.793 on the test set.

###  3.4 The 2D Approach Scoring the Best (_ResNet50Proj_)

We further tried different variants of Resnet[11] and SqueezeNet[12].

Figure 3: The AUC score progress on the hold-out validation set for different
resnet models. Pink: resnet34 with no augmentations, DarkBlue: resnet34 with
augmentations, LightBlue: resnet50 with augmentations

We extracted the lungs using the registration-based segmentations, computed
all 3 projections, split by lung side and processed with the augmentation we
listed in section 3.2. We trained the networks and aggregated the results on
all 3 projections and computed the mean score to obtain the final results back
at CT level. We tried different approaches in aggregating the results
including training a small neural network, but found the simpler mean
aggregation to give the highest score. We thus obtained our highest score in
the 2D approach using a resnet-50 network [11] pretrained on Imagenet[13] with
an AUC on our hold-out validation set of 0.877; however this result was not
submitted. Our first submission to the competition was a resnet34 model with
no augmentations which obtained on the hold-out validation set and on the test
set the same AUC score of 0.825. In Figure 3 we can see the AUC progress on
different models we tried.

##  4 Exploiting Volumetric Data as a Whole

###  4.1 3D Convolutions

In an attempt to make use of the whole volume at once, we used SqueezeNet in a
3D version, based on the implementation found in the repository
444https://github.com/okankop/Efficient-3DCNNs. In order to work with volumes
of different sizes, we used batch size equal to 1. To handle volumes with a
large number of slices, we used Apex555https://github.com/NVIDIA/apex library
for reducing the burden on our GPU. In a preliminary experiment we considered
only the case affected vs. not affected. We noticed the bad results during the
training: after some epochs the prediction scores stagnated, for all volumes,
between $0.4$ and $0.6$. We concluded that 3D convolutions are not able to
capture the important information on our small training set of volumetric
images.

###  4.2 Slices fusion

In our attempt to associate the entire volumetric image to a label, we
constructed a hybrid approach. We fed the volume slice by slice into a
convolutional neural network, fused the resulted feature maps at channel level
and continued with another small convolutional network into a prediction. The
initial convolutional neural network is composed from the encoder part of a
U-net[14] architecture which was pretrained on a segmentation task at the end
of which we applied a squeeze connection to reduce the number of channels,
fused the resulting feature maps so that the slices processed in parallel by
the CNN would now be treated as channels of a single input, then applied a
resnet-like small network to compile the features into a label. We no longer
use the masks to extract the lungs but instead use a simple threshold based
segmentation to compute the boundaries of the body and crop out the space
around the it. We again split by lung side and used only horizontal flip as a
preprocessing. The resulting volume is resized to the fixed size of (128, 256,
256) The network could then be fed images in batches multiple of 128
representing the slices of a volume.

To make maximum use of the GPU memory, we used the Apex library to train using
mixed precision, in a distributed manner on 2 GPUs. We could fit 2 times 128
images into the memory corresponding to 2 volumes.

The approach turned out to be cumbersome. The time to process an epoch was
relatively high and the convergence of the network seemed slow. After 2 days
of training we decided to stop and the network reached an AUC of around 0.6 on
the hold-out validation set.

##  5 Sequencing Volumetric Data: a Slice by Slice Classification Approach

Having a closer look at the training set, one can observe that usually the
lesions on the lungs are located only on a small number of slices from the
whole volume. A natural idea is to try a 2D model that could differentiate
between healthy lung slices and lung slices with lesions (caverns, pleurisy
and affections) and construct the CT report based on the findings at slice
level. For this purpose we need training data labeled at slice level and not
CT level.

The first approach was to try to automatically detect the slices presenting
lesions in the training set, using a lung nodule
detector666https://github.com/BCV-Uniandes/LungCancerDiagnosis-pytorch
constructed by the winners of a challenge in cancerous nodules detection. The
results were bad, the model not being able to recognize the slices showing
caverns although these correspond to big, obvious regions.

Therefore, we started to manually select from each volume of the training set
the slices with lesions. We actually found that this was not as time-consuming
as we initially thought, by processing only the volumes labeled with lesions,
and it definitely was worth the effort, as the increase in performance shows.
The caverns are usually big and obvious and the affections are either nodules
or dusty lungs images (which may indicate pneumonia), with very rare cases of
pneumotorax (the lung disappearing due to the outbreak of a cavern). There are
many cases when these lesions appear on a very small number of slices and
thus, the two approaches described in sections 3 and 4 might have not been
able to reveal them.

###  5.1 InceptionNet

In our first tries using the slice by slice approach, we used InceptionNet
version 3[15]. We used the annotated data in different ways. Transforming each
slice of a volume into a picture, resizing each image to a size of $299\times
299$, cutting the picture in half to obtain the two lungs and using vertical
flip for all the pictures which are either affected, with caverns or with
pleurisy, are the data pre-processing steps for our first attempt using this
approach. This approach does not use the provided segmentation masks at all.
We only used 4 labels as output: affected, caverns, ok, pleurisy.

As input for the neural network we used several versions, having all images as
3-channel images:

a) NaiveInception. We added a linear layer on top of the last adaptive average
pooling layer of the architecture, keeping the original InceptionNet weights
freezed. With this approach we scored 0.86 AUC score on the test set,
surpassing this way the best approach based on 2D projections.

b)ThresholdInception. The other approaches consist in using some other pre-
processing steps. This time, when creating the photos from the 3D volume, we
used a Window Width and Window Level equal to 1500, -500 respectively. This
way we improved the results to 0.887 mean AUC and to 0.82 min AUC on the test
set.

c) TwoPicInception. One other approach consists in mimicking the protocol a
doctor has to follow in order to decide affections(including caverns) and
pleurisy. In order to see the affections more clearly, a doctor uses Window
Width and Window Level equal to 1500, -500 respectively, whereas for better
visualisation of pleurisy, a doctor looks at pictures with Window Width and
Window Level equal to 350, 50 respectively. With this thresholding, the liquid
surrounding the pleura becomes more observable. Figure 4 top shows the
differences between the two pictures.

In order to use information from 2 pictures during training, we used two
InceptionNet modules, with trainable parameters and concatenated the two
outputs of the adaptive average pooling layer. The decision was made based on
the output of the last linear layer applied on the concatenation discussed
above. With this approach we scored 0.89 mean AUC on the test set.

Figure 4: Top: Comparison between different threshold values for the HU units
Bottom: Attention visualization for the three layers from InceptionNet

d) AttentionInception.We wanted to gain insight into how accurate the methods
can be. We tried to check the predictions produced by the methods proposed and
discovered that the models found in a big proportion correct slices of the
volumes which contained certain affections. In order to work on the
explainability of our model, we modified the structure of ThresholdInception,
using the idea from [16], introducing an attention mechanism. Instead of
feeding the output of the last pooling layer into the linear layer, we used
dot product attention [17]. We computed similarity scores between the output
of the pooling layer and three different convolutional layers in the
architecture. After using the compatibility scores as weights for the features
extracted by the three layers, we concatenated the new features. Using a
linear layer on top, we predicted scores for the 4 categories. After plotting
the attention we noticed that the attention on the first layer selected
highlighted the whole lung area - supporting the idea that we don’t need the
segmentation masks, whereas the second layer of attention highlighted
affections on the lungs. With this approach we scored 0.85 mean AUC. We
believe this lower performance is due to the fact that the attention on the
last layer was not good. Checking the visualisation for that layer we noticed
useless areas highlighted (Fig. 4, bottom). Because of the limit imposed on
the number of submissions, we stopped investigating this direction.

For all the methods above based on InceptionNet, training was performed for 30
epochs on one GPU Nvidia RTX 2070 with 8GB of memory, using Stochastic
gradient descent optimizer. We divided the learning rate at each 10 epochs by
10 and used binary cross-entropy as loss function.

In order to establish the diagnosis for a volume we applied the following
heuristic: we applied the inference step on all the pictures/slices from the
volume and for each of the possible classes we took the maximum score
encountered; if only one slice was found with an affection score higher than
0.8, then we divided the score of affected by 2.

###  5.2 EfficientNet

In an effort to use a powerful, yet small footprint network, in our last
approaches we used efficientnet[18], specifically the b4 variant which has
only 19M parameters but reaches top 1 accuracy of 82,6% on Imagenet. We used a
Pytorch implementation pre-trained on Imagenet[13].

The preprocessing we used here is similar to the ones we used before and took
place at run-time on load. We applied the registration-based mask per slice
based on a threshold to crop the body and remove much of the surrounding
space, split the lungs into left/right (just by using splitting the image in
half) and applied the same series of augmentations as in the previous
approaches. The split and cropped image has dimension 256 $\times$ 256 and
after randomly cropping it reduces to 224 $\times$ 224\. For ease of working
we also kept the size of the volumetric image depth to a fixed 128 slices per
volume.

If otherwise specified for this approaches as for the others we used a window
level of -500 and range of 1500 corresponding to the most common values used
in areas of acute differing attenuation values (example: lungs) where air and
vessels will sit side by side.

As input we tried several options, all of them maintaining 3 channels per
image:

a) Micro-volumes (_MicroVolSlice_): The importance of volumetric data is
evident. This seemed especially apparent when we try to manually identify
caverns which can present as rounded or irregularly shaped black centers
surrounded by a white contoure. The caverns can range in size from small with
a thin contoure line to large with thick and diffuse borders. The small
caverns we found especially hard to identify as it can be confused with a
section of a larger blood vessel. As untrained individuals, to eliminate the
confusion we traced the potential cavern a few slices up or down to verify if
it continues into a vessel or forms a pathology. To try and mitigate this type
of confusion in a model we composed the 3 channels of the image from 3
consecutive (or equidistant) slices. In case the slice is at the beginning or
end of the sequence we simply duplicated it to fill the channels.

Figure 5: Top left: sample of micro-volume images. Top right: Sample of false-
color images. Bottom: Sample of ”naive” images.

b) False-color (_FalseColorlSlice_): To make use of the entire range of values
of an MRI image we established 3 intervals in the Hounsfield units range to
correspond to the 3 channels of an image. The window size and level used
777https://radiopaedia.org/articles/windowing-ct are (1500, -500)
corresponding to the usual values used for lung imaging, (350, 40) called
narrow window (used when examining areas of similar attenuation, for example,
soft tissue) and (500, -600) a narrower window of the usual values for lung
imaging in an attempt to retain more information around the values
corresponding to blood vessels and soft tissues. The result with this method
however was not submitted to the site as the result on the hold-out set was
poorer than the others.

c) Naive (_NaivelSlice_): The image is simply duplicated into the 3 channels.

To compile the final results for each volume we aggregate the individual
results per slice and choose the max of each each label across all the slides,
which are then used to compute the AUC.

Loss: Cross entropy vs Binary cross entropy : A strange case comes from the
fact that using the CrossEntropyLoss (on a multi-label classification) without
softmax before the loss (thus assigning a predominant label for each slice) we
obtained higher results than using BinaryCrossEntropyLoss. The nature of the
results is also very different, the first giving results on the extremes while
the latter hovering around 0.5, but both giving decent results around $90\%$
AUC.

Figure 6: The schematic of the slice approach (microvolumes). It follows a
simple flow. The volumetric image, split by left/right side is cropped using a
simple threshold based segmentation, then composed (in this case) to
microvolumes, augmented and passed through the model. The output from all the
slices of a side of a volume is then aggregated and the max per label selected
to compose the final result for a side.

Micro-volumes and just repeating the image gave similar results on the test
set (92.2% and 92.4% respectively), however the training on the ”naive” case
was done on 130 epochs on 3 GPUs with batch 56 $\times$ 3 whereas the ”micro-
volumes” case was done on 60 epochs on 2 GPUs with batch $56\times 2$.
Therefore, the approach with the highest score on our hold out set was c) the
simple one which also represented the highest of the submitted scores. Our
second highest submitted model is represented by the approach in a) micro-
volumes.

For training we used Nvidia RTX 2070 with 8GB of memory. We again, used the
Apex library from NVidia to train using mixed precision and
DistributedDataParallel with one process per GPU.

##  6 Comparative results

###  6.1 Results on the competition test set

Table 1 summarizes the results obtained in the competition on the test set.
Submissions were made only for the methods based on 2D projections and the
ones based on predictions at slice level; as shown on the hold-out validation
data, the attempts to use the whole volume using 3D convolutions or fusing the
information at slice level did not obtain good results and therefore were not
used in the competition test phase.

Table 1: Results reported on the test set, in the order of submission. The first two entries use 2d projections, while all the others make predictions at slice levels. (CE suffix represents models with CrossEntropyLoss and BCE models with BinaryCrossEntropyLoss Method | mean AUC | min AUC
---|---|---
ResNet50Proj | 0.825 | 0.766
PreProcProj | 0.793 | 0.703
NaiveInception | 0.860 | 0.772
ThresholdInception | 0.887 | 0.821
AttentionInception | 0.853 | 0.788
TwoPicInception | 0.892 | 0.830
NaiveSliceCE | 0.924 | 0.885
MicroVolSliceCE | 0.922 | 0.860
NaiveSliceBCE | 0.899 | 0.862

###  6.2 Discussion

By comparing the results both on our hold-out validation set and on the test
set, the following conclusions can be drawn.

  * •

As indicated by the low training accuracy, the approaches using the entire
volumetric data as a whole corresponding to the segmented lungs (described in
section 4), involving 3D convolutions or slice fusion, seem to be overwhelmed
by the amount of parameters to fit and are not able to identify the lesions in
cases where these are small or present only on a few slices of the CT, or
either converge slowly.

  * •

The 2D approaches based on projections computed over the segmented volume
(described in section 3) give (unreasonable) good results, which indicates
that simple (normalized) statistics like mean, maximum and standard deviation,
when used together, are able to catch important information about the presence
of lesions in lung CTs. The quality of segmentation of the lungs is of
critical importance in this case, as a bad segmentation may introduce noise
into the projections. After obtaining the set of 2D projections, data
augmentation increased the generalization capability of the classifier.

  * •

The best approach, surpassing significantly the ones based on 2D projections,
exploits all the information present in the segmented volumetric lungs in a
slice-wise manner. Instead of predicting the presence of the affection per CT,
we predict it for each slice. To obtain the report back at lung level the
probabilities over slices are aggregated by extracting the maximum. An
important pre-processing step consisted in fixing the window and range levels
to specific values used by radiologists when inspecting lung CTs.

##  7 Conclusions

Volumetric images like CTs and MRIs provide rich information about the
internal body structure, necessary in the diagnosis of many affections. With
the advancements of neural networks, automatic diagnosis in volumetric images
became possible at high precision, useful for prioritizing patients and
assisting doctors in final decisions. After a thorough experimental analysis
of various architectures, the current paper devised an approach able to
produce highly accurate CT reports about the presence of tuberculosis related
affections. The method, based on computing predictions at slice level, has,
beside high accuracy in predicting lesion type, the advantage of offering more
information in terms of localization of the lesions. With a current mean AUC
score of 0.924 on test data, its performance can be increased if more data,
capturing various cases, is provided in the training phase.

##  8 Acknowledgements

Our strong belief is that data science without domain knowledge can never
reach its full potential. We would like to thank Mirela Iordache, an
outstanding radiologist at the Regional Institute of Oncology in Iasi, for
sharing her knowledge and valuable insights on the medical niche approached in
the competition.

## References

  * [1] Serge Kozlovski, Vitali Liauchuk, Yashin Dicente Cid, Aleh Tarasau, Vassili Kovalev, and Henning Müller.  Overview of ImageCLEFtuberculosis 2020 - automatic CT-based report generation.  In CLEF2020 Working Notes, CEUR Workshop Proceedings, Thessaloniki, Greece, September 22-25 2020. CEUR-WS.org $<$http://ceur-ws.org$>$.
  * [2] Bogdan Ionescu, Henning Müller, Renaud Péteri, Asma Ben Abacha, Vivek Datla, Sadid A. Hasan, Dina Demner-Fushman, Serge Kozlovski, Vitali Liauchuk, Yashin Dicente Cid, Vassili Kovalev, Obioma Pelka, Christoph M. Friedrich, Alba García Seco de Herrera, Van-Tu Ninh, Tu-Khiem Le, Liting Zhou, Luca Piras, Michael Riegler, Pål Halvorsen, Minh-Triet Tran, Mathias Lux, Cathal Gurrin, Duc-Tien Dang-Nguyen, Jon Chamberlain, Adrian Clark, Antonio Campello, Dimitri Fichou, Raul Berari, Paul Brie, Mihai Dogariu, Liviu Daniel Ştefan, and Mihai Gabriel Constantin.  Overview of the imageclef 2020: Multimedia retrieval in medical, lifelogging, nature, and internet applications.  In Experimental IR Meets Multilinguality, Multimodality, and Interaction, volume 12260 of Proceedings of the 11th International Conference of the CLEF Association (CLEF 2020), Thessaloniki, Greece, September 22-25 2020. LNCS Lecture Notes in Computer Science, Springer.
  * [3] Yashin Dicente Cid, Vitali Liauchuk, Dzmitri Klimuk, Aleh Tarasau, Vassili Kovalev, and Henning Müller.  Overview of imagecleftuberculosis 2019-automatic ct-based report generation and tuberculosis severity assessment.  In CLEF (Working Notes), 2019.
  * [4] Vitali Liauchuk.  Imageclef 2019: Projection-based ct image analysis for tb severity scoring and ct report generation.  In CLEF (Working Notes), 2019.
  * [5] Yashin Dicente Cid, Oscar Alfonso Jiménez del Toro, Adrien Depeursinge, and Henning Müller.  Efficient and fully automatic segmentation of the lungs in ct volumes.  In Orcun Goksel, Oscar Alfonso Jiménez del Toro, Antonio Foncubierta-Rodríguez, and Henning Müller, editors, Proceedings of the VISCERAL Anatomy Grand Challenge at the 2015 IEEE ISBI, CEUR Workshop Proceedings, pages 31–35. CEUR-WS, May 2015.
  * [6] Vitali Liauchuk and Vassili Kovalev.  Imageclef 2017: Supervoxels and co-occurrence for tuberculosis ct image classification.  In CLEF2017 Working Notes, CEUR Workshop Proceedings, Dublin, Ireland, September 11-14 2017. CEUR-WS.
  * [7] Johannes Hofmanninger, Florian Prayer, Jeanny Pan, Sebastian Rohrich, Helmut Prosch, and Georg Langs.  Automatic lung segmentation in routine imaging is a data diversity problem, not a methodology problem.  arXiv preprint arXiv:2001.11767, 2020.
  * [8] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [9] Gustavo Pérez and Pablo Arbeláez.  Automated detection of lung nodules with three-dimensional convolutional neural networks.  In 13th international conference on medical information processing and analysis, volume 10572, page 1057218. International Society for Optics and Photonics, 2017.
  * [10] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [12] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size.  arXiv preprint arXiv:1602.07360, 2016.
  * [13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [14] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [15] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [16] Saumya Jetley, Nicholas A Lord, Namhoon Lee, and Philip HS Torr.  Learn to pay attention.  arXiv preprint arXiv:1804.02391, 2018.
  * [17] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [18] Mingxing Tan and Quoc V Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  arXiv preprint arXiv:1905.11946, 2019.


# DigiTac: A DIGIT-TacTip Hybrid Tactile Sensor for Comparing
Low-Cost High-Resolution Robot Touch

Nathan F. Lepora, Yijiong Lin, Ben Money-Coomes, John Lloyd
www.lepora.com/digitac, www.github.com/nlepora/digitac-design NL and JL were
supported by a Leadership Award from the Leverhulme Trust on ‘A biomimetic
forebrain for robot touch’ (RL-2016-39).All authors are with the Department of
Engineering Mathematics and Bristol Robotics Laboratory, University of
Bristol, Bristol, U.K.Corresponding author<EMAIL_ADDRESS>website: www.lepora.com/digitacProject repository:
www.github.com/nlepora/digitac-designShared data:
www.doi.org/10.5523/bris.110f0tkyy28pa2joru2pxxbrxd

###### Abstract

Deep learning combined with high-resolution tactile sensing could lead to
highly capable dexterous robots. However, progress is slow because of the
specialist equipment and expertise. The DIGIT tactile sensor offers low-cost
entry to high-resolution touch using GelSight-type sensors. Here we customize
the DIGIT to have a 3D-printed sensing surface based on the TacTip family of
soft biomimetic optical tactile sensors. The DIGIT-TacTip (DigiTac) enables
direct comparison between these distinct tactile sensor types. For this
comparison, we introduce a tactile robot system comprising a desktop arm,
mounts and 3D-printed test objects. We use tactile servo control with a
PoseNet deep learning model to compare the DIGIT, DigiTac and TacTip for edge-
and surface-following over 3D-shapes. All three sensors performed similarly at
pose prediction, but their constructions led to differing performances at
servo control, offering guidance for researchers selecting or innovating
tactile sensors. All hardware and software for reproducing this study will be
openly released.

###### Index Terms:

Force and Tactile Sensing, Open-source Robotics

##  I INTRODUCTION

Advances in deep learning combined with innovation in high-resolution tactile
sensing give a plausible route to robots with the dexterous capabilities of
the human hand. Consider, for example, the revolution in computer vision where
1000s of researchers have ready access to deep learning infrastructure and
vast amounts of openly-shared visual data. In contrast, research on robot
touch for dexterity requires access to specialist equipment such as industrial
arms integrated with tactile sensing hardware. This need for specialised
equipment and expertise is a barrier to entering the field and reproducing
research findings. Consequently, progress is slow and narrow in scope from
just a few research laboratories.

Researchers in Meta AI have developed the DIGIT as a low-cost high-resolution
tactile sensor [1]. They have open-sourced the sensor design and in
partnership with the company GelSight commercially manufacture the sensor [2].
To lower the barrier of entry to using machine learning (ML) with tactile
sensors like the DIGIT, an accompanying library of ML models and functionality
(PyTouch) was also released [3] alongside a simulation environment (TACTO)
[4]. The overall concept is to provide a complete ecosystem for teaching
robots to perceive, understand and interact through touch [5].

For low-cost high-resolution robot touch to progress, there needs to be a
comparison of tactile sensing technologies and a greater diversity of tactile
sensing hardware, ML models and test scenarios. The TacTip is a distinct high-
resolution optical tactile sensor with a 3D-printed biomimetic structure of
papillae that amplify the movement of markers on their tips [6, 7], which also
has body of research on using ML models for tactile perception and control. In
this paper, we customize the DIGIT to have a TacTip skin to make the first
direct comparison between the GelSight and TacTip families of tactile sensors
using common software and hardware. For this comparison, we make use of recent
progress in pose-based servo control using ML models on tactile images [8, 9,
10].

Test System: Low-Cost Tactile Sensor on Desktop Arm
---
Figure 1: Left: the DIGIT-TacTip (DigiTac) mounted on an affordable desktop
robot arm (Dobot MG400, $\sim$£2k). Right: close-up of sensor and a captured
high-resolution tactile image.

Our main contributions are:
1) We introduce the DIGIT-TacTip (DigiTac) that combines the DIGIT base
(lighting, camera board, housing) with a TacTip sensing module (3D-printed
skin/mount, window, soft gel). This raises challenges in adapting a design
intended for GelSight-type sensors to the TacTip sensing mechanism. In
particular, the DIGIT is designed for lateral illumination of a flat sensing
surface, whereas the TacTip works best with a curved skin under direct
lighting.
2) We apply recent developments in pose-based tactile servo control [9, 10] to
compare the sensors on a common affordable desktop robot system (Fig. 1), by
learning to slide over the edges and surfaces of unknown 3D objects while
maintaining safe, delicate contact. This raises several challenges, including
how to apply methods developed with industrial robots to lower-capability
desktop robots, how to use and compare distinct high-resolution optical
tactile sensors with the same hardware/software infrastructure, and how to
conduct the experiments in a manner that facilitates reproducible research.

All hardware and software components including 3D-printable designs for the
DigiTac will be openly released.

(a) TacTip (round) | (b) TacTip (flat) | (c) DigiTac | (d) DIGIT | (e) TacTip (fingertip)
---|---|---|---|---
|  |  |  |
Figure 2: Comparative form factors of (a) TacTip (round tip), (b) TacTip (flat
tip), (c) DigiTac (curved TacTip skin); (d) DIGIT (flat GelSight skin) and (e)
fingertip-sized TacTip [11]. The background grid spacing is 10 mm.

##  II BACKGROUND AND RELATED WORK

A benefit and constraint of the DIGIT-TACTO-PyTouch ecosystem is the focus on
one type of high-resolution optical tactile sensor: the GelSight [12, 13]. The
GelSight images indentation of an elastomer from the surface shading on a
reflective coating illuminated by multiple internal light sources. This design
concept has diversified into a family of reflection-based optical tactile
sensors [14, 15, 16, 17] including the DIGIT [1]. The first application of
deep learning to optical tactile sensing used the GelSight [18], since leading
to many ML models including progress in integrating vision and touch [19].

Another type of high-resolution optical tactile sensor uses the morphology of
the skin to transduce contact into the motion of markers [6, 7]. The TacTip is
unique in using an array of 3D-printed biomimetic internal pins to amplify the
contact due to surface indentation and shear [7]. The design is based on human
skin that is structured around dermal papillae whose motion is signalled by
nearby mechanoreceptors [20], with neural signals found recently to match the
TacTip output [21]. The tactile images from the TacTip are well-suited to ML
models, leading to applications including tactile servo control [10, 8],
pushing manipulation [22] and sim-to-real transfer of tactile pushing, rolling
and servoing [23]. A review of these capabilities was published recently [7].

More broadly, there have been many tactile sensor designs utilizing markers
[24], with recent innovation in embedding markers within and underneath the
tactile skin, e.g. [25, 26, 27, 28]. Some GelSight-type sensors also use
markers on the reflective coating [13, 17] to enable sensitivity to shear
contact that is not apparent from imaging reflection alone.

Table I: Comparison of DIGIT, DigiTac and TacTip low-cost high-resolution tactile sensors (each is less than £100 to make). | DIGIT | DigiTac | TacTip
---|---|---|---
size $l\times w\times h$ [mm] | 36$\times$26$\times$33 | 36$\times$26$\times$39 | 48 dia.$\times$55
weight [g] | 20 | 20 | 65
sensing field [mm] | 25$\times$19 | 25$\times$19 | 40 dia.
resolution [pix] | 640$\times$480 | 640$\times$480 | 1920$\times$1080
frame rate [per sec] | 60 | 60 | 120
biomimetic markers | - | 140 | 331

All tactile sensors have their pros and cons. The GelSight and TacTip have a
common reliance on imaging an internally-illuminated skin. However, there are
differences in their elasticity, robustness, friction and sensitivity to
contact features that may make their operation in practise rather different.
Our work here aims to bring together knowledge across the field to make it
possible to trade-off aspects of these distinct tactile sensors to progress
towards tactile-enabled robot dexterity.

##  III Tactile Sensors

###  III-A Tactile Sensor Comparison

The aim of this paper is to compare low-cost high-resolution tactile sensors
from the GelSight and TacTip families. For a GelSight-type sensor, we use the
DIGIT (Fig. 2d), which has a flat sensing surface of 25$\times$19 mm and VGA
camera. We compare this with a domed TacTip (Fig. 2a) of 40 mm diameter and
full-HD camera (ELP 1080p USB module). Table I compares the hardware
properties of the sensors.

However, there is an issue with comparing the sensing capabilities of the
DIGIT and TacTip. Any differences in performance may be attributable to the
cameras or sensor shapes, rather than the tactile sensing. Therefore, it is
important to standardize the hardware and software methods as far as possible
to isolate differences due to the sensing mechanism.

How can two distinct sensors be standardized? Although the TacTip and
GelSight-type sensors are superficially similar in both relying on internal
cameras, they use different internal illumination. The DIGIT/GelSight has
lateral lighting into a thick acrylic window that acts as a light-guide to
accentuate shading from the surface indentation. The TacTip has forward-facing
lighting situated to both contrast the reflective markers against a black skin
and reduce glare on a thin internal window that holds a soft optically-clear
gel below the 3D-printed skin.

Investigations indicated that the DIGIT could be customized into a TacTip by
moving the acrylic window back from the light-source and having a rounded
sensing surface. The details of this DIGIT-TacTip customization are described
next.

###  III-B DigiTac Design and Fabrication

The design of the DIGIT optical tactile sensor is based on [1]: (i) being
sufficiently compact to fit as an array of end effectors/fingertips on multi-
fingered robot hands or arms; (ii) having a sensing surface (gel) that is
robust and easily interchanged; and (iii) easing the fabrication process by
using off-the-shelf components and a snap-fit assembly.

In practise, the DIGIT has two distinct assemblies:
(A1) The base unit comprising the internal lighting PCB, camera PCB and
housing (Fig. LABEL:fig:3C-F same as [1, Fig. 2D-G]);
(A2) The sensing surface on a thick acrylic window/light-guide and housing
that snap fits onto the base. This sensing surface based on the GelSight
family of sensors, comprising a painted elastomer, 6 mm-thick acrylic window
and snap-fit holder [1, Fig. 2A-C]. Several elastomers were trialled,
including coated markers and a transparent surface [1, Fig. 4].

Here we adapt the DIGIT sensing surface into a TacTip sensing surface by:
(D1) The plastic snap-fit holder is redesigned to support a curved compliant
skin on the underside of which are 140 2 mm-papillae tipped by markers (Fig
LABEL:fig:4). The entire piece is fabricated on a multi-material 3D-printer
(Stratasys J826), using a rubber polymer (Agilus30, black) for the
skin/papillae and acrylic resin (VeroWhite) for the holder/markers.
(D2) The acrylic window is narrowed to 1 mm thick and moved above the light
source to just below the complient skin. The original window acted as a light
guide for lateral illumination of the GelSight-type sensing surface. However,
the main function of the window for the TacTip is to contain a compliant gel
(see D3 below). A thinner window allows more direct marker illumination for a
sufficiently curved skin, while also avoiding spurious reflections and saving
weight.
(D3) The enclosure between the skin and window is filled with a soft
optically-clear gel as in other TacTip sensors [7, 6]. Here we used a silicone
gel (Techsil, Shore A Hardness 15), which was injected through a small (1.5
mm-dia.) filling hole near the retaining screw on the front of the sensing
surface. To stop the gel leaking, the filling hole is then sealed with a small
plug, which is also made by 3D-printing.

---
Figure 3: Pose-based tactile servo control with PoseNet convolutional neural
network model [10, Fig. 2][9, Fig. 4]. Table II: Tactile image data collection
for sliding contacts: top: pose parameter ranges; bottom: slide ranges to
pose. 5000 uniformly-random samples are collected per stimulus.

| pose parameter ranges
---|---
stimulus |  $x$-position |  $y$-position |  $z$-position | angle
edge | 0 mm | [$-5$,$+5$] mm | [$-1$,$+1$] mm | [$-45^{\circ}$,$+45^{\circ}$]
surface | 0 mm | [$-5$,$-1$] mm | 0 mm | [$-30^{\circ}$,$+30^{\circ}$]

| slide parameter ranges
---|---
stimulus |  $x$-position |  $y$-position |  $z$-position | angle
edge | [$-5$,$+5$] mm | [$-5$,$+5$] mm | 0 mm | [$-5^{\circ}$,$+5^{\circ}$]
surface | [$-5$,$+5$] mm | 0 mm | 0 mm | [$-5^{\circ}$,$+5^{\circ}$]

The DigiTac sensing surface is assembled in a similar way as other TacTip
sensors (see the Soft Robotics Toolkit for a step-by-step guide). The use of
multi-material 3D-printing will ease future customization of the design,
having already led to a wide range of TacTip form-factors and integrated
robotic systems (Fig 2; see also [7, Fig. 4]). There is a competitive online
marketplace for 3D-printing, making it easy to obtain the DigiTac sensing
surface from the open-sourced designs.

The DigiTac differs from the DIGIT sensor by having a thicker, curved sensing
surface (Fig. 2 and Table I), which we will see later is far softer than the
original elastomer. The DIGIT and DigiTac are sufficently small and light to
be compatible with some grippers (e.g. the Allegro hand shown in [1]).
However, they are too bulky for anthropomorphic robot hands, where a custom
TacTip (Fig. 2E) is currently the only integrated high-resolution optical
tactile sensor.

##  IV Experiments

###  IV-A Task: Tactile Servo Control on a Contour and Surface

Here we use servo control around a contour and over a surface to test and
compare the various tactile sensors. Tactile servo control is analogous to
visual servo control, with tactile data used in a feedback loop to control the
robot motion. In pose-based tactile servo control [10], the pose error
${}^{S}\\!E_{R}$ between the sensor pose ${}^{F}\\!P_{S}$ and a reference pose
${}^{F}\\!P_{R}$ (in a local feature frame $F$) drives the feedback loop (Fig.
3 top).

For high-resolution optical tactile sensors, the sensor pose ${}^{F}\\!P_{S}$
in the feature frame (here of an edge or surface) can be predicted using a
suitably-trained ‘PoseNet’ convolutional neural network on a tactile image [9]
(Fig. 3). For robust pose predictions while interacting with the object, it is
necessary to train the model to be insensitive to contact shear motion by
sliding the sensor along the object to its labelled pose [8, 9, 10].

Here we use two tactile servo control tasks adapted from the fully 3D-tasks
reported in earlier work [10] to 2D-tasks implementable on lower-capability
desktop robot arms:
Task 1: Edge-following is around horizontal flat shapes (here we use circle,
square and circular-wave objects described in [10]). The PoseNet model is
trained on a straight edge of the square block. During training, the sensor
pose is varied in its perpendicular distance from the edge and across a range
of orientations, using a horizontal sliding motion to move it into position
(Table II). Unlabelled variations in contact depth ($z$) are introduced to
train the model to be insensitive to sub-mm variations in object height due to
imperfect levelling of the robot (which is very difficult to avoid in
practise).
Task 2: Surface-following is carried out around the walls of the shapes
described above with the tactile sensor oriented so the sensing surface is
vertical. The PoseNet model is trained on a straight vertical wall of the
square block. The labelled pose is varied in distance, normal to the surface,
and over a range of surface angles, following a 2D sliding motion tangential
to the surface (Table II). A subtlety for servoing over surfaces, is that the
tool centre point (TCP) needs adjusting so the $z$-axis of the end-effector
frame coincides with the tip of the sensor.

The PoseNet models are trained on 75% of 5000 random poses (uniformly
distributed), with test accuracies reported on the remaining 25%. Network
hyperparameters are from ref. [10, Table II] without further tuning, except
the input layers were changed to correspond to 160$\times$120 pixel subsampled
tactile images for the DIGIT and DigiTac and 128$\times$128 pixels for the
TacTip, with the latter two binarized using adaptive thresholding. Networks
were trained over 100 epochs ($\sim$30 minutes on a standard PC with 6Gb GPU).

The tactile sensors are compared using the model performance on the edge and
surface test data and the deviation of the controlled trajectories from the
known object perimeters. These tests also probe the model generalization for
each sensor to curved edges and surfaces not experienced in training.

###  IV-B Low-cost test platform

---
Figure 4: Schematic of test platform: A) desktop robot arm; B) tactile sensor
on mount; C) 3D-printed stimulus on mount; D) robot mounting plate; E) base
plate with mounting grid.

To aid in making the robot test platform accessible to other laboratories, we
use a low-cost desktop robot along with laser-cut and 3D-printed accessories
for all comparison experiments.

To give some context, typically the hardware and software infrastructure
required for researching robot touch has been:
(I1) Robot manipulators, such as ABB and Universal Robotics industrial robot
arms. These are highly capable and accurate (e.g. ABB-IRB120 repeatability
$\pm 0.01$ mm), but are large, expensive items of equipment that must be
installed and operated safely in a laboratory or industrial setting.
(I2) Software libraries for capturing tactile data, processing with ML models
and controlling the robots in real-time. Different laboratories have various
solutions. In Bristol Robotics Laboratory, we have developed Python libraries,
VSP (Visual Stream Processor) and CRI (Common Robot Interface) for
asynchronous tactile image capture and interfacing with proprietary robot
APIs, available on GitHub.

Here we introduce a test platform to ease the setup and repeatability of
tactile experiments using a low-cost safe alternative to industrial robot arms
(Fig. 4):
(P1) Desktop robot arm, for which we use a Dobot MG400 4-axis arm designed for
affordable automation. The base and control unit has footprint 190
mm$\times$190 mm, payload 750 g, maximum reach 440 mm and repeatability $\pm
0.05\,$mm. As we describe later, the accuracy of tactile models trained using
this arm is similar to larger industrial robot arms. The main constraint is
that only the $(x,y,z)$-position and rotation around the $z$-axis of the end
effector are actuated.
(P2) Base plate with a mounting plate for the desktop robot and a grid of
holes for mounting stimuli in precise locations relative to the robot. The
base plate was laser cut from an acrylic sheet of size
600$\times$400$\times$10 mm, with 25 mm grid-spacing. The robot mounting plate
was also laser cut from 10 mm-thick acrylic then screwed to the base plate.
(P3) Normal and right-angle mounts for the DIGIT and TacTip sensors so they
can be used as end effectors. These were 3D-printed from designs adapted from
an end-flange supplied with the MG400. A choice of mounts allows the tactile
sensing surface to be oriented in a horizontal or vertical plane, as the end
effector can rotate only around a vertical axis.
(P4) Test stimuli with mounts to attach to the base plate. Here we use
circular, square and circular-wave shapes of $\sim$100 mm diameter with 30 mm
walls, 3D-printed in ABS.

##  V RESULTS

(a) DIGIT | (b) DigiTac | (c) TacTip
---|---|---
|  |
|  |
(d) RGB image | (e) BW image | (f) BW image
Figure 5: Top: views of the DIGIT, DigiTac and TacTip (round) pressing on an
edge. Bottom: corresponding tactile images.

###  V-A Appraisal of DIGIT, DigiTac and TacTip

For an initial qualitative comparison of the three optical tactile sensors, we
pressed them against a 3D-printed straight edge and captured the tactile
images (Fig. 5).

From the external appearance of the deformation, the Digi-Tac and TacTip (Figs
5b,c) are much softer than the DIGIT (Fig. 5a), with a large compression of
several millimeters over the contact region and a corresponding bulging of
part of the sensing surface not in contact. In contrast, the DIGIT sensing
surface is fairly stiff with sub-mm deformation. The maximum compression of
the DigiTac or TacTip is about 6-8 mm before damage will occur and for the
DIGIT about 1-2 mm.

Another difference is that the DIGIT sensing surface has higher friction than
the DigiTac. It is fairly easy to slide a DigiTac or TacTip over a 3D-printed
flat surface with slight texture, whereas the DIGIT tends to stick then slip.

From the tactile images, the edge is clearly visible with the DIGIT as a
shaded line at the point of contact (Fig. 5d). For the DigiTac and TacTip, the
edge can be located by where the markers are more widely spaced (Figs 5e,f).
Thus, it is more difficult to see edge location by eye; however, we will see
later that the edge can be precisely localized by using a convolutional neural
network on the entire tactile image.

The DIGIT is sensitive enough for the texture of the 3D-print to be just
visible (Fig 5d, right of edge). This information is absent from static touch
with the DigiTac/TacTip. In this respect, the DIGIT is superior to the human
sense of touch, which does not have the sensitivity to feel that the 3D-print
is textured by statically holding a fingertip on the surface.

###  V-B Sensor Comparison using Edge Following

Next, we compare the tactile sensors on predicting edge position and angle
during sliding contacts, then use this pose model for tactile servo control
around the edges of three test objects: a circular disk, square block and
circular-wave.

For offline edge-pose prediction on the test data, both the DigiTac and TacTip
are similarly accurate with edge position error 0.16 mm and angle error
$\sim$1.6∘ (Table LABEL:tab:3). These errors are $\sim$1.5% of the ranges (10
mm and 90∘), with little scatter when the predictions are plotted against
ground truths (Fig. LABEL:fig:8). It appears the smaller size of the DigiTac
and fewer markers (Table I) does not affect its capability for predicting edge
pose.

In comparison, the DIGIT has similar edge position error (0.19 mm vs 0.16 mm)
and angle (1.82∘ vs $\sim$1.6∘) compared with the DigiTac. To isolate the
effects of sensor surface material and geometry from subsequent processing, we
used the same neural network architectures and hyperparameters with binary
tactile images for the DigiTac and shaded images for the DIGIT. While these
results could change with a different choices of neural networks, all
accuracies are sub-mm and around 1-2 degrees, which are appropriate for the
contour-following experiments reported below.

When applied to edge following on the three test shapes, the DIGIT, DigiTac
and TacTip all accurately traced around the shapes to sub-mm accuracy (Fig. 6,
two left columns). For all three sensors and all three shapes, the mean
absolute position errors were mainly sub-millimeter (Table III, edge stimuli),
with the TacTip the most accurate (0.4-0.5 mm), followed by both the DIGIT and
DigiTac (0.6-1.2 mm). Overall, the circular wave is most demanding, followed
by the square (because of the corners) and the circle is the easiest. All
position errors are larger than the test predictions ($\sim\,$0.15 mm), as
expected because the servo control task introduces factors such as the corners
on the square and curves of the circular-wave that differ from the straight
edges used in training.

The angle prediction performance showed more variability in how accurately the
sensor maintains normality to the edge during servoing (Fig. 6; Table III).
For just the circle and square, the TacTip has the lowest angle error
($\lesssim$5∘), followed by both the DIGIT and DigiTac (10-15∘), which we
attribute to the larger domed TacTip behaving better under shear. All sensors
were inaccurate on the circular-wave (20-30∘), because of the difficulty of
contour following around the shape.

DIGIT: Edge Following |
---|---
|
|
|
DigiTac: Edge Following | DigiTac: Surface Following
|
|
|
TacTip: Edge following | TacTip: Surface following
|
|
|
Figure 6: Edge and surface following with the DIGIT, DigiTac and TacTip for the disk, square and circular-wave shapes.  Table III: Accuracy of edge and surface following for the DIGIT, DigiTac and TacTip. sensor | stimulus | shape | position MAE | angle MAE
---|---|---|---|---
DIGIT | edge | circle | 0.6 mm | 10.7∘
| edge | square | 0.9 mm | 12.7∘
| edge | circular-wave | 1.2 mm | 29.7∘
DigiTac | edge | circle | 0.6 mm | 15.1∘
| edge | square | 1.0 mm | 15.0∘
| edge | circular-wave | 1.1 mm | 26.2∘
| surface | circle | 0.7 mm | 12.9∘
| surface | square | 1.5 mm | 10.2∘
| surface | circular-wave | 1.0 mm | 27.9∘
TacTip | edge | circle | 0.4 mm | 1.6∘
| edge | square | 0.5 mm | 4.6∘
| edge | circular-wave | 0.4 mm | 19.6∘
| surface | circle | 0.6 mm | 15.4∘
| surface | square | 1.5 mm | 14.7∘
| surface | circular-wave | 1.6 mm | 23.0∘

###  V-C Sensor Comparison using Surface Following

The final comparison of the tactile sensors is on predicting surface angle and
contact depth during sliding contacts, which is then used for tactile servo
control around the vertical walls of the same three test objects used above.

For offline surface-pose prediction on the test data, both the DigiTac and
TacTip accurately predict surface pose, with the DigiTac giving the best
contact depth error (0.06 mm) and angle error (0.5∘) (Table LABEL:tab:3).
These correspond to 1.5% and $\sim$1% of the ranges (4 mm and 60∘), with
little scatter on the prediction vs ground truth plots (Fig. LABEL:fig:8). The
accuracy in contact depth is similar to the repeatability (0.05 mm) of the
desktop robot, which may be a limiting factor.

For the DIGIT, the angle error (2.47∘) and contact depth error (0.22 mm) are
high compared to those for DigiTac and TacTip (Table LABEL:tab:3). It was not
possible to successfully follow a surface with this model. We attribute this
difficulty as due to the much stiffer sensing surface, which allowed only a
narrow (1 mm) range of training depths and during surface following left
little tolerance to errors during the servo control.

When applied to surface following on the three test shapes, both the DigiTac
and TacTip had similar accuracy (0.6-1.5 mm), with the circle the most
accurately traced at $\sim$0.5 mm error (Fig. 6, Table III). Both sensors
struggled to turn the corners of the square, and for the DigiTac we needed to
advance the angular set point to 5∘ for the task to complete. Like edge-
following, the circular-wave was the most demanding shape to accurately trace.
The angle errors were similar for the DigiTac/TacTip, being least for the
circle and square (10∘-15∘) and largest on the circular-disk ($\sim$25∘).

##  VI Discussion

In this paper, we compared low-cost high-resolution tactile sensors from the
GelSight and TacTip families, by adapting the DIGIT (GelSight) to have a
TacTip sensing surface, The DIGIT-TacTip, or DigiTac allowed comparison of
these two tactile sensor types with the same hardware and software.

Overall, the TacTip, DigiTac and DIGIT optical tactile sensors all performed
well at pose prediction and pose-based tactile servo control, with pose-
prediction accuracy of $\sim\,$0.1 mm and $\sim$1-2∘, and servo control
accuracy typically better than 1 mm. There was some biasing of contact angle
during the servo control tasks, mainly for the DIGIT and DigiTac. However,
they still performed accurately at edge and surface following, except the
DIGIT was not suited for sliding over surfaces because of its relatively flat
and stiff elastomer.

The most significant difference between the sensors is their material
properties and construction. The DIGIT sensing surface is flat and fairly
inelastic compared to the soft curved sensing surface of the TacTip. GelSight-
type sensors have flat sensing surfaces comprised of a molded elastomer
illuminated laterally, which are highly effective at imaging fine surface
detail. In contrast, TacTip-type sensors come in various shapes and sizes
(Fig. 2) with a flexible 3D-printed skin and compliance from a soft gel. For
the servo control tasks considered here, a soft curved tactile sensor was more
practical to fit into curved surface and had greater tolerance for safe
contact.

The robustness of the sensors also depends on their material properties and
construction, which is important because touch involves contact with the
potential for damage. Our tests were demanding because: (i) thousands of
sliding contacts were needed to train the pose-prediction models; (ii) surface
and edge following shears the sensor surface as it rubs against the object;
(iii) surface following can also lead to collisions with the test object.
During the tests, we broke one DigiTac by ripping the skin where it joins the
housing and one DIGIT by shearing off the elastomer from a collision.

Because our aim in comparing tactile sensor technologies is to encourage
innovation in the field, we will open-source the DigiTac sensing surface and
software infrastructure including the robot interface and libraries for the
experiments. We will also release all data and models used in this paper. As a
final comment, we encourage others to adopt a similarly open perspective, and
look forward to seeing further comparison and open release of other low-cost
optical tactile sensors.

Acknowledgements: We thank Mike Lambeta and Roberto Calandra for donating the
DIGIT sensors, and Stephen Redmond for discussions including suggesting the
name ‘DigiTac’.

## References

  * [1] M. Lambeta, P. Chou, S. Tian, B. Yang, B. Maloon, V. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, D. Jayaraman, and R. Calandra.  DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation.  IEEE Robotics and Automation Letters, 5(3):3838–3845, July 2020\.
  * [2] DIGIT.  http://www.digit.ml/.
  * [3] M. Lambeta, H. Xu, J. Xu, P. Chou, S. Wang, T. Darrell, and R. Calandra.  PyTouch: A Machine Learning Library for Touch Processing.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13208–13214, Xi’an, China, May 2021.
  * [4] S. Wang, M. Lambeta, P. Chou, and R. Calandra.  TACTO: A Fast, Flexible, and Open-Source Simulator for High-Resolution Vision-Based Tactile Sensors.  IEEE Robotics and Automation Letters, 7(2):3930–3937, April 2022\.
  * [5] Teaching robots to perceive, understand, and interact through touch.  https://ai.facebook.com/blog/teaching-robots-to-perceive-understand-and-interact-through-touch/.
  * [6] B. Ward-Cherrier, N. Pestell, L. Cramphorn, B. Winstone, M. E. Giannaccini, J. Rossiter, and N. Lepora.  The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic Morphologies.  Soft Robotics, 5(2):216–227, 2018.
  * [7] N. Lepora.  Soft Biomimetic Optical Tactile Sensing With the TacTip: A Review.  IEEE Sensors Journal, 21(19):21131–21143, October 2021.
  * [8] N. Lepora, A. Church, C. de Kerckhove, R. Hadsell, and J. Lloyd.  From Pixels to Percepts: Highly Robust Edge Perception and Contour Following Using Deep Learning and an Optical Biomimetic Tactile Sensor.  IEEE Robotics and Automation Letters, 4(2):2101–2107, 2019.
  * [9] N. Lepora and J. Lloyd.  Optimal Deep Learning for Robot Touch: Training Accurate Pose Models of 3D Surfaces and Edges.  IEEE Robotics & Automation Magazine, 27(2):66–77, June 2020.
  * [10] N. Lepora and J. Lloyd.  Pose-Based Tactile Servoing: Controlled Soft Touch Using Deep Learning.  IEEE Robotics & Automation Magazine, 28(4):43–55, December 2021\.
  * [11] N. Lepora, A. Stinchcombe, C. Ford, A. Brown, J. Lloyd, M. Catalano, M. Bianchi, and B. Ward-Cherrier.  Towards integrated tactile sensorimotor control in anthropomorphic soft robotic hands.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an China, May 2021.
  * [12] M. K. Johnson and E. H. Adelson.  Retrographic sensing for the measurement of surface texture and shape.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 1070–1077, 2009.
  * [13] W. Yuan, S. Dong, and E. Adelson.  GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force.  Sensors, 17(12):2762, 2017.
  * [14] D. Gomes, Z. Lin, and S. Luo.  GelTip: A Finger-shaped Optical Tactile Sensor for Robotic Manipulation.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9903–9909, Las Vegas, NV, USA, October 2020.
  * [15] B. Romero, F. Veiga, and E. Adelson.  Soft, Round, High Resolution Tactile Fingertip Sensors for Dexterous Robotic Manipulation.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4796–4802, Paris, France, May 2020.
  * [16] A. Padmanabha, F. Ebert, S. Tian, R. Calandra, C. Finn, and S. Levine.  OmniTact: A Multi-Directional High-Resolution Touch Sensor.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 618–624, Paris, France, May 2020.
  * [17] I. Taylor, S. Dong, and A. Rodriguez.  GelSlim3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing Finger.  In 2022 IEEE International Conference on Robotics and Automation (ICRA), pages 10781–10787, Philadelphia, PA, USA, May 2022\.
  * [18] W. Yuan, C. Zhu, A. Owens, M. Srinivasan, and E. Adelson.  Shape-independent hardness estimation using deep learning and a GelSight tactile sensor.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 951–958, Singapore, Singapore, May 2017.
  * [19] S. Luo, N. Lepora, U. Martinez-Hernandez, Joao Bimbo, and H. Liu.  Editorial: ViTac: Integrating Vision and Touch for Multimodal and Cross-Modal Perception.  Frontiers in Robotics and AI, 8, 2021.
  * [20] C. Chorley, C. Melhuish, T. Pipe, and J. Rossiter.  Development of a tactile sensor based on biologically inspired edge encoding.  In International Conference on Advanced Robotics, pages 1–6, 2009.
  * [21] N. Pestell, T. Griffith, and N. Lepora.  Artificial SA-I and RA-I afferents for tactile sensing of ridges and gratings.  Journal of The Royal Society Interface, 19(189):20210822, April 2022\.
  * [22] J. Lloyd and N. Lepora.  Goal-Driven Robotic Pushing Using Tactile and Proprioceptive Feedback.  IEEE Transactions on Robotics, 38(2):1201–1212, April 2022.
  * [23] A. Church, J. Lloyd, R. Hadsell, and N. Lepora.  Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation.  In Proceedings of the 5th Conference on Robot Learning, pages 1645–1654. PMLR, October 2021.
  * [24] K. Kamiyama, H. Kajimoto, Naoki Kawakami, and Susumu Tachi.  Evaluation of a vision-based tactile sensor.  In IEEE International Conference on Robotics and Automation (ICRA), pages 1542–1547 Vol.2, New Orleans, LA, USA, 2004\.
  * [25] G. Zhang, Y. Du, H. Yu, and M. Wang.  DelTact: A Vision-based Tactile Sensor Using Dense Color Pattern.  arXiv preprint arXiv:2202.02179, February 2022.
  * [26] X. Lin and M. Wiertlewski.  Sensing the Frictional State of a Robotic Skin via Subtractive Color Mixing.  IEEE Robotics and Automation Letters, 4(3):2386–2392, July 2019\.
  * [27] W. Li, J. Konstantinova, Y. Noh, Z. Ma, A. Alomainy, and K. Althoefer.  An Elastomer-based Flexible Optical Force and Tactile Sensor.  In 2019 2nd IEEE International Conference on Soft Robotics (RoboSoft), pages 361–366, Seoul, Korea (South), April 2019\.
  * [28] C. Sferrazza and R. D’Andrea.  Design, Motivation and Evaluation of a Full-Resolution Optical Tactile Sensor.  Sensors, 19(4):928, February 2019.