* [97] Pietzonka, P. & Seifert, U.  Universal trade-off between power, efficiency, and constancy in steady-state heat engines.  _Phys. Rev. Lett._ 120, 190602 (2018).
  * [98] Pietzonka, P., Barato, A. C. & Seifert, U.  Universal bound on the efficiency of molecular motors.  _J. Stat. Mech._ 2016, 124004 (2016).  URL https://doi.org/10.1088/1742-5468/2016/12/124004.
  * [99] Barato, A. C. & Seifert, U.  Cost and precision of Brownian clocks.  _Phys. Rev. X_ 6, 041053 (2016).  URL https://doi.org/10.1103/PhysRevX.6.041053.
  * [100] Kheradsoud, S. _et al._ Power, efficiency and fluctuations in a quantum point contact as steady-state thermoelectric heat engine.  _Entropy_ 21, 777 (2019).  URL https://www.mdpi.com/1099-4300/21/8/777.
  * [101] Koyuk, T., Seifert, U. & Pietzonka, P.  A generalization of the thermodynamic uncertainty relation to periodically driven systems.  _J. Phys. A: Math. Theor._ 52, 02LT02 (2018).  URL https://doi.org/10.1088/1751-8121/aaeec4.
  * [102] Miller, H. J. D., Mohammady, M. H., Perarnau-Llobet, M. & Guarnieri, G.  Thermodynamic uncertainty relation in slowly driven quantum heat engines.  _Phys. Rev. Lett._ 126, 210603 (2021).  URL https://link.aps.org/doi/10.1103/PhysRevLett.126.210603.
  * [103] Miller, H. J. D., Mohammady, M. H., Perarnau-Llobet, M. & Guarnieri, G.  Joint statistics of work and entropy production along quantum trajectories.  _Phys. Rev. E_ 103, 052138 (2021).  URL https://link.aps.org/doi/10.1103/PhysRevE.103.052138.
  * [104] Koyuk, T. & Seifert, U.  Operationally accessible bounds on fluctuations and entropy production in periodically driven systems.  _Phys. Rev. Lett._ 122, 230601 (2019).  URL https://link.aps.org/doi/10.1103/PhysRevLett.122.230601.
  * [105] Harunari, P. E., Fiore, C. E. & Proesmans, K.  Exact statistics and thermodynamic uncertainty relations for a periodically driven electron pump.  _J. Phys. A: Math. Theor._ 53, 374001 (2020).  URL https://doi.org/10.1088/1751-8121/aba05e.
  * [106] Speck, T. & Seifert, U.  The Jarzynski relation, fluctuation theorems, and stochastic thermodynamics for non-Markovian processes.  _J. Stat. Mech._ 2007, L09002–L09002 (2007).  URL https://doi.org/10.1088/1742-5468/2007/09/l09002.
  * [107] Li, J., Horowitz, J. M., Gingrich, T. R. & Fakhri, N.  Quantifying dissipation using fluctuating currents.  _Nat. Commun._ 10, 1666 (2019).  URL https://doi.org/10.1038/s41467-019-09631-x.
  * [108] Manikandan, S. K., Gupta, D. & Krishnamurthy, S.  Inferring entropy production from short experiments.  _Phys. Rev. Lett._ 124, 120603 (2020).  URL https://link.aps.org/doi/10.1103/PhysRevLett.124.120603.
  * [109] Skinner, D. J. & Dunkel, J.  Improved bounds on entropy production in living systems.  _Proc. Natl. Acad. Sci. U.S.A._ 118 (2021).  URL https://doi.org/10.1073/pnas.2024300118.
  * [110] Schuler, S., Speck, T., Tietz, C., Wrachtrup, J. & Seifert, U.  Experimental test of the fluctuation theorem for a driven two-level system with time-dependent rates.  _Phys. Rev. Lett._ 94, 180602 (2005).  URL https://doi.org/10.1103/PhysRevLett.94.180602.
  * [111] Tietz, C., Schuler, S., Speck, T., Seifert, U. & Wrachtrup, J.  Measurement of stochastic entropy production.  _Phys. Rev. Lett._ 97, 050602 (2006).  URL https://doi.org/10.1103/PhysRevLett.97.050602.
  * [112] Küng, B. _et al._ Irreversibility on the level of single-electron tunneling.  _Phys. Rev. X_ 2, 011001 (2012).  URL https://link.aps.org/doi/10.1103/PhysRevX.2.011001.
  * [113] Koski, J. V. _et al._ Distribution of entropy production in a single-electron box.  _Nat. Phys._ 9, 644–648 (2013).  URL https://doi.org/10.1038/nphys2711.
  * [114] Jop, P., Petrosyan, A. & Ciliberto, S.  Work and dissipation fluctuations near the stochastic resonance of a colloidal particle.  _Europhys. Lett._ 81, 50005 (2008).  URL https://doi.org/10.1209/0295-5075/81/50005.
  * [115] Ritort, F.  _Nonequilibrium Fluctuations in Small Systems: From Physics to Biology_ , chap. 2, 31–123 (John Wiley & Sons, Ltd, 2007).  URL https://doi.org/10.1002/9780470238080.ch2.
  * [116] Chvosta, P., Reineker, P. & Schulz, M.  Probability distribution of work done on a two-level system during a nonequilibrium isothermal process.  _Phys. Rev. E_ 75, 041124 (2007).  URL https://link.aps.org/doi/10.1103/PhysRevE.75.041124.
  * [117] Chvosta, P., Einax, M., Holubec, V., Ryabov, A. & Maass, P.  Energetics and performance of a microscopic heat engine based on exact calculations of work and heat distributions.  _J. Stat. Mech._ 2010, P03002 (2010).  URL https://doi.org/10.1088/1742-5468/2010/03/p03002.
  * [118] Verley, G., Van den Broeck, C. & Esposito, M.  Modulated two-level system: Exact work statistics.  _Phys. Rev. E_ 88, 032137 (2013).  URL https://link.aps.org/doi/10.1103/PhysRevE.88.032137.
  * [119] Barato, A. C. & Chetrite, R.  Current fluctuations in periodically driven systems.  _J. Stat. Mech._ 2018, 053207 (2018).  URL https://doi.org/10.1088/1742-5468/aabfc5.
  * [120] Mandaiya, A. & Khaymovich, I. M.  Time-reversal symmetric crooks and Gallavotti-Cohen fluctuation relations in driven classical Markovian systems.  _J. Stat. Mech._ 2019, 054006 (2019).  URL https://doi.org/10.1088/1742-5468/ab11c1.
  * [121] Salazar, D. S. P.  Work distribution in thermal processes.  _Phys. Rev. E_ 101, 030101 (2020).  URL https://link.aps.org/doi/10.1103/PhysRevE.101.030101.
  * [122] Chvosta, P., Lips, D., Holubec, V., Ryabov, A. & Maass, P.  Statistics of work performed by optical tweezers with general time-variation of their stiffness.  _J. Phys. A: Math. Theor._ 53, 275001 (2020).  URL https://doi.org/10.1088/1751-8121/ab95c2.
  * [123] Slater, L. J.  _Confluent Hypergeometric Functions_ (Cambridge University Press, Cambridge, 1960).
  * [124] Trepagnier, E. H. _et al._ Experimental test of Hatano and Sasa’s nonequilibrium steady-state equality.  _Proc. Natl. Acad. Sci. U.S.A._ 101, 15038 (2004).  URL https://doi.org/10.1073/pnas.0406405101.
  * [125] Carberry, D. M. _et al._ Fluctuations and irreversibility: An experimental demonstration of a second-law-like theorem using a colloidal particle held in an optical trap.  _Phys. Rev. Lett._ 92, 140601 (2004).  URL https://doi.org/10.1103/PhysRevLett.92.140601.
  * [126] Carberry, D. M., Baker, M. A. B., Wang, G. M., Sevick, E. M. & Evans, D. J.  An optical trap experiment to demonstrate fluctuation theorems in viscoelastic media.  _J. Optics A_ 9, S204 (2007).  URL http://stacks.iop.org/1464-4258/9/i=8/a=S13.
  * [127] Andrieux, D. _et al._ Entropy production and time asymmetry in nonequilibrium fluctuations.  _Phys. Rev. Lett._ 98, 150601 (2007).  URL https://doi.org/10.1103/PhysRevLett.98.150601.
  * [128] Khan, M. & Sood, A. K.  Irreversibility-to-reversibility crossover in transient response of an optically trapped particle.  _Europhys. Lett._ 94, 60003 (2011).  URL http://stacks.iop.org/0295-5075/94/i=6/a=60003.
  * [129] Mestres, P., Martinez, I. A., Ortiz-Ambriz, A., Rica, R. A. & Roldan, E.  Realization of nonequilibrium thermodynamic processes using external colored noise.  _Phys. Rev. E_ 90, 032116 (2014).
  * [130] van Zon, R. & Cohen, E. G. D.  Stationary and transient work-fluctuation theorems for a dragged Brownian particle.  _Phys. Rev. E_ 67, 046102 (2003).  URL https://doi.org/10.1103/PhysRevE.67.046102.
  * [131] van Zon, R. & Cohen, E. G. D.  Extended heat-fluctuation theorems for a system with deterministic and stochastic forces.  _Phys. Rev. E_ 69, 056121 (2004).  URL https://doi.org/10.1103/PhysRevE.69.056121.
  * [132] Cohen, E. G. D.  Properties of nonequilibrium steady states: A path integral approach.  _J. Stat. Mech._ 2008, P07014 (2008).  URL https://doi.org/10.1088/1742-5468/2008/07/p07014.
  * [133] Nickelsen, D. & Engel, A.  Asymptotics of work distributions: The pre-exponential factor.  _Eur. Phys. J. B_ 82, 207 (2011).  URL https://doi.org/10.1140/epjb/e2011-20133-y.
  * [134] Subaşi, Y. & Jarzynski, C.  Microcanonical work and fluctuation relations for an open system: An exactly solvable model.  _Phys. Rev. E_ 88, 042136 (2013).  URL https://doi.org/10.1103/PhysRevE.88.042136.
  * [135] Kim, K., Kwon, C. & Park, H.  Heat fluctuations and initial ensembles.  _Phys. Rev. E_ 90, 032117 (2014).  URL https://doi.org/10.1103/PhysRevE.90.032117.
  * [136] Kwon, C., Noh, J. D. & Park, H.  Nonequilibrium fluctuations for linear diffusion dynamics.  _Phys. Rev. E_ 83, 061145 (2011).  URL https://doi.org/10.1103/PhysRevE.83.061145.
  * [137] Holubec, V. & Ryabov, A.  Work and power fluctuations in a critical heat engine.  _Phys. Rev. E_ 96, 030102 (2017).  URL https://doi.org/10.1103/PhysRevE.96.030102.
  * [138] Hoppenau, J. & Engel, A.  On the work distribution in quasi-static processes.  _J. Stat. Mech._ 2013, P06004 (2013).
  * [139] Engel, A.  Asymptotics of work distributions in nonequilibrium systems.  _Phys. Rev. E_ 80, 021120 (2009).
  * [140] Noh, J. D., Kwon, C. & Park, H.  Multiple dynamic transitions in nonequilibrium work fluctuations.  _Phys. Rev. Lett._ 111, 130601 (2013).
  * [141] Holubec, V., Lips, D., Ryabov, A., Chvosta, P. & Maass, P.  On asymptotic behavior of work distributions for driven Brownian motion.  _Eur. Phys. J. B_ 88, 340 (2015).  URL https://doi.org/10.1140/epjb/e2015-60635-x.
  * [142] Manikandan, S. K. & Krishnamurthy, S.  Asymptotics of work distributions in a stochastically driven system.  _Eur. Phys. J. B_ 90, 258 (2017).  URL https://doi.org/10.1140/epjb/e2017-80432-9.
  * [143] Deza, R. R., Izús, G. G. & Wio, H. S.  Fluctuation theorems from non-equilibrium Onsager-Machlup theory for a Brownian particle in a time-dependent harmonic potential.  _Centr. Eur. J. Phys._ 7, 472 (2009).
  * [144] Campisi, M. & Fazio, R.  The power of a critical heat engine.  _Nat. Commun._ 7, 11895 (2016).  URL https://doi.org/10.1038/ncomms11895.
  * [145] Koza, Z.  General technique of calculating the drift velocity and diffusion coefficient in arbitrary periodic systems.  _J. Phys. A: Math. Gen._ 32, 7637–7651 (1999).  URL https://doi.org/10.1088/0305-4470/32/44/303.
  * [146] Lips, D., Ryabov, A. & Maass, P.  Brownian asymmetric simple exclusion process.  _Phys. Rev. Lett._ 121, 160601 (2018).  URL https://link.aps.org/doi/10.1103/PhysRevLett.121.160601.
  * [147] Krishnamurthy, S., Ghosh, S., Chatterji, D., Ganapathy, R. & Sood, A. K.  A micrometre-sized heat engine operating between bacterial reservoirs.  _Nature Physics_ 12, 1134–1138 (2016).  URL https://doi.org/10.1038/nphys3870.
  * [148] Pietzonka, P., Fodor, E., Lohrmann, C., Cates, M. E. & Seifert, U.  Autonomous engines driven by active matter: Energetics and design principles.  _Phys. Rev. X_ 9, 041032 (2019).  URL https://link.aps.org/doi/10.1103/PhysRevX.9.041032.
  * [149] Holubec, V. & Marathe, R.  Underdamped active Brownian heat engine.  _Phys. Rev. E_ 102, 060101 (2020).  URL https://link.aps.org/doi/10.1103/PhysRevE.102.060101.
  * [150] Ekeh, T., Cates, M. E. & Fodor, E.  Thermodynamic cycles with active matter.  _Phys. Rev. E_ 102, 010101 (2020).  URL https://link.aps.org/doi/10.1103/PhysRevE.102.010101.
  * [151] Fodor, É. & Cates, M. E.  Active engines: Thermodynamics moves forward.  _EPL (Europhysics Letters)_ 134, 10003 (2021).  URL https://doi.org/10.1209/0295-5075/134/10003.
  * [152] Agarwalla, B. K. & Segal, D.  Assessing the validity of the thermodynamic uncertainty relation in quantum systems.  _Phys. Rev. B_ 98, 155438 (2018).  URL https://link.aps.org/doi/10.1103/PhysRevB.98.155438.


∎

11institutetext: T. Hassan ($\star$Corresponding Author) 22institutetext:
Center for Cyber-Physical Systems (C2PS), Department of Electrical Engineering
and Computer Sciences, Khalifa University, Abu Dhabi, United Arab Emirates.
22email<EMAIL_ADDRESS>33institutetext: S. Akçay 44institutetext:
Department of Computer Sciences, Durham University, Durham, United Kingdom.
55institutetext: M. Bennamoun 66institutetext: Department of Computer Science
and Software Engineering, The University of Western Australia, Perth,
Australia. 77institutetext: S. Khan 88institutetext: Mohamed bin Zayed
University of Artificial Intelligence, Abu Dhabi, United Arab Emirates.
99institutetext: N. Werghi 1010institutetext: Center for Cyber-Physical
Systems (C2PS), Department of Electrical Engineering and Computer Sciences,
Khalifa University, Abu Dhabi, United Arab Emirates.

# Tensor Pooling Driven Instance Segmentation Framework for Baggage Threat
Recognition

Taimur Hassan$\star$ Samet Akçay  Mohammed Bennamoun  Salman Khan  Naoufel
Werghi

(Received: date / Accepted: date)

###### Abstract

Automated systems designed for screening contraband items from the X-ray
imagery are still facing difficulties with high clutter, concealment, and
extreme occlusion. In this paper, we addressed this challenge using a novel
multi-scale contour instance segmentation framework that effectively
identifies the cluttered contraband data within the baggage X-ray scans.
Unlike standard models that employ region-based or keypoint-based techniques
to generate multiple boxes around objects, we propose to derive proposals
according to the hierarchy of the regions defined by the contours. The
proposed framework is rigorously validated on three public datasets, dubbed
GDXray, SIXray, and OPIXray, where it outperforms the state-of-the-art methods
by achieving the mean average precision score of 0.9779, 0.9614, and 0.8396,
respectively. Furthermore, to the best of our knowledge, this is the first
contour instance segmentation framework that leverages multi-scale information
to recognize cluttered and concealed contraband data from the colored and
grayscale security X-ray imagery.

###### Keywords:

Aviation Security Structure Tensors Instance Segmentation Baggage X-ray Scans.

††journal: Neural Computing and Applications

##  1 Introduction

X-ray imagery is a widely used modality for non-destructive testing
Tang2020TII , especially for screening illegal and smuggled items at airports,
cargoes, and malls. Manual baggage inspection is a tiring task and susceptible
to errors caused due to exhausting work routines and less experienced
personnel. Initial systems proposed to address these problems employed
conventional machine learning bastan2013BMVC . Driven by hand-engineered
features, these methods are only applicable to limited data and confined
environmental settings turcsany2013improving . Recently, attention has turned
to deep learning methods, which gave a neat boost in accuracy and
generalization capacity towards screening prohibited baggage items Hu2020ACCV
; akccay2016transfer . However, deep learning methods are also prone to
clutter, and occlusion akcay2018using . This limitation emanates from the
proposal generation strategies which have been designed for the color images
gaus2019evaluation . Unlike RGB scans, X-ray imagery lack texture and exhibit
low-intensity variations between cluttered objects. This intrinsic difference
makes the region-based or anchor-based proposal generation methods such as
Mask R-CNN maskrcnn , Faster R-CNN fasterrcnn , RetinaNet retinanet , and YOLO
yolov3 less robust for detecting the cluttered contraband data akcay2018using
. Moreover, the problem is further accentuated by the class imbalance nature
of the contraband items in the real-world gaus2019evaluation . Despite the
considerate strategies proposed to alleviate the occlusion and the imbalance
nature opixray ; miao2019sixray , recognizing threatening objects in highly
cluttered and concealed scenarios is still an open problem ackay2020 .

###  1.1 Contributions

In this paper, we propose a novel multi-scale contour instance segmentation
framework for identifying suspicious items using X-ray scans. Unlike standard
models that employ region-based or keypoint-based techniques to generate
multiple boxes around objects akccay2016transfer ; akcay2018using ;
gaus2019evaluating , we propose to derive proposals according to the hierarchy
of the regions defined by the contours. The insight driving this approach is
that contours are the most reliable cue in the X-ray scans due to the lack of
surface texture. For example, the occluded items exhibit different
transitional patterns based upon their orientation, contrast, and intensity.
We try to amplify and exploit this information through the multi-scale scan
decomposition, which boosts the proposed framework’s capacity for detecting
the underlying contraband data in the presence of clutter. Furthermore, we are
also motivated by the fact that organic material’s suspicious items show only
their outlines in the X-ray scans hassan2019 . To summarize, the main features
of this paper are:

  * •

Detection of overlapping suspicious items by analyzing their predominant
orientations across multiple scales within the candidate scan. Unlike
hassan2019 ; Hassan2020ACCV ; hassan2020Sensors , we propose a novel tensor
pooling strategy to decompose the scan across various scales and fuses them
via a single multi-scale tensor. This scheme results in more salient contour
maps (see Figure 1), boosting our framework’s capacity for handling dulled,
concealed, and overlapping items.

  * •

A thorough validation on three publicly available large-scale baggage X-ray
datasets, including the OPIXray opixray , which is the only dataset allowing a
quantitative measure of the level of occlusion.

  * •

Unlike state-of-the-art methods such as CST hassan2019 , TST Hassan2020ACCV ,
and DTS hassan2020Sensors , the performance of the proposed framework to
detect occluded items has been quantitatively evaluated on OPIXray opixray
dataset. Please see Table 4 for more details.

##  2 Related Work

Many researchers have developed computer-aided screening systems to identify
potential baggage threats mery2016 . While a majority of these frameworks are
based on conventional machine learning bastan2013object , the recent works
also employ supervised akccay2016transfer , and unsupervised akccay2019skip
deep learning, and these methods outperform conventional approaches both in
terms of performance, and efficiency akcay2018using . In this section, we
discuss some of the major baggage threat detection works. We refer the readers
to Mery2017TMSC ; ackay2020 for an exhaustive survey.

###  2.1 Traditional Methods

The early baggage screening systems were driven via classification
turcsany2013improving , segmentation heitz2010 and detection bastan2015
approaches to identify potential threats and smuggled items. Here, the work of
Bastan et al. bastan2013BMVC is appreciable, which identifies the suspicious
and illegal items within the multi-view X-ray imagery through fused SIFT and
SPIN driven SVM model. Similarly, SURF heitz2010 , and FAST-SURF
kundegorski2016 have also been used with the Bag of Words bastan2011 to
identify threatening items from the security X-ray imagery. Moreover,
approaches like adapted implicit shape model riffo2015automated and adaptive
sparse representation mery2016 were also commendable for screening suspicious
objects from the X-ray scans.

###  2.2 Deep Learning Frameworks

The deep learning-based baggage screening frameworks have been broadly
categorized into supervised and unsupervised learning schemes.

####  2.2.1 Supervised Methods

The initial deep learning approaches involved scan-level classification to
identify the suspicious baggage content akccay2016transfer . However, with the
recent advancements in object detection, researchers also employed
sophisticated detectors like RetinaNet retinanet , YOLO yolo ; yolov2 , and
Faster R-CNN fasterrcnn to not only recognize the contraband items from the
baggage X-ray scans but also to localize them via bounding boxes
akcay2018using . Moreover, researchers also proposed semantic segmentation
an2019 and instance segmentation Hassan2020ACCV models to recognize
threatening and smuggled items from the grayscale and colored X-ray imagery.
Apart from this, Xiao et al. _45 presented an efficient implementation of
Faster R-CNN fasterrcnn to detect suspicious data from the TeraHertz imagery.
Dhiraj et al. _42 used Faster R-CNN fasterrcnn , YOLOv2 yolov2 , and Tiny YOLO
yolov2 to screen baggage threats contained within the scans of a publicly
available GDXray dataset mery2015gdxray . Gaus et al. gaus2019evaluation
utilized RetinaNet retinanet , Faster R-CNN fasterrcnn , Mask R-CNN maskrcnn
(driven through ResNets he2016deep , VGG-16 vgg16 , and SqueezeNet
i2016squeezenet ) to detect prohibited baggage items. In another approach
gaus2019evaluating , they analyzed the transferability of these models on a
similarly styled X-ray imagery contained within their local dataset as well as
the SIXray10 subset of the publicly available SIXray dataset miao2019sixray .
Similarly, Akçay et al. akcay2018using compared Faster R-CNN fasterrcnn ,
YOLOv2 yolov2 , R-FCN rfcn , and sliding-window CNN with the AlexNet alexnet
driven SVM model to recognize occluded contraband items from the X-ray
imagery. Miao et al. miao2019sixray explored the imbalanced nature of the
contraband items in the real-world by developing a class-balanced hierarchical
refinement (CHR) framework. Furthermore, they extensively tested their
framework (backboned through different classification models) on their
publicly released SIXray miao2019sixray dataset. Wei et al. opixray presented
a plug-and-play module dubbed De-occlusion Attention Module (DOAM) that can be
coupled with any object detector to enhance its capacity towards screening
occluded contraband items. DOAM was validated on the publicly available
OPIXray opixray dataset, which is the first of its kind in providing
quantitative assessments of baggage screening frameworks under low, partial,
and full occlusion opixray .  Apart from this, Hassan et al. hassan2019 also
addressed the imbalanced nature of the contraband data by developing the
cascaded structure tensors (CST) based baggage threat detector. CST hassan2019
generates a balanced set of contour-based proposals, which are then utilized
in training the backbone model to screen the normal and abnormal baggage items
within the candidate scan hassan2019 . Similarly, to overcome the need to
train the threat detection systems on large-scale and well-annotated data,
Hassan et al. hassan2020Sensors introduced meta-transfer learning-based dual
tensor-shot (DTS) detector. DTS hassan2020Sensors analyzes the scan’s saliency
to produce low and high-density contour maps from which the suspicious
contraband items are identified effectively with few-shot training
hassan2020Sensors . In another approach, Hassan et al. Hassan2020ACCV
developed an instance segmentation-based threat detection framework that
filters the contours of the suspicious items from the regular content via
trainable structure tensors (TST) Hassan2020ACCV to identify them accurately
within the security X-ray imagery.

####  2.2.2 Unsupervised Methods

While most baggage screening frameworks involved supervised learning,
researchers have also explored adversarial learning to screen contraband data
as anomalies. Akçay et al. akcay2018ganomaly , among others, laid the
foundation of unsupervised baggage threat detection by proposing GANomaly
akcay2018ganomaly , an encoder-decoder-encoder network trained in an
adversarial manner to recognize prohibited items within baggage X-ray scans.
In another work, they proposed Skip-GANomaly akccay2019skip which employs
skip-connections in an encoder-decoder topology that not only gives better
latent representations for detecting baggage threats but also reduces the
overall computational complexity of GANomaly akcay2018ganomaly .

Figure 1: (A) An exemplar X-ray scan from the GDXray dataset mery2015gdxray ,
(B) contour map obtained through the modified structure tensors in hassan2019
and Hassan2020ACCV , (C) contour map obtained through proposed tensor pooling
strategy.

The rest of the paper is organized as follows: Section 3 presents the proposed
framework. Section 4 describes the experimental setup. Section 5 discusses the
results obtained with three public baggage X-ray datasets. Section 6 concludes
the paper and enlists future directions.

Figure 2: Block diagram of the proposed framework. The input scan is passed to
the tensor pooling module to extract the tensor representations encoding the
baggage items’ contours at different orientations. These representations are
fused into a single multi-scale tensor and passed afterward to an asymmetric
encoder-decoder backbone that segments and recognizes the contraband item’s
contours while suppressing the rest of the baggage content. For each detected
contour, the corresponding bounding box and mask is generated to localize the
detected contraband items. The abbreviations are CV: Convolution, BN: Batch
Normalization, SPB: Shape Preserving Block, IB: Identity Block, MP: Max
Pooling, AP: Average Pooling, ZP: Zero Padding, SM: Softmax.

##  3 Proposed Approach

The block diagram of the proposed framework is depicted in Figure 2. The input
scan is fed to the tensor pooling module (block A) to generate a multi-scale
tensor representation, revealing the baggage content’s transitional patterns
at multiple predominant orientations and across various scales. Afterward, the
multi-scale tensor is passed to the encoder-decoder backbone (block B),
implementing the newly proposed contour maps-based instance segmentation. This
block extracts the contours of the prohibited data while eliminating the
irrelevant scan content. In the third stage (block C), each extracted contour,
reflecting the contraband item instance, is utilized in generating the
respective mask and the bounding box for localization. In the subsequent
sections, we present a detailed description of each module within the proposed
framework.

###  3.1 Tensor Pooling Module

The tensor pooling module decomposes the input scan into $n$ levels of a
pyramid. From each level of the pyramid, the baggage content’s transitional
patterns are generated by analyzing their distribution of orientations within
the associated image gradients. In the proposed tensor pooling scheme, we
highlight the transitional patterns in $N$ image gradients (corresponding to
$N$ directions) by computing the following $N\times N$ block-structured
symmetric matrix hassan2019 ; Hassan2020ACCV :

$\begin{bmatrix}\phi*(\nabla^{0}.\nabla^{0})&\phi*(\nabla^{1}.\nabla^{0})&\cdots&\phi*(\nabla^{N-1}.\nabla^{0})\\\
\phi*(\nabla^{0}.\nabla^{1})&\phi*(\nabla^{1}.\nabla^{1})&\cdots&\phi*(\nabla^{N-1}.\nabla^{1})\\\
\vdots&\vdots&\ddots&\vdots\\\
\phi*(\nabla^{0}.\nabla^{N-1})&\phi*(\nabla^{1}.\nabla^{N-1})&\cdots&\phi*(\nabla^{N-1}.\nabla^{N-1})\\\
\end{bmatrix},$ (1)

Each tensor ($\phi*(\nabla^{k}.\nabla^{m})$) in the above block-matrix is an
outer product of two image gradients and a smoothing filter $\phi$. Moreover,
the orientation ($\theta$), of the image gradient $\nabla^{j}$, is computed
through: $\theta=\frac{2\pi j}{N}$, where $j$ ranges from $0$ to $N-1$. Since
the block-structured matrix in Eq. 1 is symmetric, we obtain
$\frac{N(N-1)}{2}$ unique tensors. From this group, we derive the coherent
tensor, reflecting the baggage items’ predominant orientations. The coherent
tensor is a single tensor representation generated by adding the most useful
tensors out of the $\frac{N(N+1)}{2}$ unique tensor set. Here, it should be
noted that these useful tensors are selected by ranking all the
$\frac{N(N+1)}{2}$ unique tensors according to their norm.

Moreover, the coherent tensor also reveals the variations in the intensity of
the cluttered baggage items, aiding in generating individual contours for each
item. However, this scheme analyzes only the intensity variations of the
baggage items at a single scale, limiting the extraction of the objects having
lower transitions with the background hassan2019 ; Hassan2020ACCV . To address
this limitation, we propose a multi-scale tensor fusing the X-ray scan
transitions from coarsest to finest levels so that each item, even having a
low-intensity difference with the background, can be adequately highlighted.
For example, see the boundaries of: the razor in a multi-scale tensor
representation in Figure 1 (C), the straight knife in Figure 3 (G), the two
knives and a gun in Figure 3 (H), and the two guns and a knife in Figure 3
(I).

1 Input:  X-ray scan ($I$), Scaling Factor ($n$), Number of Orientations ($N$)

2Output:  Multi-scale Tensor ($M_{t}$)

3[$r$, $c$] = size($I$)

4Initialize $M_{t}$ (of size $r\times c$) with zeros

5Set $\eta=2$ // pyramid pooling factor

6for  _$i=0$ to $n-1$_ do

7 if  _$i$ is 0_ then

8 $\Im$ = ComputeTensors($I$, $N$) // $\Im$: Tensors

9 $\Im_{c}$ = GetCoherentTensor($\Im$)

10 $M_{t}$ = $M_{t}$ \+ $\Im_{c}$

11 else

12 [$s$, $t$] = size($I$)

13 if  _(min( $s$, $t$) % $\eta)\neq 0$ or min($s$, $t$) $<\eta$_ then

14 break

15 end if

16

17 $I$ = Pool($I$, $\eta$)

18 $\Im_{c}$ = Pool($\Im_{c}$, $\eta$)

19 $I$ = $I\times\Im_{c}$

20 $\Im$ = ComputeTensors($I$, $N$)

21 $\Im_{c}$ = GetCoherentTensor($\Im$)

22 $M_{t}$ = $M_{t}$ \+ Unpool($\Im_{c}$, $\eta^{i}$)

23 end if

24

25 end for

Algorithm 1 Tensor Pooling Module Figure 3: Difference between conventional
structure tensors (used in hassan2019 ; Hassan2020ACCV ), and proposed multi-
scale tensor approach. First row shows the original scans from OPIXray opixray
, GDXray mery2015gdxray , and SIXray miao2019sixray dataset. The second row
shows the output for the conventional structure tensors hassan2019 ;
Hassan2020ACCV . The third row shows the output for the proposed tensor
pooling module.

As mentioned earlier, the multi-scale tensors are computed through pyramid
pooling (up to $n^{th}$ level). At any $l^{th}$ level, (such that $2\leq l\leq
n$), we multiply, pixel-wise, the decomposed image with the transitions
obtained at the previous ($l-1$) levels. In so doing, we ensure that the edges
of the contraband items (procured earlier) are retained across each scale. The
full procedure of the proposed tensor pooling module is depicted in Algorithm
1 and also shown in Figure 2.

Figure 4: Contour instance segmentation from multi-scale tensors. The first
column shows the original scans, the second column shows the multi-scale
tensor representations, the third column shows the ground truths, and the
fourth column shows the extracted contours of the contraband items.

The multi-scale tensor is then passed to the proposed encoder-decoder model to
extract the contours of the individual suspicious items. A detailed discussion
about contour instance segmentation is presented in the subsequent section.

###  3.2 Contour Instance Segmentation

The contour instance segmentation is performed through the proposed asymmetric
encoder-decoder network, which assigns the pixels in the multi-scale tensors
to one of the following categories $\mathcal{C}_{k=1:\mathcal{M}+1}$ where
$\mathcal{M}$ denotes the number of prohibited items’ instances to which we
add the class background which include background and irrelevant pixels (i.e.,
pixels belonging to a non-suspicious baggage content).

Furthermore, to differentiate between the contours of the normal and
suspicious items, the custom shape-preserving (SPB) and identity blocks (IB)
have been added within the encoder topology. The SPB, as depicted in Figures 2
and 5 (A), integrates the multi-scale tensor map (after scaling) in the
feature map extraction to enforce further the attention on prohibited items’
outlines. The IB (Figure 5-A), inspired by ResNet architecture he2016deep ,
acts as a residual block to emphasize the feature maps of the previous layer.

Apart from this, the whole network encompasses one input, one zero-padding, 22
convolution, 20 batch normalization, 12 activation, four pooling, two
multiply, six addition, three lambda (that implements the custom functions),
and one reshape layer. Moreover, we use skip-connections (via addition) within
the encoder-decoder to refine the extracted items’ boundaries. The number of
parameters within the network is 1,308,160, from which around 6,912 parameters
are non-trainable. The detailed summary of the proposed model (including the
architectural details of the SPB and IB blocks) is available in the source
code repository111The source code of the proposed framework along with its
complete documentation is available at
https://github.com/taimurhassan/tensorpooling..

Figure 5: (A) Shape Preserving Block (SPB), (B) Identity Block (IB).

###  3.3 Bounding Box and Mask Generation

After segmenting the contours, we perform morphological post-processing to
remove tiny and isolated fragments. The obtained outlines contain both open
and closed contours of the underlying suspicious items. The closed contours
can directly lead towards generating the corresponding item’s mask. For open
contours, we join their endpoints and then derive their masks through
morphological reconstruction. Afterward, we generate the items’ bounding boxes
from the masks as shown in Figure 2 (C).

##  4 Experimental Setup

This section presents the details about the experimental protocols, datasets,
and evaluation metrics which were in order to assess the proposed system’s
performance and compare it with state-of-the-art methods.

###  4.1 Datasets

We validated the proposed framework on three different publicly available
baggage X-ray datasets, namely, GDXray mery2015gdxray , SIXray miao2019sixray
, and OPIXray opixray . The detailed description of these datasets are
presented below.

####  4.1.1 GDXray

GDXray mery2015gdxray was first introduced in 2015 and it contains 19,407
high-resolution grayscale X-ray scans. The dataset is primarily designed for
the non-destructive testing purposes and the scans within GDXray
mery2015gdxray are arranged into five categories, i.e., welds, baggage,
casting, settings and nature. But baggage is the only relevant group for this
study and it contains 8,150 grayscale X-ray scans. Moreover, the dataset also
presents the detailed annotations for the prohibited items such as shuriken,
knives, guns, and razors. As per the dataset standard, 400 scans from GDXray
mery2015gdxray were used for training purposes, while the remaining scans were
used for testing purposes.

####  4.1.2 SIXray

SIXray miao2019sixray is a recently introduced large-scale security inspection
X-ray dataset. It contains a total of 1,059,231 colored X-ray scans from which
8,929 scans are positive (containing prohibited items such as knives,
wrenches, guns, pliers, hammer and scissors along with their ground truths),
and 1,050,302 are negative (containing only the normal items). To validate the
performance against class imbalance, the authors of the dataset presented
three subset schemes of the dataset, namely, SIXray10, SIXray100, and
SIXray1000 miao2019sixray . Moreover, SIXray miao2019sixray is also the
largest and most challenging dataset (to date) designed to assess threat
detection frameworks’ performance towards screening extremely cluttered and
highly imbalanced contraband data miao2019sixray ; hassan2020Sensors . As per
the SIXray miao2019sixray dataset standard, we used 80% scans for the training
and the rest of 20% for testing.

####  4.1.3 OPIXray

OPIXray opixray is the most recent baggage X-ray dataset (released publicly
for the research community in 2020). It contains 8,885 colored X-ray scans. As
per the dataset standard, out of these 8,885 scans, 7,109 are to be utilized
for the training purposes, while the remaining 1,776 are to be used for
testing purposes, to detect scissor, straight knife, multi-tool knife, folding
knife, and utility knife. Moreover, the dataset authors also quantified
occlusion within the test scans into three levels, i.e., OP1, OP2, and OP3.
OP1 indicates that the contraband items within the candidate scan contain no
or slight occlusion, OP2 depicts a partial occlusion, while OP3 represents
severe or full occlusion cases.

We also want to highlight here that the resolution of the scans within each
dataset varies significantly (except for OPIXray opixray ). For example, on
GDXray mery2015gdxray , the scan resolution varies as $2688\times 2208$,
$900\times 1430$, $850\times 850$ and $601\times 1241$, etc. Similarly, on
SIXray miao2019sixray , the scan resolution varies as $681\times 549\times 3$,
$801\times 482\times 3$, $649\times 571\times 3$, $1024\times 640\times 3$,
and $675\times 382\times 3$, etc. But on OPIXray opixray , the resolution of
all the scans is 1225x954x3. In order to process all the scans with the
proposed framework, we have re-sized them to the common resolution of
$576\times 768\times 3$, which is extensively used in the recently published
frameworks hassan2019 ; Hassan2020ACCV ; hassan2020Sensors .

###  4.2 Training and Implementation Details

The proposed framework was developed using Python 3.7.4 with TensorFlow 2.2.0
and Keras APIs on a machine havingIntel Core<EMAIL_ADDRESS>GHz CPU, 128 GB
RAM and an NVIDIA Quadro RTX 6000 with cuDNN v7.5, and a CUDA Toolkit
10.1.243. Some utility functions are also implemented using MATLAB R2021a.
Apart from this, the training on each dataset was conducted for a maximum of
50 epochs using ADADELTA Zeiler2012ADADELTA as an optimizer (with the default
learning and decay rate configurations) and a batch size of 4. Moreover, 10%
of the training samples from each dataset were used for the validation (after
each epoch). For the loss function, we used the focal loss retinanet expressed
below:

$l_{f}=-\frac{1}{b_{s}}\sum\limits_{i=0}^{b_{s}-1}\sum\limits_{j=0}^{c-1}\alpha(1-p(l_{i,j}))^{\gamma}t_{i,j}\log(p(l_{i,j}))$
(2)

where $c$ represents the total number of classes, and $b_{s}$ denotes the
batch size. $p(l_{i,j})$ denotes the predicted probability of the logit
$l_{i,j}$ generated from $i^{th}$ training sample for the $j^{th}$ class,
$t_{i,j}$ tells if the $i^{th}$ training sample actually belongs to the
$j^{th}$ class or not, the term $\alpha(1-p(l_{i,j}))^{\gamma}$ represents the
scaling factor that gives more weight to the imbalanced classes (in other
words, it penalizes the network to give emphasize to the classes for which the
network obtain low prediction scores). Through rigorous experiments, we
empirically selected the optimal value of $\alpha$ and $\gamma$ as 0.25 and 2,
respectively, as they result in faster learning for each dataset while
simultaneously showing good resistance to the imbalanced data. Apart from
this, architecturally, the kernel sizes within the proposed encoder-decoder
backbone vary as 3x3 and 7x7, whereas the number of kernels varies as 64, 128,
256, 512, 1024, and 2048. Moreover, the pooling size within the network
remained 2x2 across various network depths to perform the feature
decomposition (at each depth) by the factor of 2. For more architectural and
implementation details of the proposed framework, we refer the reader to the
source code, which we have released publicly for the research community on
GitHub1.

###  4.3 Evaluation Metrics

In order to assess the proposed approach and compare it with the existing
works, we used the following evaluation metrics:

####  4.3.1 Intersection-over-Union

Intersection-over-Union (IoU) tells how accurately the suspicious items have
been extracted, and it is measured by checking the pixel-level overlap between
the predictions and the ground truths. Mathematically, IoU is defined as:

$IoU=\frac{T_{p}}{T_{p}+F_{p}+F_{n}},$ (3)

where $T_{p}$ are true positives (indicating that the pixels of the contraband
items are correctly predicted w.r.t the ground truth), $F_{p}$ represents
false positives (indicating that the background pixels are incorrectly
classified as positives), and $F_{n}$ represents false negatives (meaning that
the pixels of the contraband items are misclassified as background).
Furthermore, we also calculated the mean IoU ($\mu$IoU) by taking an average
of the IoU score for each contraband item class.

####  4.3.2 Dice Coefficient

Apart from IoU scores, we also computed the dice coefficient (DC) scores to
assess the proposed system’s performance for extracting the contraband items.
DC is calculated through:

$DC=\frac{2T_{p}}{2T_{p}+F_{p}+F_{n}},$ (4)

Compared to IoU, DC gives more weightage to the true positives (as evident
from Eq. 4). Moreover, the mean DC ($\mu$DC) is calculated by averaging DC
scores for each category.

####  4.3.3 Mean Average Precision

The mean average precision (mAP) (in the proposed study) is computed by taking
the mean of average precision (AP) score calculated for each contraband item
class for the IoU threshold $\geq$ 0.5. Mathematically, mAP is expressed
below:

$mAP=\sum_{i=0}^{n_{c}-1}AP(i),$ (5)

where $n_{c}$ denotes the number of contraband items in each dataset. Here, we
want to highlight that to achieve fair comparison with the state-of-the-art,
we have used the original bounding box ground truths of each dataset for
measuring the proposed framework’s performance towards extracting the
suspicious and illegal items.

##  5 Results

In this section, we present the detailed results obtained with GDXray
mery2015gdxray , SIXray miao2019sixray , and OPIXray opixray datasets. Before
going into the experimental results, we present detailed ablation studies to
determine the proposed framework’s hyper-parameters. We also report a detailed
comparison of the proposed encoder-decoder network with the popular
segmentation models.

###  5.1 Ablation Studies

The ablation studies in this paper aim to determine the optimal values for 1)
the number of orientations and scaling levels within the tensor pooling module
and 2) the choice of the backbone model for performing the contour instance
segmentation.

(A) (B)

(C) (D) (E) (F)

Figure 6: Detection performance of the proposed system in terms of mAP (A, B,
C), and computational time in terms of seconds (D, E, F) obtained for GDXray
mery2015gdxray , SIXray miao2019sixray , and OPIXray opixray datasets,
respectively.

####  5.1.1 Number of Orientations and the Scaling Levels

The tensor pooling module highlights the baggage content transitions in the
image gradients oriented in $N$ directions and up to $n$ scaling levels.
Increasing these parameters helps generate the best contour representation
leading towards a more robust detection, but also incurs additional
computational cost. As depicted in Figure 6 (A), we can see that for GDXray
dataset mery2015gdxray with $N=2$, $n=2$, we obtain an mAP score of 0.82. With
the combination $N=5$, $n=5$, we get 16.54% improvements in the detection
performance but at the expense of a 97.71% increase in computational time (see
Figure 6-D). Similarly, on the SIXray dataset miao2019sixray , we obtain
18.36% improvements in the detection performance (by increasing $N$ and $n$)
at the expense of 95.88% in the computational time (see Figure 6 (B, E)). The
same behavior is also noticed for OPIXray dataset opixray in Figure 6 (C, F).
Considering all the combinations depicted in Figure 6, we found that $N=4$ and
$n=3$ provide the best trade-off between the detection and run-time
performance across all three datasets.

####  5.1.2 Choice of a Backbone Model

The proposed backbone model has been specifically designed to segment the
suspicious items’ contours while discarding the normal baggage content. In
this series of experiments, we compared the proposed asymmetric encoder-
decoder model’s performance with popular encoder-decoder, scene parsing, and
fully convolutional networks. In terms of $\mu$DC and $\mu$IoU, we report the
performance results in Table 1. We can observe that the proposed framework
achieves the best extraction performance on OPIXray opixray and SIXray
miao2019sixray dataset, leading the second-best UNet ronneberger2015unet by
2.34% and 3.72%. On the GDXray mery2015gdxray , however, it lags from the
FCN-8 fcn8 and PSPNet zhao2017pyramid by 6.54% and 5.91%, respectively. But as
our model outperforms all the other architectures on the large-scale SIXray
miao2019sixray and OPIXray opixray datasets, we chose it as a backbone for the
rest of the experimentation.

Table 1: Performance comparison of the proposed backbone network with PSPNet zhao2017pyramid , UNet ronneberger2015unet and FCN-8 fcn8 for recognizing the boundaries of the contraband items. The best and second-best performances are in bold and underline, respectively. Moreover, the abbreviations are: Met: Metric, Data: Dataset, GDX: GDXray mery2015gdxray , SIX: SIXray miao2019sixray , OPI: OPIXray opixray . Met | Data | Proposed | PSPNet | UNet | FCN-8
---|---|---|---|---|---
$\mu$IoU | GDX | 0.4994 | 0.5585 | 0.4921 | 0.5648
| SIX | 0.7072 | 0.5659 | 0.6700 | 0.6613
| OPI | 0.7393 | 0.5645 | 0.7159 | 0.5543
$\mu$DC | GDX | 0.6661 | 0.7167 | 0.6596 | 0.7219
| SIX | 0.8285 | 0.7227 | 0.8024 | 0.7961
| OPI | 0.8501 | 0.7217 | 0.8344 | 0.7132
Table 2: Performance comparison between state-of-the-art baggage threat detection frameworks on GDXray (GDX), SIXray (SIX), and OPIXray (OPI) dataset in terms of mAP scores. ’-’ indicates that the respective score is not computed. Moreover, the abbreviations are: Data: Dataset, GDX: GDXray mery2015gdxray , SIX: SIXray miao2019sixray , OPI: OPIXray opixray , PF: Proposed Framework, and FD: FCOS fcos \+ DOAM opixray . Data | Items | PF | CST | TST | FD
---|---|---|---|---|---
GDX | Gun | 0.9872 | 0.9101 | 0.9761 | -
| Razor | 0.9691 | 0.8826 | 0.9453 | -
| Shuriken | 0.9735 | 0.9917 | 0.9847 | -
| Knife | 0.9820 | 0.9945 | 0.9632 | -
| mAP | 0.9779 | 0.9281 | 0.9672 | -
SIX | Gun | 0.9863 | 0.9911 | 0.9734 | -
| Knife | 0.9811 | 0.9347 | 0.9681 | -
| Wrench | 0.9882 | 0.9915 | 0.9421 | -
| Scissor | 0.9341 | 0.9938 | 0.9348 | -
| Pliers | 0.9619 | 0.9267 | 0.9573 | -
| Hammer | 0.9172 | 0.9189 | 0.9342 | -
| mAP | 0.9614 | 0.9595 | 0.9516 | -
OPI | Folding | 0.8528 | - | 0.8024 | 0.8671
| Straight | 0.7649 | - | 0.5613 | 0.6858
| Scissor | 0.8803 | - | 0.8934 | 0.9023
| Multi | 0.8941 | - | 0.7802 | 0.8767
| Utility | 0.8062 | - | 0.7289 | 0.7884
| mAP | 0.8396 | - | 0.7532 | 0.8241

###  5.2 Evaluation on GDXray Dataset

The performance of the proposed framework and of the state-of-the-art methods
on the GDXray mery2015gdxray dataset are reported in Table 2. We can observe
here that the proposed framework outperforms the CST hassan2019 and the TST
framework Hassan2020ACCV by 4.98% and 1.07%, respectively. Furthermore, we
wanted to highlight the fact that CST hassan2019 is only an object detection
scheme, i.e., it can only localize the detected items but cannot generate
their masks. Masks are very important for the human observers in cross-
verifying the baggage screening results (and identifying the false positives),
especially from the cluttered and challenging grayscale scans. In Figure 7, we
report some of the cluttered and challenging cases showcasing the
effectiveness of the proposed framework in extracting the overlapping
contraband items. For example, see the extraction of merged knife instances in
(H), and the cluttered shuriken in (J, L). We can also appreciate how
accurately the razors have been extracted in (J, L). Extracting such low
contrast objects in the competitive CST framework requires suppressing first
all the sharp transitions in an iterative fashion hassan2019 .

Figure 7: Qualitative evaluations of the proposed framework on GDXray
mery2015gdxray dataset. Please zoom-in for best visualization.

###  5.3 Evaluations on SIXray Dataset

The proposed framework has been evaluated on the whole SIXray dataset
miao2019sixray (containing 1,050,302 negative scans and 8,929 positive scans)
and also on each of its subsets miao2019sixray . In Table 2, we can observe
that the proposed framework achieves an overall performance gain of 0.190% and
0.980% over CST hassan2019 and TST Hassan2020ACCV framework, respectively. In
Table 3, we report the results obtained with each subset of the SIXray dataset
miao2019sixray , reflecting different imbalanced normal and prohibited item
categories. The results further confirm the superiority of the proposed
framework against other state-of-the-art solutions, especially w.r.t the CHR
miao2019sixray , and gaus2019evaluating . In addition to this, in an extremely
challenging SIXray1000 subset, we notice that the proposed framework leads the
second-best TST framework Hassan2020ACCV by 3.22%, and CHR miao2019sixray by
44.36%.

Apart from this, Figure 8 depicts the qualitative evaluations of the proposed
framework on the SIXray miao2019sixray dataset. In this figure, the first row
shows examples containing one instance of the suspicious item, whereas the
second and third rows show scans containing two or more instances of the
suspicious items. Here, we can appreciate how accurately the proposed scheme
has picked the cluttered knife in (B). Moreover, we can also observe the
extracted chopper (knife) in (D) despite having similar contrast with the
background. More examples such as (F, H, and J) demonstrate the proposed
framework’s capacity in picking the cluttered items from the SIXray dataset
miao2019sixray .

Figure 8: Qualitative evaluations of the proposed framework on SIXray miao2019sixray dataset. Please zoom-in for a best visualization. Table 3: Performance comparison of proposed framework with state-of-the-art solutions on SIXray subsets. For fair comparison, all models are evaluated using ResNet-50 he2016deep as a backbone. Moreover, the abbreviations are: SIX-10: SIXray10 miao2019sixray , SIX-100: SIXray100 miao2019sixray , SIX-1k: SIXray1000 miao2019sixray , and PF: Proposed Framework Subset | PF | DTS | CHR | gaus2019evaluating | TST
---|---|---|---|---|---
SIX-10 | 0.9793 | 0.8053 | 0.7794 | 0.8600 | 0.9601
SIX-100 | 0.8951 | 0.6791 | 0.5787 | - | 0.8749
SIX-1k | 0.8136 | 0.4527 | 0.3700 | - | 0.7814

###  5.4 Evaluations on OPIXray Dataset

The performance evaluation of the proposed framework on OPIXray dataset
opixray is reported in Table 2. We can observe here that the proposed system
achieves an overall mAP score of 0.8396, outperforming the second-best DOAM
framework opixray (driven via FCOS fcos ) by 1.55%. Here, although the
performance of both frameworks is identical, we still achieve a significant
lead of 7.91% over the DOAM opixray for extracting the straight knives.

Concerning the level of occlusion (as aforementioned, OPIXray opixray splits
the test data into three subsets, OP1, OP2, OP3, according to the level of
occlusion), we can see in Table 4 that the proposed framework achieves the
best performance at each occlusion level as compared to the second-best DOAM
opixray framework driven by the single-shot detector (SSD) Liu2016SSD .

Figure 9 reports some qualitative evaluation, where we can appreciate the
recognition of the cluttered scissor (e.g. see B and F), and overlapping
straight knife (in H). We can also notice the detection of the partially
occluded folding and straight knife in (D) and (J).

Figure 9: Qualitative evaluations of the proposed framework on OPIXray opixray
dataset. Please zoom-in for a best visualization.

###  5.5 Failure Cases

In Figure 10, we report examples of failure cases encountered during the
testing. In cases (B, H, N, and P), we can see that the proposed framework
could not pick-up the whole regions of the contraband items, even though the
items were detected correctly. However, such cases are observed in highly
occluded scans such as (A and G), where it is difficult, even for a human
observer, to distinguish the items’ regions properly. The second type of
failure corresponds to the pixels misclassification as shown in (D) where some
of the gun’s pixels have been misclassified as knife. We can address these
scenarios through post-processing steps like blob removal and region filling.
The third failure case relates to the proposed framework’s inability to
generate a single bounding box for the same item. Such a case is depicted in
(F), where two bounding boxes were generated for the single orange knife item.
One possible remedy here is to generate the bounding boxes based upon the
minimum and maximum mask value in both image dimensions for each label.
Another type of failure is shown in (J) and (L). Here, the scans contain only
normal baggage content, but some pixels occupying tiny regions have been
misclassified as false positive (i.e., knife). We can also address this kind
of failure through blob removal scheme.

Examining the failure cases’ statistical distributions, we found a majority of
86.09% cases belonging to the curable categories (i.e., second, third, and
fourth), meaning that the proposed framework’s performance can be further
improved using the post-processing techniques mentioned above.

Table 4: Performance comparison of proposed framework with DOAM opixray (backboned through SSD Liu2016SSD ) on different occlusion levels of OPIXray opixray dataset. Method | OP1 | OP2 | OP3
---|---|---|---
Proposed | 0.7946 | 0.7382 | 0.7291
DOAM + SSD opixray | 0.7787 | 0.7245 | 0.7078
SSD Liu2016SSD | 0.7545 | 0.6954 | 0.6630

##  6 Conclusion

In this work, we proposed a novel contour-driven approach for detecting
cluttered and occluded contraband items (and their instances) within the
baggage X-ray scans, hypothesizing that contours are the most robust cues
given the lack of texture in the X-ray imagery. We concretized this original
approach through a tensor pooling module, producing multi-scale tensor maps
highlighting the items’ outlines within the X-ray scans and an instance
segmentation model acting on this representation. We validated our approach on
three publicly available datasets encompassing gray-level and colored scans
and showcased its overall superiority over competitive frameworks in various
aspects. For instance, the proposed framework outperforms the state-of-the-art
methods opixray ; hassan2019 ; Hassan2020ACCV ; hassan2020Sensors by 1.07%,
0.190%, and 1.55% on GDXray mery2015gdxray , SIXray miao2019sixray , and
OPIXray opixray dataset, respectively. Furthermore, on each SIXray subsets
(i.e., SIXray10, SIXray100, SIXray1000) miao2019sixray , the proposed
framework leads the state-of-the-art by 1.92%, 2.02%, and 3.22%, respectively.

In future, we aim to apply the proposed framework to recognize 3D printed
contraband items from the X-ray scans. Such items exhibit poor visibility in
the X-ray scans because of their organic material, making them an enticing and
challenging case to investigate and address.

Figure 10: Failure cases from GDXray mery2015gdxray , SIXray miao2019sixray ,
and OPIXray opixray dataset.

## Conflicts of Interest Declarations

Funding: This work is supported by a research fund from ADEK (Grant Number:
AARE19-156) and Khalifa University (Grant Number: CIRA-2019-047).

Conflict of Interest: The authors have no conflicts of interest to declare
that are relevant to this article.

Financial and Non-Financial interests: All the authors declare that they have
no financial or non-financial interests to disclose for this article.

Employment: The authors conducted this research during their employment in the
following institutes:

  * •

T. Hassan (Khalifa University, UAE),

  * •

S. Akçay (Durham University, UK),

  * •

M. Bennamoun (The University of Western Australia, Australia),

  * •

S. Khan (Mohamed bin Zayed University of Artificial Intelligence, UAE), and

  * •

N. Werghi (Khalifa University, UAE).

Ethics Approval: All the authors declare that no prior ethical approval was
required from their institutes to conduct this research.

Consent for Participate and Publication: All the authors declare that no prior
consent was needed to disseminate this article as there were no human (or
animal) participants involved in this research.

Availability of Data and Material: All the datasets that have been used in
this article are publicly available.

Code Availability: The source code of the proposed framework is released
publicly on GitHub1.

Authors’ Contributions: T. Hassan formulated the idea, wrote the manuscript,
and performed the experiments. S. Akçay improved the initial design of the
framework and contributed to manuscript writing. M. Bennamoun co-supervised
the whole research, reviewed the manuscript and experiments. S. Khan reviewed
the manuscript, experiments and improved the manuscript writing. N. Werghi
supervised the whole research, contributed to manuscript writing, and reviewed
the experimentation.

## References

  * (1) Z. Tang, E. Tian, Y. Wang, L. Wang, and T. Yang, “Nondestructive Defect Detection in Castings by Using Spatial Attention Bilinear Convolutional Neural Network,”  IEEE Transactions on Industrial Informatics, 2020.
  * (2) M. Bastan, W. Byeon, and T. Breuel, “Object Recognition in Multi-View Dual Energy X-ray Images,”  British Machine Vision Conference, 2013.
  * (3) D. Turcsany, A. Mouton, and T. P. Breckon, “Improving Feature-based Object Recognition for X-ray Baggage Security Screening using Primed Visual Words,” in 2013 IEEE International Conference on Industrial Technology (ICIT), pp. 1140–1145, IEEE, 2013.
  * (4) B. Hu, C. Zhang, L. Wang, Q. Zhang, and Y. Liu, “Multi-label X-ray Imagery Classification via Bottom-up Attention and Meta Fusion,”  Asian Conference on Computer Vision (ACCV), 2020.
  * (5) S. Akçay et al., “Transfer learning using convolutional neural networks for object classification within X-ray baggage security imagery,” in IEEE ICIP, pp. 1057–1061, 2016.
  * (6) S. Akcay et al., “Using Deep Convolutional Neural Network Architectures for Object Classification and Detection Within X-ray Baggage Security Imagery,” IEEE Transactions on Information Forensics and Security, vol. 13, no. 9, pp. 2203–2215, 2018.
  * (7) Y. F. A. Gaus et al., “Evaluation of a Dual Convolutional Neural Network Architecture for Object-wise Anomaly Detection in Cluttered X-ray Security Imagery,” in 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, 2019.
  * (8) K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in IEEE International Conference on Computer Vision (ICCV), pp. 2961–2969, 2017.
  * (9) S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, June 2015.
  * (10) T. Y. Lin et al., “Focal Loss for Dense Object Detection,”  IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * (11) J. Redmon and A. Farhadi, “YOLOv3: An Incremental Improvement,”  arXiv, 2018.
  * (12) Y. Wei et al., “Occluded Prohibited Items Detection: An X-ray Security Inspection Benchmark and De-occlusion Attention Module,” 2020.
  * (13) C. Miao et al., “SIXray: A Large-scale Security Inspection X-ray Benchmark for Prohibited Item Discovery in Overlapping Images,” in IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2119–2128, 2019.
  * (14) S. Akçay and T. Breckon, “Towards Automatic Threat Detection: A Survey of Advances of Deep Learning within X-ray Security Imaging,” preprint arXiv:2001.01293, 2020.
  * (15) Y. F. A. Gaus et al., “Evaluating the Transferability and Adversarial Discrimination of Convolutional Neural Networks for Threat Object Detection and Classification within X-ray Security Imagery,” arXiv preprint arXiv:1911.08966, 2019.
  * (16) M. Bastan et al., “Object recognition in multi-view dual energy X-ray images.,” in BMVC, vol. 1, p. 11, 2013.
  * (17) S. Akçay, A. Atapour-Abarghouei, and T. P. Breckon, “Skip-GANomaly: Skip Connected and Adversarially Trained Encoder-Decoder Anomaly Detection,” International Joint Conference on Neural Networks (IJCNN), 2019.
  * (18) D. Mery, E. Svec, M. Arias, V. Riffo, J. M. Saavedra, and S. Banerjee, “Modern Computer Vision Techniques for X-Ray Testing in Baggage Inspection,”  IEEE Transactions on Systems, Man, and Cybernetics: Systems, Volume: 47, Issue: 4, pp. 682–692 April 2017.
  * (19) G. Heitz and G. Chechik, “Object Separation in X-ray Image Sets,” in IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), p. 2093–2100, 2010.
  * (20) M. Bastan, “Multi-view Object Detection In Dual-energy X-ray Images,” Machine Vision and Applications, p. 1045–1060, 2015.
  * (21) M. Bastan, M. R. Yousefi, and T. M. Breuel, “Visual Words on Baggage X-ray Images,”  14th International Conference on Computer Analysis of Images and Patterns, August 2011.
  * (22) M. E. Kundegorski et al., “On using Feature Descriptors as Visual Words for Object Detection within X-ray Baggage Security Screening,” in IEEE International Conference on Imaging for Crime Detection and Prevention (ICDP), 2016.
  * (23) D. Mery, E. Svec, and M. Arias, “Object Recognition in Baggage Inspection Using Adaptive Sparse Representations of X-ray Images,” in Pacific-Rim Symposium on Image and Video Technology, p. 709–720, 2016.
  * (24) V. Riffo and D. Mery, “Automated Detection of Threat Objects Using Adapted Implicit Shape Model,”  IEEE Transactions on Systems, Man, and Cybernetics: Systems, Volume: 46, Issue: 4, pp. 472–482, 2016.
  * (25) Z. Liu, J. Li, Y. Shu, and D. Zhang, “Detection and Recognition of Security Detection Object Based on YOLO9000,” in 2018 5th International Conference on Systems and Informatics (ICSAI), pp. 278–282, IEEE, 2018.
  * (26) M. Xu et al., “Prohibited Item Detection in Airport X-Ray Security Images via Attention Mechanism Based CNN,” in Chinese Conference on Pattern Recognition and Computer Vision, p. 429–439, 2018\.
  * (27) N. Jaccard et al., “Detection of Concealed Cars in Complex Cargo X-ray Imagery using Deep Learning,” in Journal of X-Ray Science and Technology, p. 323–339, 2017.
  * (28) L. D. Griffin, M. Caldwell, J. T. A. Andrews, and H. Bohler, “Unexpected Item in the Bagging Area: Anomaly Detection in X-ray Security Images.,”  IEEE Transactions on Information Forensics and Security, pp. 1539–1553, 2019.
  * (29) J. An, H. Zhang, Y. Zhu, and J. Yang, “Semantic Segmentation for Prohibited Items in Baggage Inspection,”  International Conference on Intelligence Science and Big Data Engineering. Visual Data Engineering, pp. 495–-505, 2019.
  * (30) L. Zou, T. Yusuke, and I. Hitoshi, “Dangerous Objects Detection of X-ray Images Using Convolution Neural Network,”  Security with Intelligent Computing and Big-data Services, 2018\.
  * (31) J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,”  IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * (32) J. Redmon and A. Farhadi, “YOLO9000: Better, Faster, Stronger,”  IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * (33) C. Szegedy et al., “Going Deeper with Convolutions,”  IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * (34) H. Xiao et al., “R-PCNN Method to Rapidly Detect Objects on THz Images in Human Body Security Checks,”  IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation, October 2018\.
  * (35) K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,”  IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * (36) F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and $<$0.5MB model size,”  arXiv preprint arXiv:1602.07360, 2016.
  * (37) K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * (38) Dhiraj and D. KumarJain, “An evaluation of deep learning based object detection strategies for threat object detection in baggage security imagery,”  Pattern Recognition Letters, January 2019.
  * (39) D. Mery et al., “GDXray: The database of X-ray images for nondestructive testing,”  Journal of Nondestructive Evaluation, Volume 34, Issue: 4, 2015\.
  * (40) A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,”  Advances in Neural Information Processing Systems, 2012.
  * (41) J. Dai, Y. Li, K. He, and J. Sun, “R-FCN: Object Detection via Region-based Fully Convolutional Networks,” in Advances in Neural Information Processing Systems, pp. 379–387, 2016.
  * (42) T. Hassan et al., “Detecting Prohibited Items in X-ray Images: A Contour Proposal Learning Approach,” in Accepted in 27th IEEE International Conference on Image Processing (ICIP), 2020.
  * (43) T. Hassan and N. Werghi, “Trainable Structure Tensors for Autonomous Baggage Threat Detection Under Extreme Occlusion,”  Asian Conference on Computer Vision (ACCV), September 2020.
  * (44) T. Hassan, M. Shafay, S. Akçay, S. Khan, M. Bennamoun, E. Damiani, and N. Werghi, “Meta-Transfer Learning Driven Tensor-Shot Detector for the Autonomous Localization and Recognition of Concealed Baggage Threats,”  MDPI Sensors, November 2020.
  * (45) S. Akçay, A. Atapour-Abarghouei, and T. P. Breckon, “GANomaly: Semi-Supervised Anomaly Detection via Adversarial Training,” in Asian Conference on Computer Vision, pp. 622–637, Springer, 2018.
  * (46) M. D. Zeiler, “ADADELTA: An Adaptive Learning Rate Method,”  arXiv:1212.5701, 2012.
  * (47) O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation.”  arXiv:1505.04597, 2015.
  * (48) J. Long, E. Shelhamer, and T. Darrell, “Fully Convolutional Networks for Semantic Segmentation,” in IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440, 2015.
  * (49) H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid Scene Parsing Network,” in IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2881–2890, 2017.
  * (50) Z. Tian, C. Shen, H. Chen, and T. He, “FCOS: Fully Convolutional One-Stage Object Detection,”  IEEE International Conference on Computer Vision (CVPR), 2019\.
  * (51) W. Liu et al., “SSD: Single Shot MultiBox Detector,” in European Conference on Computer Vision, 2016.