# Status of the TORCH time-of-flight project

N. Harnew<EMAIL_ADDRESS>S. Bhasin  T. Blake  N.H. Brook  T.
Conneely  D. Cussans  M. van Dijk  R. Forty  C. Frei  E.P.M. Gabriel  R. Gao
T.J. Gershon  T. Gys  T. Hadavizadeh  T.H. Hancock  M. Kreps  J. Milnes  D.
Piedigrossi  J. Rademacker  Denys Wilkinson Laboratory, University of Oxford,
Keble Road, Oxford OX1 3RH, United Kingdom  University of Bath, Claverton
Down, Bath BA2 7AY, United Kingdom.  H.H. Wills Physics Laboratory, University
of Bristol, Tyndall Avenue, Bristol BS8 1TL, United Kingdom  European
Organisation for Nuclear Research (CERN), CH-1211 Geneva 23, Switzerland
School of Physics and Astronomy, University of Edinburgh, James Clerk Maxwell
Building, Edinburgh EH9 3FD, United Kingdom  Department of Physics, University
of Warwick, Coventry, CV4 7AL, United Kingdom  Photek Ltd., 26 Castleham Road,
St Leonards on Sea, East Sussex, TN38 9NS, United Kingdom

###### Abstract

TORCH is a time-of-flight detector, designed to provide charged $\pi/K$
particle identification up to a momentum of 10 GeV/$c$ for a 10 m flight path.
To achieve this level of performance, a time resolution of 15 ps per incident
particle is required. TORCH uses a plane of quartz of 1 cm thickness as a
source of Cherenkov photons, which are then focussed onto square Micro-Channel
Plate Photomultipliers (MCP-PMTs) of active area 53 $\times$ 53 mm2, segmented
into 8 $\times$ 128 pixels equivalent. A small-scale TORCH demonstrator with a
customised MCP-PMT and associated readout electronics has been successfully
operated in a 5 GeV/$c$ mixed pion/proton beam at the CERN PS facility.
Preliminary results indicate that a single-photon resolution better than 100
ps can be achieved. The expected performance of a full-scale TORCH detector
for the Upgrade II of the LHCb experiment is also discussed.

###### keywords:

Time-of-flight , particle identification , Cherenkov radiation , micro-channel
plate photomultipliers , LHCb upgrade

###### PACS:

29.40.Cs , 29.40.Gx

##  1 Introduction

The TORCH (Time Of internally Reflected CHerenkov light) detector will measure
the time-of-flight (ToF) of charged particles over large areas, with the aim
to provide Particle IDentification (PID) of pions, kaons and protons up to 10
GeV/$c$ momentum and beyond [1]. The difference in ToF between pions and kaons
over a $\sim$10 m flight path at 10 GeV/$c$ is 35 ps, hence to achieve
positive identification of kaons, TORCH aims for a time resolution of
$\sim$10-15 ps per track. A specific application of TORCH is for the LHCb
Upgrade II experiment, where the detector would occupy an area of 30 m2 in
front of the current RICH 2 detector [2]. The proposed experimental
arrangement is shown in Fig. 1.

TORCH combines timing measurements with DIRC-style reconstruction, a technique
pioneered by the BaBar DIRC [3] and Belle II TOP [4] collaborations. The
production of Cherenkov light is prompt, hence TORCH uses planes of 1 cm thick
quartz as a source of fast signal, which also facilitates a modular design.
Cherenkov photons travel to the periphery of the quartz plates by total
internal reflection where they are reflected by a cylindrical mirror surface
of a quartz block. This focuses the photons onto a plane of pixellated Micro-
Channel Plate Photomultipliers (MCP-PMTs) where their positions and arrival
times are measured. The expectation is that typically 30 photons will be
detected per charged track, hence the required ToF resolution dictates the
timing of single photons to a precision of around 70 ps.

A schematic of the TORCH geometry in the longitudinal and transverse planes is
shown in Fig. 2, showing the focussing block and the LHCb modular arrangement.
For every photon hit in the MCP-PMTs, the Cherenkov angle $\theta_{c}$

Figure 1: A schematic of the LHCb experiment, showing TORCH located directly
upstream of the RICH 2 detector.

and the photon path length of propogation through the quartz $L$ is measured.
From knowledge of the dispersion relation within the quartz, a correction for
chromatic dispersion is then made to the photon time of propogation. From
simulation, a $\sim$1 mrad precision is required on the measurement of the
angles in both planes to achieve the required intrinsic timing resolution [1].

Figure 2: Schematics of a TORCH module showing possible reflection paths: (a)
the focussing block and MCP-PMT plane, (b) a single LHCb module.

##  2 MCP-PMT development

In order to achieve the desired 1 mrad angular precision, TORCH requires a
photon detector with a fine granularity in the focussing direction and a
coarse granularity in the non-focussing direction. The pixel structure of MCP-
PMTs can in principle be adjusted to the resolution required provided the
charge footprint is small enough. An anode granulatity of $128\times 8$ pixels
is chosen with a $53\times 53$ mm2 active area on a 60 mm pitch. MCP-PMTs are
well known for fast timing of single photon signals, $\sim$30 ps, however tube
lifetime has been an issue in the past.

A major focus of the TORCH project has been on the development of the MCP-
PMTs, which has been conducted in collaboration with an industrial partner,
Photek (UK). A three-phase R&D programme was defined. Phase 1 saw the
development of a single-channel MCP-PMT with extended lifetime, accomplished
with an atomic-layer deposition (ALD) coating of the MCPs [5], and where an
excellent timing resolution of better than $\sim$35 ps was also achieved [6].
The extended lifetime is required for the harsh environment of the LHC, where
integrated anode charges of at least 5 Ccm-2 are expected. Lifetime
measurements have since been conducted over a period of 2.5 years using
single-photon illumination from a blue LED, and are now reaching an integrated
charge of 6.16 Ccm-2. Figure 3 demonstrates the results. Some loss of quantum
efficiency is seen above 3 Ccm-2, with a factor 2 loss at 5 Ccm-2. A gain drop
is also measured, but which can be recovered by an increase of MCP high
voltage [7]. The MCP-PMT lifetime is close to the required performance, and it
is expected that the Phase 3 tubes will improve on this.

Figure 3: The quantum efficiency of a Photek Phase 1 MCP-PMT as a function of
wavelength and collected integrated charge, as measured on the tube axis.

The Phase 2 MCP-PMTs are circular in construction with a 40 mm diameter and a
square 26.5 $\times$ 26.5 mm2 active area containing 4 $\times$ 32 pixels, a
quarter size of the final geometry. Beam tests on these tubes were
successfully completed in 2015/16 and are reported elsewhere [8].

The final TORCH devices, the Phase 3 tubes, have high active area ($>$80%),
the required granularity, ALD coating to give the extended lifetime, and
excellent time resolution. Ten Phase 3 MCP-PMTs have been delivered from
Photek and are currently under test. The MCP-PMT has a square 53 $\times$ 53
mm2 active area with 64 $\times$ 64 pixels. The effective resolution of 128
$\times$ 8 pixels is achieved in the focussing and non-focussing directions by
exploiting charge sharing between pixels [9] and by ganging electronically 64
pixels into 8 (or 4), respectively. The MCP-PMT anode is connected to a PCB
via Anisotropic Conductive Film (ACF) onto which a readout connector is
mounted. The readout configuration of the first evaluated tube has 64 $\times$
4 pixels, compatible with a previous version of electronics, and used in the
test-beam of November 2017. The second configuration gives 64 $\times$ 8
pixels per tube for use with new electronics in the test-beam of June 2018
[10].

##  3 Beam tests with a small-scale TORCH demonstrator

Several test-beam campaigns have been conducted between 2015–2018 at the CERN
PS T9 beamline with a 5 GeV/$c$ mixed pion/proton beam. The small-scale TORCH
demonstrator [8] consists of a $12\times 35\times 1$ cm3 quartz radiator plate
with a matching focusing block, both manufactured by Schott, Germany. The
radiator plate is mounted in an almost vertical position, tilted backwards by
5∘ with respect to the horizontal incidence of the beam.

We report here preliminary results from the November 2017 campaign, where the
demonstrator is read out with a single Phase 3 MCP-PMT in the 64 $\times$ 4
pixel configuration. The customised electronics system [10] uses the NINO32
[11] and HPTDC [12] chipsets, developed for fast timing applications of the
ALICE experiment. Proton-pion selection is achieved independent of TORCH using
a pair of upstream Cherenkov counters, and two borosilicate finger counters
(T1 and T2) separated by a $\sim$11 m flight path provide a time reference.

The pattern of measured MCP-PMT hits for 5 GeV/$c$ pions is shown in Fig.4
(a). Here clustering has been applied over simultaneous MCP-PMT column hits to
obtain the centroid position of each photon. The beam impinges approximately
14 cm below the plate centre-line and close to the plate side, a position
which has been chosen to give a cleanly resolved pattern. The Cherenkov cones
which are internally reflected in the quartz radiator result in hyperbola-like
patterns at the MCP-PMT plane; reflections off module sides result in a
folding of this pattern. Chromatic dispersion spreads the lines into bands.
Since the quartz is read out by only a single MCP-PMT, the full pattern is
only sampled, which accounts for the observed discontinuities.

For the timing measurement, a simultaneous correction is made for time-walk
and integral non-linearities of the NINO/HPTDC electronics using a data-driven
method [8]. For each single 4-wide pixel row, the measured MCP time-stamp for
each cluster is plotted relative to the downstream borosilicate station (T2)
versus the measured 64-wide column position. An example data distribution is

Figure 4: The patterns of hits measured in the TORCH demonstrator with 5
GeV/$c$ pions. (a) 64 $\times$ 4 pixel readout in November 2017 with the beam
14 cm below the quartz radiator centre-line and the MCP-PMT located in its
nominal position on the focal plane, and (b) 64 $\times$ 8 pixel readout in
June 2018 with the beam at the quartz centre-line and the MCP-PMT shifted 5 mm
upwards with respect to configuration (a).

shown in Fig. 5 (a) showing good agreement when compared to simulation. The
distribution of residuals between the measured and simulated times of arrival
is shown in Fig. 5 (b). Core distributions have resolutions (sigmas) of
approximately 100 - 125 ps (which is photon energy and MCP-PMT row dependent).
The tails are due to imperfect calibration and backscattering from the MCP top
surface. The timing resolution of the timing reference is $\sim$50 ps and,
subtracted in quadrature, gives $\sim$85 - 115 ps time resolution of the TORCH
demonstrator, approaching the target resolution of 70 ps per photon. Future
improvements are possible such as incorporating charge to width calibrations
of the front-end electronics and reducing the current limitation imposed by
the 100 ps time binning of the HPTDC.

Figure 5: (a) The time-of-arrival of single Cherenkov photons from a 5 GeV/$c$
pion beam, relative to the T2 beam time-reference station, as a function of
detected 64-wide column pixel number. The overlaid lines represent the
simulated patterns for light reflected only off the front and back faces of
the radiator plate (purple), light undergoing one (red), two (orange) and
three (yellow) reflections off the side faces. The top left distributions
correspond to multiple reflections from the bottom horizontal face. (b) The
residuals between observed and simulated Cherenkov photon arrival times for
pixel row 4 (November 2017 data).

##  4 Development of half-length TORCH prototype

A prototype of a half-length TORCH module, 125 $\times$ 66 $\times$ 1 cm3
(length, width and thickness), is currently under construction. The module
will be equipped with ten MCP-PMTs ($\sim$5000 channels). The radiator plate
and focussing block were both procured from Nikon, Japan. As an incremental
step, the small-scale TORCH demonstrator was equipped with a single 64
$\times$ 8 pixel MCP-PMT and upgraded electronics and tested in a 5 GeV/$c$
pion beam in June 2018. The pattern of measured MCP-PMT hits is shown in Fig.
4 (b). Results are currently being analysed, and calibrations and timing
measurements are in progress. The full-scale module is planned for test-beam
running in October/November 2018.

##  5 TORCH for the LHCb Upgrade II

The RICH system currently provides PID for the LHCb experiment [13], where
discrimination of pions, kaons and protons is essential for CP violation
measurements, exotic spectroscopy and particle tagging. However, LHCb has no
positive kaon or proton identification below $\sim$10 GeV/$c$. Therefore the
proposal is to install TORCH immediately upstream of the RICH 2 detector,
where it would be located $\sim$9.5 m from the proton-proton interaction
region [2]. Here the total area of TORCH would be 5 $\times$ 6 m2, divided
into 18 modules, each 66 cm wide and 2.5 m high with 11 MCP-PMTs per module.

Studies are underway to evaluate the performance of TORCH in the LHCb
experiment. A luminosity of 2 $\times$ 1033 cm-2s-1 has been simulated
($\times$ 5 the current LHCb luminosity), including collision pile-up. The
track entry point in the quartz radiator of TORCH is provided by position
information from the tracking system of the LHCb spectrometer. The expected
hit distributions for the $\pi/K/p$ hypotheses are compared to the MCP-PMT
photon spatial hits and arrival times, and log-likelihoods are then computed.

Figure 6 shows the efficiency of TORCH to positively identify kaons and
protons as a function of momentum and the probability of misidentification.
Good separation between $\pi$/K/p in the 2$-$10 GeV/$c$ range and beyond is
observed. Studies have also started on key physics channels and tagging
performance, and these will form the basis of a Technical Proposal to
construct a full-scale TORCH detector for the start-up of LHC Run 4, with
installation in the Long Shutdown 3 (LS3) in 2024.

Figure 6: The efficiency of TORCH in LHCb to positively identify (a) kaons and
(b) protons as a function of momentum and the probability that they are
misidentied. The curves are for two different delta-log-likelihood cuts and
for a luminosity of 2 $\times$ 1033 cm-2s-1. The simulated sample is for heavy
flavour decays in $pp$ collisions, including pile-up.

##  6 Summary

The performance of a small-scale TORCH demonstrator in a 5 GeV/$c$ mixed
pion/proton beam has been reported. A customised 64 $\times$ 4 pixelated MCP-
PMT has been prototyped, and timing resolutions of 85–115 ps per photon have
been measured. With future improvements, it is hoped to achieve the desired
per-photon resolution of 70 ps. A half-length TORCH module incorportating new
optics is under construction, incorporating final 64 $\times$ 8 pixelated MCP-
PMTs and a new generation of electronics. Studies are underway to prepare a
Technical Proposal for TORCH to be incorporated in the Upgrade-II of the LHCb
experiment.

## Acknowledgments

The support is acknowledged of the Science and Technology Research Council,
UK, and of the European Research Council through an FP7 Advanced Grant
(ERC-2011-AdG 299175-TORCH).

## References

  * [1] M. Charles and R. Forty: TORCH: Time of flight identification with Cherenkov radiation, Nucl. Instrum. Meth. A639 (2011) 173 – 176.
  * [2] The LHCb Collaboration, Expression of Interest for a Phase-II LHCb Upgrade, CERN-LHCC-2017-003 (2017).
Physics case for an LHCb Upgrade II, LHCB-PUB-2018-009, CERN-LHCC-2018-027
(2018).

  * [3] I. Adam et al., The DIRC Particle Identification System for the BABAR Experiment. Nucl. Instrum. Meth. A538 (2005) 281 – 357.
  * [4] T. Abe et al., Belle II Technical Design Report’. ArXiv:1011.0352, (2010).
U. Tamponi (on behalf of the BelleII Collaboration), The TOP counter of Belle
II: status and first results. These proceedings.

  * [5] T. M. Conneely, J. S. Milnes, J. Howorth (Photek Ltd), Characterisation and lifetime measurements of ALD coated microchannel plates in a sealed photomultiplier tube, Nucl. Instrum. Meth. A732 (2013) 388 – 391.
  * [6] T. Gys et al., Performance and lifetime of micro-channel plate tubes for the TORCH detector, Nucl. Instrum. Meth. A766 (2014) 171 – 172.
  * [7] T. Gys et al., The TORCH detector R&D: Status and perspectives, Nucl. Instrum. Meth. A876 (2017) 156 – 159.
  * [8] N. Brook et al., Testbeam studies of a TORCH prototype detector, Nucl. Instrum. Meth. A908 (2018) 256 – 268.
  * [9] L. Castillo Garcia et al., Development, characterization and beam tests of a small-scale TORCH prototype module, JINST 11 (2016) C05022.
  * [10] R. Gao et al., Development of TORCH readout electronics for customised MCPs, JINST 11 (2016) C04012.
  * [11] M. Despeisse, F. Powolny, P. Jarron and J. Lapington, Multi-channel amplifier-discriminator for highly time-resolved detection, Nuclear Science, IEEE Transactions 58 (2011) 202 – 208.
  * [12] J. Christiansen, High Performance Time to Digital Converter, CERN/EP-MIC (2002).
  * [13] A. Augusto Alves et al., The LHCb Detector at the LHC, JINST 3 (2008) S08005.


(165) $\displaystyle
t_{\alpha_{1}\alpha_{2}}(x_{1})t_{\beta_{1}\beta_{2}}(x_{2})$
$\displaystyle\sim\frac{1}{x_{12}^{\Delta_{t}}}\frac{\lambda_{ttt}}{2\sqrt{2}}\left(\delta_{\alpha_{1}\beta_{1}}t_{\alpha_{2}\beta_{2}}+\delta_{\alpha_{1}\beta_{2}}t_{\alpha_{2}\beta_{1}}+\delta_{\alpha_{2}\beta_{1}}t_{\alpha_{1}\beta_{2}}+\delta_{\alpha_{2}\beta_{2}}t_{\alpha_{1}\beta_{1}}-\frac{4}{N}\delta_{\alpha_{1}\alpha_{2}}t_{\beta_{1}\beta_{2}}-\frac{4}{N}\delta_{\beta_{1}\beta_{2}}t_{\alpha_{1}\alpha_{2}}\right)$
(166)

Our normalization here agrees with Ref. Chester  _et al._ , 2020. Note, we
have only included the operator $t$ on the RHS of the OPE (166). Also, we did
not include any descendants.

Now, starting from Eq. (100) and performing perturbative RG in $J$,

$\displaystyle\delta S$ $\displaystyle=-\frac{J^{2}}{2}\int
d\tau_{1}d\tau_{2}{\cal
T}\\{O_{\alpha_{1}\alpha_{2}}(\tau_{1})O_{\beta_{1}\beta_{2}}(\tau_{2})\\}t_{\alpha_{1}\alpha_{2}}(\tau_{1})t_{\beta_{1}\beta_{2}}(\tau_{2})\to-\frac{J^{2}\lambda_{ttt}}{\sqrt{2}}\int\frac{d\tau_{1}d\tau_{2}}{|\tau_{1}-\tau_{2}|^{\Delta_{t}}}{\cal
T}\\{O_{\gamma\alpha}(\tau_{1})O_{\gamma\beta}(\tau_{2})\\}t_{\alpha\beta}\left(\frac{\tau_{1}+\tau_{2}}{2}\right)$
(167)

Here ${\cal T}$ stands for time-ordering and we used the OPE (166) in the last
step. Now

$\displaystyle
O_{\gamma\alpha}O_{\gamma\beta}+O_{\gamma\beta}O_{\gamma\alpha}=c(N,N_{b})O_{\alpha\beta}+d(N,N_{b})\delta_{\alpha\beta}$
(168)

where

$\displaystyle c(N,N_{b})$
$\displaystyle=-\frac{1}{2N}(4(N-4)N^{2}_{b}+4(N^{2}-6N+8)N_{b}+N(N^{2}-4N+8))$
$\displaystyle d(N,N_{b})$
$\displaystyle=\frac{1}{N^{2}}(N-2)N_{b}(N_{b}+N-2)(2N_{b}+N-4)(2N_{b}+N)$
(169)

This result can be obtained by using the representation of the impurity in
terms of $b_{\alpha}$, $b^{\dagger}_{\alpha}$ operators discussed in section
VI.1.1. For the $O(3)$ case $N_{b}=S$ \- the spin of the impurity and Eq.
(169) also works for half-integer $S$. Now integrating over small
$\tau_{1}-\tau_{2}$ in (167) and using $\Delta_{t}\approx 1$, we obtain

$\displaystyle\frac{dJ}{d\ell}=\frac{\lambda_{ttt}c(N,N_{b})}{\sqrt{2}}J^{2}$
(170)

We next compute the anomalous dimension of an operator $A$ acting on the
impurity. We will be mostly interested in the $N=3$ case with $A=S_{\alpha}$.
Consider a correlation function with $A$ insertion, $\langle
A(\tau)\ldots\rangle$, where ellipses stand for other operators. Performing
perturbation theory in $J$:

$\displaystyle\langle A(\tau)\ldots\rangle-\langle A(\tau)\ldots\rangle_{0}$
$\displaystyle=+\frac{J^{2}}{2}\int d\tau_{1}d\tau_{2}\big{(}\langle{\cal
T}\\{A(\tau)O_{\alpha_{1}\alpha_{2}}(\tau_{1})O_{\beta_{1}\beta_{2}}(\tau_{2})\ldots\\}\rangle_{0}-\langle{\cal
T}\\{A(\tau)\ldots\\}\rangle_{0}\langle{\cal
T}\\{O_{\alpha_{1}\alpha_{2}}(\tau_{1})O_{\beta_{1}\beta_{2}}(\tau_{2})\\}\rangle_{0}\big{)}$
$\displaystyle\times\langle
t_{\alpha_{1}\alpha_{2}}(\tau_{1})t_{\beta_{1}\beta_{2}}(\tau_{2})\rangle_{0}$
(171)

The expectation values on the RHS are understood to be taken in the $J=0$
theory. From now on, we omit ellipses. Using (165),

$\displaystyle\langle A(\tau)\rangle-\langle
A(\tau)\rangle_{0}=\frac{J^{2}}{2}\int\frac{d\tau_{1}d\tau_{2}}{|\tau_{1}-\tau_{2}|^{2\Delta_{t}}}\big{(}\langle{\cal
T}\\{A(\tau)O_{\alpha\beta}(\tau_{1})O_{\alpha\beta}(\tau_{2})\\}\rangle_{0}-\langle{\cal
T}\\{A(\tau)\\}\rangle_{0}\langle{\cal
T}\\{O_{\alpha\beta}(\tau_{1})O_{\alpha\beta}(\tau_{2})\\}\rangle_{0}\big{)}$
(172)

Now, $O_{\alpha\beta}O_{\alpha\beta}=\frac{1}{2}Nd(N,N_{b})$. Thus, the RHS is
only non-vanishing when $\tau_{1}$ and $\tau_{2}$ are on different sides of
$\tau$:

$\displaystyle\langle A(\tau)\rangle-\langle
A(\tau)\rangle_{0}=J^{2}\int_{\tau}^{\infty}d\tau_{1}\int_{-\infty}^{\tau}d\tau_{2}\frac{1}{|\tau_{1}-\tau_{2}|^{2\Delta_{t}}}\langle[O_{\alpha\beta},A]O_{\alpha\beta}\rangle_{0}\sim
J^{2}\log\Lambda\langle[O_{\alpha\beta},A]O_{\alpha\beta}\rangle_{0}$ (173)

where we’ve isolated the $UV$ divergent part coming from $\tau_{1}$,
$\tau_{2}$ close to $\tau$. Now, let’s take $A=S_{\gamma\delta}$. We have,

$\displaystyle[O_{\alpha\beta},S_{\gamma\delta}]O_{\alpha\beta}=2i[O_{\alpha\gamma},O_{\alpha\delta}]=-\frac{1}{2}(N-2)(2N_{b}+N)(2N_{b}+N-4)S_{\gamma\delta}$
(174)

which can again be obtained using the representation of $O_{\alpha\beta}$ in
terms of $b$, $b^{\dagger}$. Thus, the anomalous dimension

$\displaystyle\Delta_{S_{\alpha\beta}}=\frac{J^{2}}{2}(N-2)(2N_{b}+N)(2N_{b}+N-4)$
(175)

which coincides with (98) for $N=3$.

## References

  * Kitaev  _et al._ (2009) A. Kitaev, V. Lebedev, and M. Feigel’man, AIP Conference Proceedings (2009), 10.1063/1.3149495.
  * Ryu  _et al._ (2010) S. Ryu, A. P. Schnyder, A. Furusaki, and A. W. W. Ludwig, New Journal of Physics 12, 065010 (2010).
  * Hasan and Kane (2010) M. Z. Hasan and C. L. Kane, Rev. Mod. Phys. 82, 3045 (2010).
  * Qi and Zhang (2011) X.-L. Qi and S.-C. Zhang, Rev. Mod. Phys. 83, 1057 (2011).
  * Turner and Vishwanath (2013) A. Turner and A. Vishwanath,  _Topological Insulators: Chapter 11. Beyond Band Insulators: Topology of Semimetals and Interacting Phases_, Contemporary Concepts of Condensed Matter Science (Elsevier Science, 2013).
  * Wen (2017) X.-G. Wen, Rev. Mod. Phys. 89, 041004 (2017).
  * Senthil (2015) T. Senthil, Annual Review of Condensed Matter Physics 6, 299 (2015).
  * Kestner  _et al._ (2011) J. P. Kestner, B. Wang, J. D. Sau, and S. Das Sarma, Phys. Rev. B 83, 174409 (2011).
  * Scaffidi  _et al._ (2017) T. Scaffidi, D. E. Parker, and R. Vasseur, Physical Review X 7 (2017), 10.1103/physrevx.7.041048.
  * Kainaris  _et al._ (2017) N. Kainaris, R. A. Santos, D. B. Gutman, and S. T. Carr, Fortschritte der Physik 65, 1600054 (2017).
  * Chen  _et al._ (2018) C. Chen, W. Yan, C. S. Ting, Y. Chen, and F. J. Burnell, Phys. Rev. B 98, 161106 (2018).
  * Verresen  _et al._ (2018) R. Verresen, N. G. Jones, and F. Pollmann, Phys. Rev. Lett. 120, 057001 (2018).
  * Jiang  _et al._ (2018) H.-C. Jiang, Z.-X. Li, A. Seidel, and D.-H. Lee, Science Bulletin 63, 753 (2018).
  * Parker  _et al._ (2018) D. E. Parker, T. Scaffidi, and R. Vasseur, Phys. Rev. B 97, 165114 (2018).
  * Keselman  _et al._ (2018) A. Keselman, E. Berg, and P. Azaria, Phys. Rev. B 98, 214501 (2018).
  * Jones and Verresen (2019) N. G. Jones and R. Verresen, Journal of Statistical Physics 175, 1164 (2019).
  * Verresen  _et al._ (2019) R. Verresen, R. Thorngren, N. G. Jones, and F. Pollmann, “Gapless topological phases and symmetry-enriched quantum criticality,” (2019), arXiv:1905.06969 [cond-mat.str-el] .
  * Verresen (2020) R. Verresen, “Topology and edge states survive quantum criticality between topological insulators,” (2020), arXiv:2003.05453 [cond-mat.str-el] .
  * Verresen  _et al._ (2017) R. Verresen, R. Moessner, and F. Pollmann, Phys. Rev. B 96, 165124 (2017).
  * Zhang and Wang (2017) L. Zhang and F. Wang, Phys. Rev. Lett. 118, 087201 (2017).
  * Ding  _et al._ (2018) C. Ding, L. Zhang, and W. Guo, Phys. Rev. Lett. 120, 235701 (2018).
  * Weber  _et al._ (2018) L. Weber, F. Parisen Toldin, and S. Wessel, Phys. Rev. B 98, 140403 (2018).
  * Weber and Wessel (2019) L. Weber and S. Wessel, Phys. Rev. B 100, 054437 (2019).
  * Zhu  _et al._ (2021) W. Zhu, C. Ding, L. Zhang, and W. Guo, Phys. Rev. B 103, 024412 (2021).
  * Cardy (1996) J. Cardy,  _Scaling and Renormalization in Statistical Physics_, Cambridge Lecture Notes in Physics (Cambridge University Press, 1996).
  * Diehl (1997) H. W. Diehl, International Journal of Modern Physics B 11, 3503 (1997), cond-mat/9610143 .
  * Metlitski (2020) M. A. Metlitski, “Boundary criticality of the O(N) model in d = 3 critically revisited,” (2020), arXiv:2009.05119 [cond-mat.str-el] .
  * Parisen Toldin (2021) F. Parisen Toldin, Phys. Rev. Lett. 126, 135701 (2021).
  * Hu  _et al._ (2021) M. Hu, Y. Deng, and J.-P. Lv, “Extraordinary-log surface phase transition in the three-dimensional $xy$ model,” (2021), arXiv:2104.05152 [cond-mat.stat-mech] .
  * Matsumoto  _et al._ (2001) M. Matsumoto, C. Yasuda, S. Todo, and H. Takayama, Phys. Rev. B 65, 014407 (2001).
  * Emery and Kivelson (1992) V. J. Emery and S. Kivelson, Phys. Rev. B 46, 10812 (1992).
  * Affleck (1995) I. Affleck, Acta Phys. Polon. B 26, 1869 (1995), arXiv:cond-mat/9512099 .
  * Parcollet  _et al._ (1998) O. Parcollet, A. Georges, G. Kotliar, and A. Sengupta, Phys. Rev. B 58, 3794 (1998).
  * Sachdev  _et al._ (1999) S. Sachdev, C. Buragohain, and M. Vojta, Science 286, 2479 (1999), https://science.sciencemag.org/content/286/5449/2479.full.pdf .
  * Vojta  _et al._ (2000) M. Vojta, C. Buragohain, and S. Sachdev, Phys. Rev. B 61, 15152 (2000).
  * Sachdev (2001) S. Sachdev, Physica C: Superconductivity 357-360, 78 (2001).
  * Sachdev and Vojta (2003) S. Sachdev and M. Vojta, Phys. Rev. B 68, 064419 (2003).
  * Florens  _et al._ (2006) S. Florens, L. Fritz, and M. Vojta, Phys. Rev. Lett. 96, 036601 (2006).
  * Florens  _et al._ (2007) S. Florens, L. Fritz, and M. Vojta, Phys. Rev. B 75, 224420 (2007).
  * Kitaev (2015a) A. Kitaev, “Hidden correlations in the Hawking radiation and thermal noise,” Talk at KITP http://online.kitp.ucsb.edu/online/joint98/kitaev/ (2015a).
  * Kitaev (2015b) A. Kitaev, “A simple model of quantum holography,” Talks at KITP http://online.kitp.ucsb.edu/online/entangled15/kitaev/ and http://online.kitp.ucsb.edu/online/entangled15/kitaev2/ (2015b).
  * Maldacena and Stanford (2016) J. Maldacena and D. Stanford, Phys. Rev. D 94, 106002 (2016).
  * Sachdev (2015) S. Sachdev, Phys. Rev. X 5, 041025 (2015).
  * Gu  _et al._ (2020) Y. Gu, A. Kitaev, S. Sachdev, and G. Tarnopolsky, Journal of High Energy Physics 2020, 157 (2020).
  * Georges  _et al._ (2001) A. Georges, O. Parcollet, and S. Sachdev, Phys. Rev. B 63, 134406 (2001).
  * Moshe and Zinn-Justin (2003) M. Moshe and J. Zinn-Justin, Physics Reports 385, 69 (2003).
  * Chubukov and Sachdev (1993) A. V. Chubukov and S. Sachdev, Phys. Rev. Lett. 71, 169 (1993).
  * Sachdev (2011) S. Sachdev,  _Quantum Phase Transitions_ (Cambridge University Press, 2011).
  * Note (1) Recall that we are already considering an extended problem where fermion operators are physical.
  * Note (2) Strictly speaking, these correlation functions are ill-defined since the Majorana operators are fictitious. They should really be thought of as some bilocal fields in the path integral language and we are interested in their saddle point values from which we can compute physical correlation functions.
  * Wang  _et al._ (2006) L. Wang, K. S. D. Beach, and A. W. Sandvik, Phys. Rev. B 73, 014431 (2006).
  * Chester  _et al._ (2020) S. M. Chester, W. Landry, J. Liu, D. Poland, D. Simmons-Duffin, N. Su, and A. Vichi, (2020), arXiv:2011.14647 [hep-th] .
  * Note (3) We thank Shai Chester for explaining the normalization conventions in Ref. O3bootstrap to us.
  * Note (4) We’ve normalized $O_{\alpha\beta}$ so that it reduces to (93) for $N=3$.
  * Gracey (2002) J. A. Gracey, Phys. Rev. E 66, 027102 (2002).
  * Höglund  _et al._ (2007) K. H. Höglund, A. W. Sandvik, and S. Sachdev, Phys. Rev. Lett. 98, 087203 (2007).
  * Höglund and Sandvik (2007) K. H. Höglund and A. W. Sandvik, Phys. Rev. Lett. 99, 027205 (2007).
  * Note (5) We were in fact already doing so when looking for large-$N$ solutions to these two models earlier.
  * Note (6) In Model 2A, the operator $S_{\alpha\beta}\propto\mathrm{i}\gamma_{\alpha}\gamma_{\beta}$ couples directly to the bulk $\phi$ fields. When $N=3$, $S_{\alpha}$ is defined by $S_{\alpha}=\frac{1}{2}\epsilon^{\alpha\beta\gamma}S_{\beta\gamma}$ and is proportional to $\mathrm{i}\gamma_{0}\gamma_{\alpha}$ in either of the fermion-number parity sectors.
  * Jian  _et al._ (2020) C.-M. Jian, Y. Xu, X.-C. Wu, and C. Xu, arXiv preprint arXiv:2004.07852 (2020).
  * Note (7) One can also regularize both $D(\mathrm{i}\omega)$ and $\Sigma(\mathrm{i}\omega)$ near $\omega=0$ in a way compatible with (145) and that both integrands become smooth. As a result, values of the two integrations will be different from our choice, although the sum of them is still be same.


# Detect Only What You Specify : Object Detection with Linguistic Target

Moyuru Yamada
Fujitsu Laboratories Ltd.
4-1-1 Kamikodanaka, Nakahara-ku, Kawasaki, Kanagawa 211-8588, Japan
<EMAIL_ADDRESS>
###### Abstract

Object detection is a computer vision task of predicting a set of bounding
boxes and category labels for each object of interest in a given image. The
category is related to a linguistic symbol such as ’dog’ or ’person’ and there
should be relationships among them. However the object detector only learns to
classify the categories and does not treat them as the linguistic symbols.
Multi-modal models often use the pre-trained object detector to extract object
features from the image, but the models are separated from the detector and
the extracted visual features does not change with their linguistic input. We
rethink the object detection as a vision-and-language reasoning task. We then
propose targeted detection task, where detection targets are given by a
natural language and the goal of the task is to detect only all the target
objects in a given image. There are no detection if the target is not given.
Commonly used modern object detectors have many hand-designed components like
anchor and it is difficult to fuse the textual inputs into the complex
pipeline. We thus propose Language-Targeted Detector (LTD) for the targeted
detection based on a recently proposed Transformer-based detector. LTD is a
encoder-decoder architecture and our conditional decoder allows the model to
reason about the encoded image with the textual input as the linguistic
context. We evaluate detection performances of LTD on COCO object detection
dataset and also show that our model improves the detection results with the
textual input grounding to the visual object.

##  1 Introduction

Figure 1: Language-targeted detector (LTD) detects all the target objects in
an image (top). Multiple targets can be given by the natural language
(middle). The detection model can also learn special tokens to detect specific
objects, e.g., ’[all]’ for all objects of interest (bottom).

Object detection is a computer vision task of predicting a set of bounding
boxes and category labels for each object of interest in a given image [1].
Each category is related to a linguistic symbol such as ’dog’ or ’person’, and
there should be relationships among them. However the object detector only
learns to classify the categories and does not treat them as the linguistic
symbols. On the other hand, humans can ground the linguistic symbols to the
visual objects and detect the various objects targeted by a natural language.
We might also change the way of reasoning depending on what we detect, i.e.,
detection target. For example, if we are asked to count the number of dogs, we
focus on dog-like shapes, or if we need to distinguish between sheep and cows,
we pay attention to the differences. From the above aspect, we rethink the
object detection as a vision-and-language reasoning task. As a first step, we
propose targeted detection task, where detection targets are given by a
natural language as a linguistic context and the goal of the task is to detect
only all the target objects in a given image.

On the targeted detection, grounding the linguistic targets to the visual
objects in a detection pipeline is a key challenge. Commonly used modern
object detectors [7, 24] adopt multi-stage structure and have many hand-
designed components like anchors [17]. It is difficult to fuse the textual
inputs into the complex pipeline. A recently proposed Transformer-based
detector [1] has significantly simplified the pipeline. However there are no
report how to design a multi-modal Transformer for the object detection.

The targeted detection is similar to referring expressions, which are natural
language utterances that indicate particular objects within a scene, e.g.,
”the dog on the sofa” [29]. Kazemzadeh _et al_. have proposed referring
expression comprehension [13], which is a task to localize the object
mentioned by the given natural referring expression. However this task is
typically formulated as selecting the best region from a set of proposals in
an image with a textual input [20, 29]. Thus, this task does not fit for
detecting multiple objects in an image, e.g., two dogs might be on the sofa.
Moreover, since most works use a pre-trained object detector to generate the
region proposals from the image [3, 16], the detector does not treat the
textual input.

We propose Language-Targeted Detector (LTD) for the targeted detection task,
as shown in Fig. 1. A main feature of LTD is to have a textual input for
specifying what objects to detect. Thus, there are no detection if the target
is not given as the input. Unlike the referring expression comprehension, the
targeted detection task is required to detect multiple objects with multiple
targets given by the textual input. Instead of using the pre-trained object
detector, we train our detection model with the textual input in end-to-end
manner. We evaluate LTD on the most popular object detection dataset, COCO
[2]. We show that 1) our detection model dynamically changes detected objects
depending on the targets given in a natural language, and 2) by using the
textual input as a context, our model improves the detection results compared
to the normal object detectors. We believe the targeted detection task has the
promising potential that allows the detectors to consider more complex
contexts on the vision-and-language reasoning tasks.

Our contributions are as follows:

  * •

We propose targeted detection task, where detection targets are given by a
natural language as a linguistic context and the goal of the task is to detect
only all the target objects in a given image.

  * •

We then propose a new object detector with a textual input for the task, and
the model demonstrates that the given linguistic targets change how the
detector attends to the encoded image and improve the object detection.

Figure 2: Overview of Language-Targeted Detector (LTD). LTD uses a CNN
backbone to extract a 2D representation of an input image. It is fed into a
Transformer encoder with a positional encoding. The encoder outputs the final
representation of the visual features. A conditional Transformer decoder then
takes a fixed number of object queries and target queries as an input
sequence, and attends to the encoder outputs. The object queries are
placeholders for detecting objects. The target queries are textual tokens
which condition the detected objects. Shared prediction heads predict a set of
bounding box, object category, and target index from an output of the decoder.

##  2 Related Work

###  2.1 Object Detection

After the success of deep learning on image classification [8, 14], deep
learning has been applied to object detection and many studies have
continuously improved their detection performances [7, 24]. Deep learning-
based object detectors use deep convolutional neural network (CNN) [8, 10, 25]
to extract visual features from an image. These object detectors can be
categorized into two-stage, one-stage, and end-to-end ones.

Two-stage detectors such as Faster R-CNN [24] suffer from high computational
cost but achieve higher accuracy than one-stage ones, and thus their pre-
trained model is often used to extract regional object features from the image
for vision-and-language reasoning tasks. Since one-stage detectors such as
YOLO [23] and SSD [18] are faster than the two-stage ones, they fit for the
real-time detection. Both two-stage and one-stage methods need complicated and
hand-designed post-processing to generate the final bounding box predictions.

Recently, Transformer-based object detection method [1] has proposed and it
has significantly simplified object detection pipeline by removing the hand-
crafted anchor and non-maximum suppression. The Transformer-based detector
directly predicts a set of bounding box and object category end-to-end with
Hungarian bipartite matching. It is a transformer encoder-decoder architecture
and can reason about the relations of objects in the image by the attention
mechanism.

These object detectors are single-modal model and take only an image as an
input. They cannot alter detection targets after the training with defined
object categories. Unlike them, we propose a targeted detector which enables
the model to alter the detection targets by giving the targets in natural
language as a linguistic context.

###  2.2 Multi-Modal Transformer

Recent studies have shown significant improvements in natural language
processing (NLP) tasks with a multi-layer Transformer [28], such as GPT [22],
BERT [4], and RoBERTa [19]. Motivated by these promising results, the multi-
layer Transformer has been utilized in multi-modal models for vision-and-
language tasks. UNITER [3] and Unicoder-VL [16] proposed a single-stream
architecture like BERT, which fuses two modalities. ViLBERT [20] consists of
two single-modal streams, and has a language Transfomer followed by a cross-
modal Transformer. These Transformer-based multi-modal models outperform
conventional state-of-the-art models [5, 15] on most vision and language tasks
such as VQA and NLVR.

The multi-modal Transformer takes both visual features and textual tokens as
input sequences. These input sequences are encoded to the image embeddings and
linguistic embeddings. These embeddings are then fed into the multi-layer
Transformer as an encoder. Then, they treat the above multi-modal tasks as a
classification task and output answer probabilities based on the final hidden
state of the Transformer encoder. Most works use a frozen pre-trained object
detector to extract the regional visual features [3, 16, 20], and the object
detectors does not consider the linguistic inputs. Consequently, the final
performances on the down-stream tasks depend on the extracted visual features.
Therefore, instead of using outputs of the object detector, several works
input grid features [12] or pixel features [11] directly obtained from the
stacked CNN layers to the Transformer encoder. Transformer is also used for
image captioning [9]. This model is a encoder-decoder architecture and has a
decoder to output a caption of a given image in addition to the encoder for
the image. We treat object detection as a vision-and-language task and propose
a multi-modal decoder that conditions the detected objects by a textual input.

###  2.3 Referring Expression Comprehension

The referring expressions are natural language utterances that indicate
particular objects within a scene, e.g., ”the dog on the sofa” or “the book on
the table”. Kazemzadeh _et al_. have proposed referring expression
comprehension [13], which is a task to localize the object mentioned by the
given natural referring expression. To solve this task, some work uses CNN-
LSTM structure [21, 29] to encode visual and linguistic inputs. The multi-
layer Transformer is also used as a multi-modal encoder and has achieved
state-of-the-art result [3, 20]. However these works formulate this task as
selecting the best region from a set of proposals with a textual input
expression and use the pre-trained object detector to generate the region
proposals from a given image. Thus, the object detector is separated from the
textual input. Moreover, this task does not fit for detecting multiple objects
in the image.

##  3 Language-Targeted Detector

In this section, we first describe the architecture and overall pipeline of
Language-Targeted Detector (LTD) and then introduce a conditional decoder.

Figure 3: Conditional decoder with linguistic target. The decoder is $L$
stacks of a decoder layer composed of a self-attention layer and a target-
attention layer. Each decoder layer decodes the $N$ objects with the $K$
textual tokens in parallel. The self-attention layer transforms both object
queries and target queries to ground the textual tokens to the visual objects.
The target-attention only treats the object queries as its attention queries.
The model predicts box coordinates, class labels, and target index using only
the final output of the object queries.

###  3.1 Model Architecture

An architecture overview of Language-Targeted Detector (LTD) is shown in Fig.
2. LTD is mainly consists of a ResNet backbone, a Transformer encoder, a
conditional Transformer decoder, and shared prediction heads. Most parts of
the architecture inherit the Transformer-based object detector, DETR [1]. A
main feature of LTD is the conditional decoder, which will be described in the
section 3.2.

LTD uses a Convolution Neural Network (CNN) to extract visual feature maps
$f\in\mathbb{R}^{C{\times}H{\times}W}$ from an input image
$I\in\mathbb{R}^{3{\times}H_{0}{\times}W_{0}}$ (3 color channels), where $H$,
$W$ and $H_{0}$, $W_{0}$ are the height and width of the input image and the
visual feature map, respectively. $C$ is an output channel of the visual
feature map.

The model flattens the extracted visual features and compresses the channel
$C$ into a smaller dimensional representations $z\in\mathbb{R}^{d{\times}L}$
by 1$\times$1 convolution to suppress the computational cost, where $d$ is a
dimension of the input visual feature $z_{0}$ and $L$ is its length ($L=HW$).
The model collapse the spatial information of the 2D visual representation by
this transformation, hence the model supplements it with a positional encoding
before passing it into a Transformer encoder.

The encoder is a multi-layer Transformer. Each encoder layer has a standard
Transformer block and consists of a multi-head self-attention module and a
feed forward network (FFN). The encoder outputs the final representation of
the visual features.

The decoder is a stack of a self-attention layer and a target-attention layer.
Each self-attention and target-attention layer are a standard Transformer
architecture using multi-head attention mechanisms. Unlike DETR, our decoder
takes target queries as an additional linguistic input to condition detection
targets, in addition to the object queries. We design the conditional decoder
to ground the target texts (e.g., ’dog’) to the visual objects. The textual
input is encoded to the token embeddings using a pre-trained BERT embeddings
[4].

The model have three shared prediction heads to compute the final prediction
of a bounding box, a class category, and a target index. The target index
indicates which object instance belongs to which target text. The bounding box
is predicted by a 3-layer perceptron with ReLU activation function, whereas a
linear projection layer and softmax function are used to predict the class
category and the target index, respectively.

Figure 4: Data preparation procedure for the targeted detection. Target
instances $\\{o_{1},o_{2},...,o_{R}\\}$ that belong to the randomly sampled
$S$ categories are extracted from the original annotation for the image $I$.
The category ID of each target object is converted to its name to obtain the
target text $T$. Target index indicates which target instance belongs to which
target text. We can simply choose one instance with its text annotation (i.e.,
class label or description) as the target instance in the image if the target
dataset has no class category annotations. Figure 5: Examples of data
conversion from the object detection to the targeted detection.

###  3.2 Conditional Decoder

A challenge in designing language-targeted detector is how to control the
objects to be detected by a linguistic input. To fuse the visual input and the
linguistic input, most previous studies on the multi-modal Transformer have
extended the encoder to allow both inputs [16, 20] as described in Sec 2.2.
Unlike these studies, we proposed a multi-modal decoder which conditions the
detected objects by the linguistic target, as shown in Fig. 3. This design is
based on the following two reasons.

First, looking back on the Transformer-based object detector, the authors
hypothesise that the encoder separates object instances in an image via global
attention and simplifies object extraction and localization for the decoder,
then the decoder only needs to attend to the object extremities such as heads
or legs to extract the class and object boundaries [1]. We suppose that when
the target object changes, the boundary of each object instance might not
change, but the boundaries of interest change. In addition, the linguistic
targets should be ground to the visual features at the object level not the
pixel level to control the detected objects efficiently because the pixel
level grounding is more difficult than the object level [11, 27].

Then, a disadvantage of extending the encoder is the increase in a
computational cost. The computational cost of the Transformer attention
mechanisms is highly depend on a length of an input sequence. Compared with
most multi-modal Transformer which treats object features with a typical
maximum sequence length of 100 [3, 20] as the visual inputs, the encoder of
Transformer-based detector takes much longer sequence (over 1500) as the input
[1]. Thus, we avoid further computational load due to the additional textual
input for the encoder since the encoder is computationally intensive
component.

The decoder is $L$ stacks of a decoder layer composed of a self-attention
layer and a target-attention layer. Each decoder layer decodes the $N$ objects
with the $K$ textual tokens in parallel. Objects are first encoded to the
object queries with object placeholders and query embeddings. The object
placeholder is a vector filled with zero value. Since the decoder is
permutation-invariant, we add query embeddings as learnt positional encodings
to produce different results for each object query. The object queries are sum
of both. The textual input tokens always have a special head token (`[CLS]`).
We separate different targets with another special token (`[SEP]`). The input
tokens are first converted to its ID with a pre-trained tokenizer and then
encoded to the target queries with token embeddings, segment embeddings, and
the shared query embeddings. We use a pre-trained BERT embeddings as the token
embeddings. The segment embeddings are a learned embedding indicating which
target it belongs to. The query embeddings shared with the object queries are
then added to obtain the target queries.

The self-attention layer treats both the object queries and the target queries
to reason about the relations of objects in the image and ground the
linguistic targets to the target objects. Target-attention layer extracts the
visual features to correctly predict the objects. Here, the model needs to
control the objects of interest depending on the linguistic target in the
decoder, thus our decoder only takes the object queries as an input of the
target-attention. This decoder design contributes to suppress the
deterioration of the computation speed. Some study have reported the slow
convergence of DETR [6, 26]. From preliminary experiments, this design has
also benefited this problem.

The model finally predicts box coordinates, class labels, and target indexes
using only the final output of the $N$ object queries.

##  4 Learning Targeted Detection

In this section, we will describe our approach to learn the targeted detection
task, where the detection targets are given by a natural language and the goal
of the task is to detect only all the target objects in a given image.

Model | Detection target | AP | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$
---|---|---|---|---|---|---|---
Faster RCNN-FPN | all | 40.2 | 61.0 | 43.8 | 24.2 | 43.5 | 52.0
DETR | 42.0 | 62.4 | 44.2 | 20.5 | 45.8 | 61.1
LTD-IDX-PT | 42.4 | 62.1 | 45.2 | 21.1 | 46.0 | 60.9
LTD-PT | 42.2 | 62.2 | 44.9 | 20.9 | 46.1 | 60.5
LTD | 40.6 | 60.9 | 43.0 | 19.6 | 43.5 | 59.5
LTD-IDX-PT | targeted only | 47.4 | 70.1 | 50.1 | 25.0 | 50.4 | 66.5
LTD-PT | 47.1 | 69.3 | 49.9 | 24.8 | 50.1 | 66.2
LTD | 45.8 | 69.0 | 48.1 | 24.2 | 48.4 | 66.1
Table 1: Performance comparison of the object detection on the COCO validation
set. The top section shows comparison with Faster R-CNN and DETR with the same
ResNet-50 backbone. We give the special token to our model to detect all the
objects of interest in this comparison. The bottom section shows the targeted
detection performances. -PT denotes the model initialized with the pre-trained
weights of DETR and -IDX denotes the model trained with the target index
prediction.  LTD-IDX-PT
---
AP | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$
38.4 | 58.4 | 41.4 | 18.9 | 46.3 | 76.6
Table 2: Performance of the target index prediction on the COCO validation
set. The model predicts the target index of each detected object, i.e., which
detected instance belongs to which target text.

To train a model for the targeted detection, we need to prepare an input image
$I$ with an target text $T$ and box coordinate $B$, target index $E$ of
targeted objects. To make these data for training and evaluation, we use the
most popular object detection dataset, MS COCO. As showing in Fig. 4, the
detailed procedure of the preparation are as follows.

1) The number of category samples $S$ $(1\leq S\leq M)$ is randomly choose,
where $M$ is the total number of categories in the image $I$.

2) We randomly sample $S$ categories from the list of categories. The
categories $\\{c_{1},c_{2},...,c_{s}\\}$ are the targets.

3) We then extract all instances that belong to the target categories from the
original annotation for the image $I$. Because there might be multiple
instances that belong to the same category. These extracted instances
$\\{o_{1},o_{2},...,o_{R}\\}$ are the target objects to be detected, where $R$
is the number of target objects ($R\geq S$).

4) To obtain the input target text $T$, the category ID of each target object
is converted to its name (e.g., `16` $\rightarrow$ ‘`bird`’). The obtained
target texts are then tokenized as described in Sec. 3.2.

5) Finally, target index is added to the labels to indicate which target
instance belongs to which target text.

The class label of the each targeted object is optional. In the steps 1) - 3),
we simply choose one instance with its text annotation (i.e., class label or
description) as the target instance in the image if the target dataset has no
class annotations. Thus, the targeted detection can be applied to not only
object detection datasets such as MS COCO but also other vision-and-language
datasets such as Visual Genome, which have images with regional descriptions
but does not have class annotations.

Figure 5 shows a comparison of normal object detection and our targeted
detection. Only $R$ instances in the images must be detected with the target
text $T$ for the targeted detection task, whereas all instances of interest
are the targets to be detected for the object detection task.

LTD can also learn special tokens to detect specific targets. In this paper,
we use a special `[all]` token to detect all objects of interest in the
original annotation of the object detection (i.e., the input tokens are
`[[CLS],[all],[SEP]]`).

##  5 Experiments

In this section, we first detail the experimental settings and then show that
LTD learns to detect targeted object with the textual input as the context. We
also analyze the conditional decoder of the LTD to understand the behaviors.

###  5.1 Experimental Settings

Dataset. We validate our proposed LTD on COCO 2017 detection dataset, which
contains 118k training images and 5k validation images, respectively. We
report mAP for performance evaluation following previous research [1]. We make
the target texts and the labels to be predicted from the original dataset as
described in Sec. 4. When there is no annotated object in a given image, the
target text is none and the input tokens are set to `[[CLS],[SEP]]`. To
compare the detection performances with the normal object detectors, we set a
special `[all]` token as the target text and simply use the all objects of
interest as the targets in the image with a probability of 50%.

Implementation details. We follow the experimental settings in the original
DETR [1]. The number of the encoder layers and the decoder layers is 6 each.
We use a pre-trained ResNet-50 as our backbone and an extracted 2048
dimensional visual feature is compressed to 256 dimensional input to the
encoder. Our evaluation model have 100 object queries and 40 target queries.
We use a pre-trained BERT-BASE tokenizer and embeddings to obtain the token
embeddings from the target text. The final output of each object query is used
for predicting the bounding box, the class category, and the target index. The
model is trained by minimizing a L1 loss of the bounding box and a cross-
entropy classification loss of the class category and the target index. The
learning rate is set as $10^{-4}$ for the Transformer encoder-decoder and
$10^{-5}$ for the pre-trained ResNet backbone and optimized by AdamW
optimizer. All models are trained for 300 epochs on 32 V100 GPUs with 16GB
memory within 3 days.

Figure 6: LTD dynamically changes the detected objects depending on the target
text. LTD can detect all objects with a special token (left) and a specific
target (center). Unlike just filtering the detected objects, our model detects
the objects in consideration of the linguistic context (left vs right). With
the target text of ’bowl’, LTD detects more bowls because the model targets
them.  Figure 7: DETR misclassifies the objects in the foreground (left, two
sheep are misclassified as cows). LTD also outputs the same incorrect results
when we give the special token to detect all objects (center). On the other
hand, LTD can also use the target text as the context (i.e., the targets are
sheep or cows) to lead the correct answer (right).

###  5.2 Detection Performances

We first compare the accuracy with Faster R-CNN and DETR to investigate the
effect of the textual input. Table 1 shows the results, where all models use
the same ResNet-50 backbone. -PT denotes the model initialized with the pre-
trained weights of DETR. -IDX denotes the model trained with the target index
classification. We give the special token `[all]` to detect all the objects of
interest in this comparison (denoted as ’all’ in the Detection target column).
The results show that our models achieve comparable performances to the normal
object detection models and no significant performance drop by the textual
input. The initialization with the pre-trained weights of DETR is benefit for
our model to speed up the training.

The bottom section in Table 1 shows the accuracy on the targeted detection
task. The performances on this task are significantly better than the ones on
the normal object detection on all metrics. Note that LTD learns to detects
all objects of interest with the special token and does not simply refer the
target text to classify the objects. Table 2 shows the accuracy of the target
index prediction on the targeted detection. The model predicts which detected
instance belongs to which target text. This prediction seems to be more
difficult than the category classification because grounding the target text
to the visual object is required.

As shown in Fig. 6, LTD dynamically changes the detected objects depending on
the target text. LTD can detect all objects with the special token (left) and
specific targets (center). Unlike just filtering the detected objects, our
model detects the objects in consideration of the linguistic context (left vs
right). With the target text of ’bowl’, LTD detects more bowls because the
model targets them. Figure 7 shows that our detection model improves the
detection results using the target text as the linguistic context. DETR
misclassifies the objects in the foreground (left, two sheep are misclassified
as cows). LTD also outputs the same incorrect results when we give the special
token to detect all objects (center). On the other hand, when we give the
target text as the context (i.e., the targets are sheep or cows), our model
can use the context to lead the correct answer (right).

###  5.3 Analysis of Conditional Decoder

Figure 8: Each column represents a object query (queries 0, 16, 40, 42, 48th).
With a special input token ’[all]’, LTD detects five objects, which correspond
to the bottom five pictures and the five images above them that represent the
predicted bounding box and the attention map of each object query. The text
below each image denotes the predicted class category and the objects are not
detected when there is no text there. When we give the target text ’cat’, the
decoder seems to attend to cat-like shapes. Figure 9: Self-attention weights
of the object queries (0, 16, 40, 42th) when we give the target text ’cat’.
The 40th and 42th queries strongly attend to the target queries ($\geq$ 100th)
and they are detected as the cat.

The main feature of LTD is the conditional decoder described in 3.2. To
understand its behavior, we visualize the target-attention map of the last
decoder layer in Fig. 8 and the self-attention map of the middle decoder layer
in Fig. 9, respectively.

In Fig. 8, each column represents an object query extracted from a total of
100 queries (here queries 0, 16, 40, 42, 48th). Each object query learns to
specialize on certain areas and box sizes [1]. In this example, with a special
input token `[all]` to detect all objects, LTD detects five objects, which
correspond to the bottom five pictures and the five images above them that
represent the predicted bounding box and the attention map of each object
query. The text below each image denotes the predicted class category. The
objects are not detected when there is no text there. The attention maps show
that each query mostly attends to the object extremities and boundaries. Our
decoder has the linguistic target and thus the attention maps change depend on
the target text. When we give the target text `cat`, the decoder seems to
attend to cat-like shapes because the attention map of 0th and 16th queries
looks similar to those of 16th and 42th queries which are classified as a cat.

Figure 9 shows the attention weights between the object queries and target
queries when we give the target text `cat`. Each image correspond to the
attention weight of each query (0, 16, 40, 42th). The 40th and 42th queries
strongly attend to the target queries ($\geq$ 100th) and they are detected as
the cat. We hypothesize that by using the linguistic targets grounding to the
objects, each object query can transform itself into a representation suitable
for detecting the target, and also they can check whether they are the target.

##  6 Conclusions

We introduced the linguistic contexts into the object detection and proposed
targeted detection task, where detection targets are given by natural language
and the goal of the task is to detect only all the target objects in an image.
We also proposed Language-Targeted Detector (LTD) for the proposed task based
on the recently proposed Transformer-based detector. We evaluate LTD on COCO
object detection dataset and show that our model improves the detection
performances using the target text grounding to the visual objects as the
linguistic context. Unlike the conventional uni-modal detectors, the
conditional decoder of LTD allows the model to reason about the encoded image
with the linguistic targets and dynamically control the detected objects. We
expect the object detector with the linguistic context is a new direction for
the vision-and-language reasoning tasks.

## References

  * [1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, volume 12346 of Lecture Notes in Computer Science, pages 213–229. Springer, 2020\.
  * [2] Xinlei Chen, Tsung-Yi Lin Hao Fang, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server, 2015.  arXiv preprint arXiv:1504.00325.
  * [3] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning, 2019.  arXiv preprint arXiv:1909.11740.
  * [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186, 2019.
  * [5] Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven C. H. Hoi, Xiaogang Wang, and Hongsheng Li.  Dynamic fusion with intra- and inter-modality attention flow for visual question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [6] Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li.  Fast convergence of detr with spatially modulated co-attention.  2021\.  arXiv preprint arXiv:2101.07448.
  * [7] R. Girshick.  Fast r-cnn.  In IEEE International Conference on Computer Vision (ICCV), pages 1440–1448, 2015.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [9] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares.  Image captioning: Transforming objects into words.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, 2019.
  * [10] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [11] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu.  Pixel-bert: Aligning image pixels with text by deep multi-modal transformers.  CoRR, abs/2004.00849, 2020.
  * [12] Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen.  In defense of grid features for visual question answering.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [13] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.  ReferItGame: Referring to objects in photographs of natural scenes.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 787–798, Doha, Qatar, Oct. 2014\. Association for Computational Linguistics.
  * [14] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  Commun. ACM, 60(6):84–90, 2017.
  * [15] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He.  Stacked cross attention for image-text matching.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [16] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
  * [17] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In IEEE International Conference on Computer Vision (ICCV), pages 2999–3007, 2017.
  * [18] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg.  SSD: single shot multibox detector.  In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
  * [19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.  arXiv preprint arXiv:1907.11692.
  * [20] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In Advances in Neural Information Processing Systems 32, pages 13–23. 2019.
  * [21] Ruotian Luo and Gregory Shakhnarovich.  Comprehension-guided referring expressions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [22] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  2018\.
  * [23] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [24] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 39, 2015.
  * [25] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
  * [26] Zhiqing Sun, Shengcao Cao, Yiming Yang, and Kris Kitani.  Rethinking transformer-based set prediction for object detection.  2020\.  arXiv preprint arXiv:2011.10881.
  * [27] Hao Tan and Mohit Bansal.  LXMERT: Learning cross-modality encoder representations from transformers.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100–5111, 2019.
  * [28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems 30, pages 5998–6008. 2017.
  * [29] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L. Berg.  Mattnet: Modular attention network for referring expression comprehension.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [30] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In CVPR 2021, 2021.

## Appendix

##  Appendix A Dataset Statistics

Figure 10: Total number of instances per category. Figure 11: Ratio of instances per category of object detection and targeted detection. Figure 12: Ratio of the number of instances per target on targeted detection. | Object | Targeted
---|---|---
detection | detection
Validation images | 5,000 | 5,000
Total annotated instances | 36,781 | 24,790
Categories per image | 2.93 | 1.97
Instances per image | 7.36 | 4.96
Instances per category | 2.51 | 2.51
Table 3: Main properties of COCO object detection dataset and our targeted
detection for validation. We randomly sample the object instances from the
original annotation on the object detection, thus the number of instances and
categories are slightly decreased.

We show the properties of the validation data for the target detection task.
The training and evaluation data are both based on COCO object detection
dataset [2]. The data preparation procedure is described in Sec. 4 in the main
paper.

Table 3 shows comparisons of the main properties of COCO object detection
dataset and our targeted detection. Both are validation data. Original COCO
object detection dataset has 36,781 annotated instances on the validation
data, whereas our targeted detection data has 24,790 instances. We randomly
sample the object instances from the original annotation on the object
detection, thus the number of instances and categories are slightly decreased
(about 33%). Both data contains the same 5,000 images for the validation. On
average COCO dataset contains 2.93 categories and 7.36 instances per image.
Our targeted detection data contains 1.97 categories and 4.96 instances per
image. The number of instances per category are almost same for both data
(2.51).

Figure 10 shows the total number of the instances per category and Fig. 11
shows the ratio of the instances per category of object detection and targeted
detection. These results mean that the distribution of the instances in each
category does not change from the original COCO dataset and the number of
’person’ instances is very large compared to others.

Figure 12 shows ratio of the number of instances per target on our targeted
detection data. Most images contains 5 or less instances for each target and
about 58% of the targets are 1 instance. Only one image contains 23 instances
for a specific target.

##  Appendix B Object Detection Conditioned on Context

A goal of the language-targeted detector is to improve the object detection
accuracy with a linguistic context, like an object label, a referring
expression, or a question. Current SOTA multi-modal models typically use a
pre-trained object detector and their performances heavily rely on the
detector [30]. The detector is trained to avoid misdetection but whether the
confusing object (e.g., small or occluded object) should be detected depends
on the context. For instance, if the model is asked about a dog, the detector
should extract the dog-like objects even if they are blurry. However the
traditional detector extracts regions of interest from an image regardless of
the given linguistic context and this makes it challenging to detect the
confusing object.

As a first step, we have proposed a new object detector that detects objects
conditioned on target labels and investigated how the linguistic context helps
the object detection.

##  Appendix C Comparison with Traditional Detector

Although a main advantage of our language-targeted detector (LTD) is to detect
objects in an image conditioned on target texts, in ’all’ setting in Table 1,
we give a constant target text ’[all]’ regardless of the input image to verify
the baseline accuracy of LTD. This setting means our model is not conditioned
on the target text, thus the performances in this setting must be comparable
with DETR. [6] reports that the DETR suffers from slow convergence. Our LTD
are based on the DETR and needs longer training epoch than the original DETR.
Thus, we use a pre-trained weights of DETR. Without it (LTD in Table 1), the
performances slightly drop due to lack of the training.

LTD detects the objects conditioned on target texts. In ’targeted only’
setting in Table 1, LTD uses the given target text to change how the detector
attends to the encoded image (Fig. 8) and thus LTD improves the object
detection accuracy. Unlike just filtering regions output by the traditional
detector, with the target text (e.g., ’bowl’), LTD can detect small objects
that are missed without conditioning (Fig. 6). Note that the object instances
in the image are randomly sampled (line 503-520) and the category distribution
is the same as one of the original dataset (Fig. 11 in Appendix). We do not
choose the easy instances.

In the training, we randomly set ’[all]’ token as the target text to detect
all objects of interest in the image with a probability of 50% (see 5.1).
Thus, LTD does not just use the target text as a hint to predict a class
category. In Fig. 7, without conditioning (i.e., target text is ’[all]’), LTD
misclassifies the sheep as cow. When the target texts are ’sheep’ and ’cow’,
although these cannot be a direct hints, LTD uses the textual input and
correctly predicts the object categories (see also Fig. 9).

Thank you for suggesting several tests to verify our LTD. We have already
conducted some of them. We would like to add those results in the final paper.
First, in ’targeted only’ setting, we add deceptive targets that does not
appear in the image. The number of the deceptive targets are set to
$max(1,N*r)$ where $N$ is the number of the ground-truth targets and $r$ is
rate. The accuracy dropped by around 1 point with 10% rate. There is no
further drop at 20%. Then, in ’all’ setting, we give all ground-truth targets
that appear in each image as the context. The accuracy increases by around 5
points and reaches the same as performances in ’targeted only’ setting because
of the same data distribution.

##  Appendix D Experimental Settings and Others

###  D.1 Target texts

Target text such as ’dog’ or ’bird’ is sampled from the ground-truth
annotations (see Fig.4). We described the detection performances with the
deceptive target that does not appears in the ground-truth above.

###  D.2 Training objectives

Prediction of the class category is optional but we use it during the training
of LTD because it improve the convergence speed. We use the class category in
COCO for performance comparisons. The total loss can be written as follows:

$L=k_{C}L_{C}+k_{B}L_{B}+k_{I}L_{I}$ (1)

where $L_{n},k_{n}(n={C,B,I})$ are a loss and a coefficient of the class
category, the bounding box, and the target index.

###  D.3 Generalization of model

To easily extend LTD for referring expression comprehension on Flickr30K
Entities and Visual Genome, we use BERT to encode the textual input to token
embeddings (see 3.2). LTD can take not only the object labels but also longer
text like sentences as an input.