# Excited-State Downfolding Using Ground-State Formalisms

Nicholas P. Bauman<EMAIL_ADDRESS>Physical Sciences Division,
Pacific Northwest National Laboratory, Richland, WA, 99352, USA

###### Abstract

Downfolding coupled cluster (CC) techniques are powerful tools for reducing
the dimensionality of many-body quantum problems. This work investigates how
ground-state downfolding formalisms can target excited states using non-Aufbau
reference determinants, paving the way for applications of quantum computing
in excited-state chemistry. This study focuses on doubly excited states for
which canonical equation-of-motion CC approaches struggle to describe unless
one includes higher-than-double excitations. The downfolding technique results
in state-specific effective Hamiltonians that, when diagonalized in their
respective active spaces, provide ground- and excited-state total energies
(and therefore excitation energies) comparable to high-level CC methods. The
performance of this procedure is examined with doubly excited states of H2,
Methylene, Formaldehyde, and Nitroxyl.

††preprint: AIP/123-QED

##  I  Introduction

Calculating excited states remains an important challenge in quantum chemistry
despite over a decade of strong theoretical development. The many reasons why
it is challenging to describe excited states are compounding. First, there is
the chemical nature of the problem, which includes the range of excitation
energies, the density of states, and the nature of the excited state, to name
a few factors.Vogler and Kunkely (2001); González, Escudero, and Serrano-
Andrés (2012) Then there is the task of determining an appropriate methodology
to accurately describe the state of interest. For higher-than-singly excited
states, reliable high-level theories are required to accurately capture these
states, which can be prohibitively expensive. After determining a desirable
methodology, one is subjected to the available eigensolver algorithms and
their corresponding operation count, memory requirements, and convergence
issues. Given these hurdles, it seems fitting that excited state calculations
can greatly benefit from incorporating modern technology and unconventional
approaches.

In chemistry, quantum computing has the potential to revolutionize the field
by enabling the simulation of molecular systems that are beyond the
capabilities of classical computers.Aspuru-Guzik  _et al._ (2005); Cao  _et
al._ (2019); McArdle  _et al._ (2020); Claudino (2022) The development of
quantum algorithms has been primarily and largely devoted to describing the
ground state. However, there are a handful of algorithms for excited states.
The quantum phase estimation (QPE) algorithmNielsen and Chuang (2010); Kitaev
(1995) is a powerful tool for calculating excited states, as the probability
of sampling a particular state is directly proportional to the square of the
overlap of the target state and a trial state. In earlier work, we
demonstrated the ease of use of the QPE algorithm in calculating a variety of
high-energy excited states using simple postulated initial states.Bauman  _et
al._ (2021) The variational quantum eigensolver, usually applied to ground-
state problems, can be extended to calculate excited states through the
multistate-contracted,Parrish  _et al._ (2019a, b) folded spectrum,McClean
_et al._ (2016); Tazi and Thom (2023) and Lagrangian-based approaches.Kuroiwa
and Nakagawa (2021); Greene-Diniz and Muñoz Ramo (2021); Selvarajan, Sajjan,
and Kais (2022) Other methods for excited states involve the imaginary-time
variational quantum simulator,McClean and Aspuru-Guzik (2015); McArdle  _et
al._ (2019); Motta  _et al._ (2020a) Peeters–Devreese–Soldatov energy
functional,Peeters and Devreese (1984); Soldatov (1995) quantum subspace
expansion,McClean  _et al._ (2017); Urbanek  _et al._ (2020) and the quantum
equation-of-motion approach,Ollitrault  _et al._ (2020) to name a few.
However, until quantum algorithms and hardware mature, the application of
these techniques is greatly limited. In order to perform meaningful quantum
calculations, methods for calculating excited states must be coupled with
formalisms to reduce the dimensionality of the problem (see discussion in Ref.
Kowalski and Bauman, 2020).

Mathematically rigorous formulations for reducing the dimensionality/cost of
quantum formulations are necessary for expanding the envelope of system sizes
tractable with current and near-term quantum simulators and hardware. There
have been a variety of approximations and techniques in recent years for
reducing the dimensionality and the complexity of quantum calculations.Motta
_et al._ (2020b); McArdle and Tew (2020); Takeshita  _et al._ (2020); Bylaska
_et al._ (2021); Liu, Li, and Yang (2022); Kowalski and Bauman (2022);
Callahan, Lange, and Berkelbach (2021); Fan  _et al._ (2023); Huang, Li, and
Evangelista (2023) One of the most promising formalisms is the downfolding
technique based on the double unitary coupled cluster (DUCC) ansatz,Bauman
_et al._ (2019); Metcalf  _et al._ (2020); Kowalski and Bauman (2020); Bauman,
Low, and Kowalski (2019); Chládek  _et al._ (2021); Bauman and Kowalski
(2022a, b) which constructs effective (or downfolded) Hamiltonians in a small-
dimensionality sub-space of the entire Hilbert space, which is commonly
defined as an active space. The resulting downfolded Hamiltonians integrate
out the external (out-of-active-space) Fermionic degrees of freedom from the
internal (in-the-active-space) parameters of the wave function, which can be
determined as components of the eigenvectors of the downfolded Hamiltonians in
the active space.

In earlier work, we introduced an extension of the DUCC approach for excited
states.Bauman, Low, and Kowalski (2019) The method combined excited-state
equation-of-motion CC (EOM-CC) theory operators with ground-state cluster
operators to construct a state-selective downfolded Hamiltonian for the
targeted excited state, which worked well for singly excited states. Since
this approach requires an underlying EOM-CC calculation, it is computationally
expensive to extend to describe higher-than-singly excited states and subject
to all the hurdles described earlier. So, it would seem beneficial to
circumvent any EOM-CC calculation and find a technique that utilizes the well-
defined ground-state downfolding procedure to construct excited-state
effective Hamiltonians.

One of the simplest yet attractive approaches to approximating excited states
has been to use self-consistent field (SCF) solutions.Gilbert, Besley, and
Gill (2008); Triguero, Pettersson, and Ågren (1998); Ehlert, Gühr, and
Saalfrank (2018); Evangelista, Shushkov, and Tully (2013); Derricotte and
Evangelista (2015); Barca, Gilbert, and Gill (2018) In this way, excitation
energies are given by the energy difference between two distinct SCF
solutions. This approach, often referred to as the $\Delta$-SCF approximation,
brings the advantage of orbital relaxation. One can also improve the electron-
electron correlation effects by carrying out many-body approaches on the
distinct SCF solutions to build on these solutions. This means robust high-
level ground-state methodologies can be applied to study excited states while
avoiding all of the stumbling blocks of conventional excited-state many-body
approaches.

The single-reference coupled cluster approachCoester (1958); Coester and
Kümmel (1960); Čížek (1966, 1969); Čížek and Paldus (1971); Paldus, Čížek, and
Shavitt (1972) is well known for accurately describing ground state properties
in molecular systems. However, the nature of the non-linear CC equations is
sometimes undervalued and underutilized. One of the appealing features of the
CC equations is the ability to converge to excited states with the same
symmetry as the reference function.Živković and Monkhorst (2008) Convergence
to a targeted solution is aided by starting with an SCF reference solution
corresponding to a targeted state. The difference between ground- and excited-
state solutions define the $\Delta$-CC approach, which has been strongly
demonstrated and investigated.Meissner, Balková, and Bartlett (1993);
Jankowski, Kowalski, and Jankowski (1994a, b, 1995); Kowalski and Jankowski
(1998); Jankowski and Kowalski (1999a, b, c); Piecuch, Kowalski, and
Leszczyński (2000); Podeszwa and Stolarczyk (2002); Podeszwa  _et al._ (2003);
Mayhall and Raghavachari (2010); Mewes  _et al._ (2014); Lee, Small, and Head-
Gordon (2019) It is also worth mentioning that the nature of the CC equations
to converge to excited states has been studied in the context of multi-
reference CC methods which has lead to the understanding of the ’intruder’
state problem.Paldus, Pylypow, and Jeziorski (1989)

Starting with an SCF solution for an excited state, one can utilize ground-
state downfolding techniques based on the DUCC formalism to construct a state-
specific effective Hamiltonian for the target excited state. In this work, I
investigate the utility of this procedure by applying it to the study of
doubly excited, for which conventional many-body methods struggle to describe.
Within the CC framework, one often turns the excited-state equation-of-motion
(EOM) extensions.Geertsen, Rittby, and Bartlett (1989); Comeau and Bartlett
(1993); Stanton and Bartlett (1993); Kállay and Surján (2000); Hirata,
Nooijen, and Bartlett (2000); Kowalski and Piecuch (2001a, b); Kucharski  _et
al._ (2001); Hirata (2004); Kállay and Gauss (2004) with higher-than-double
excitations to describe these states, such as the EOM approach with single,
double, and triple excitations (EOM-CCSDT)Kállay and Surján (2000); Hirata,
Nooijen, and Bartlett (2000); Kowalski and Piecuch (2001a, b); Kucharski  _et
al._ (2001) or up to quadruple excitations (EOM-CCSDTQ)Kállay and Surján
(2000); Hirata, Nooijen, and Bartlett (2000); Hirata (2004); Kállay and Gauss
(2004). The full treatment of triple excitations with the EOM-CCSDT and EOM-
CCSDT approaches have $\mathcal{N}^{8}$ and $\mathcal{N}^{10}$ computational
steps, respectfully, where $\mathcal{N}$ is a measure of the system size. In
contrast, the DUCC approaches in this paper have $\mathcal{N}^{6}$ steps when
the ground-state CCSD approach is used as a source of amplitudes while
avoiding complications associated with excited-state algorithms. The final
reduced dimensionality of each problem, expressed by the downfolded
Hamiltonian, represents system sizes amenable to current and near-term quantum
computing architecture.

##  II  Theory

###  II.1 Canonical CC Theory

Determining the excitation energy of an excited state using a non-Aufbau
reference determinant requires two separate calculations. The first uses a
standard Aufbau reference corresponding to the ground state, denoted
$\ket{\Phi_{0}}$. The second calculation uses the non-Aufbau reference
determinant that correlates with the $\mu$-th excited state, denoted
$\ket{\Phi_{\mu}}$. One can take the energy difference at this stage, which is
referred to as the $\Delta$-SCF approximation. These states’ descriptions are
then improved by post-SCF calculations using suitable methods that treat these
states separately. The exponential ansatz of coupled cluster theory is
particularly well-suited for such calculations.

The single-reference CC theory utilizes the exponential representation of the
wave function $\ket{\Psi_{\mu}}$

$\ket{\Psi_{\mu}}=e^{T_{\mu}}\ket{\Phi_{\mu}}\;,$ (1)

where $T_{\mu}$ is the cluster operator and $\ket{\Phi_{\mu}}$ is the single-
determinantal reference function corresponding to either the ground state
($\mu=0$) or an excited state ($\mu>0$). The cluster operator can be
represented through its many-body components $T_{\mu,n}$

$T_{\mu}=\sum_{n=1}^{M}T_{\mu,n}\;,$ (2)

where the individual component $T_{\mu,n}$ take the form

$T_{\mu,n}=\frac{1}{(n!)^{2}}\sum_{\begin{subarray}{c}i_{1},\ldots,i_{n}\\\
a_{1},\ldots,a_{n}\end{subarray}}t^{i_{1}\ldots i_{n}}_{\mu,a_{1}\ldots
a_{n}}E^{a_{1}\ldots a_{n}}_{i_{1}\ldots i_{n}}\;.$ (3)

In the expressions, indices $i_{1},i_{2},\ldots$ ($a_{1},a_{2},\ldots$) refer
to occupied (unoccupied) spin orbitals in the reference function
$\ket{\Phi_{\mu}}$. The excitation operators $E^{a_{1}\ldots
a_{n}}_{i_{1}\ldots i_{n}}$ are defined through strings of standard creation
($a_{p}^{\dagger}$) and annihilation ($a_{p}$) operators

$E^{a_{1}\ldots a_{n}}_{i_{1}\ldots i_{n}}=a_{a_{1}}^{\dagger}\ldots
a_{a_{n}}^{\dagger}a_{i_{n}}\ldots a_{i_{1}}\;,$ (4)

where creation and annihilation operators satisfy conventional anti-
commutation rules. When $M$ in the summation in Eq. 2 is equal to the number
of correlated electrons ($N_{e}$), then the corresponding CC formalism is
equivalent to the exact, full configuration interaction (FCI) solution, while
truncations ($M<N_{e}$) lead to the hierarchy of standard CC approximations
such as CC with singles and doubles (CCSD, $M=2$), CC with singles, doubles,
and triples (CCSDT, $M=3$), and so on.

The amplitudes $t^{i_{1}\ldots i_{n}}_{\mu,a_{1}\ldots a_{n}}$ in Eq. 3 are
determined by solving a coupled system of energy-independent non-linear
algebraic equations

$\matrixelement{\Phi^{a_{1}\ldots a_{n}}_{\mu,i_{1}\ldots
i_{n}}}{\overline{H}_{\mu}}{\Phi_{\mu}}\;,$ (5)

corresponding to the amplitudes that are being solved, where $n=1,\ldots,M$,

$\overline{H}_{\mu}=e^{-T_{\mu}}He^{T_{\mu}}=(He^{T_{\mu}})_{C}\;,$ (6)

is the similarity-transformed Hamiltonian, $C$ designates the connected part
of the operator expression, and $\ket{\Phi^{a_{1}\ldots
a_{n}}_{\mu,i_{1}\ldots i_{n}}}=a_{a_{1}}^{\dagger}\ldots
a_{a_{n}}^{\dagger}a_{i_{n}}\ldots a_{i_{1}}\ket{\Phi_{\mu}}$ are the
$n$-tuply excited determinants relative to the reference $\ket{\Phi_{\mu}}$.
Once the amplitudes are solved, the CC energy is determined using the equation

$E_{\mu}=\matrixelement{\Phi_{\mu}}{\overline{H}_{\mu}}{\Phi_{\mu}}\;.$ (7)

The excitation energy for the $\mu$-th excited state $\Delta E_{\mu}$ is the
difference between the excited state energy $E_{\mu}$ ($\mu>0$) and the ground
state energy $E_{0}$.

$\Delta E_{\mu}=E_{\mu}-E_{0}\;.$ (8)

This procedure is referred to as the $\Delta$CC method.

###  II.2 DUCC Approach

The DUCC formalism was developed for meaningful quantum chemistry calculations
on limited quantum computing resources. The method is based on constructing
effective (or downfolded) Hamiltonians in a small-dimensionality sub-space of
the entire Hilbert space. While the DUCC approach is based on coupled cluster
theory, one can not utilize the standard CC ansatz, given by Eq. 1, as it
leads to similarity-transformed Hamiltonians, Eq. 6, that are non-Hermitian.
Instead, an active space is used to introduce the DUCC ansatz that explicitly
decouples excitations describing correlation effects inside (internal) and
outside (external) of an active space, i.e.,

$\ket{\Psi_{\mu,{\rm DUCC}}}=e^{\sigma_{\mu,{\rm ext}}}e^{\sigma_{\mu,{\rm
int}}}\ket{\Phi_{\mu}}\;,$ (9)

where $\sigma_{\mu,{\rm int}}$ and $\sigma_{\mu,{\rm ext}}$ are anti-Hermitian
cluster operators

$\displaystyle\sigma_{\mu,{\rm int}}^{\dagger}$ $\displaystyle=$
$\displaystyle-\sigma_{\mu,{\rm int}}\;,$ (10) $\displaystyle\sigma_{\mu,{\rm
ext}}^{\dagger}$ $\displaystyle=$ $\displaystyle-\sigma_{\mu,{\rm ext}}\;.$
(11)

The algebraic form of the exact $\sigma_{\mu,{\rm int}}$ and $\sigma_{\mu,{\rm
ext}}$ operators can effectively be approximated using UCC formalism, i.e.,

$\displaystyle\sigma_{\mu,{\rm int}}$ $\displaystyle\simeq$ $\displaystyle
T_{\mu,{\rm int}}-T^{\dagger}_{\mu,{\rm int}}\;,$ (12)
$\displaystyle\sigma_{\mu,{\rm ext}}$ $\displaystyle\simeq$ $\displaystyle
T_{\mu,{\rm ext}}-T^{\dagger}_{\mu,{\rm ext}}\;.$ (13)

Once again, the $\mu$ notation is used because the DUCC approach can describe
excited states with a proper reference determinant choice despite starting as
a ground-state formalism.

The DUCC formalism is described in greater detail in Refs. Bauman  _et al._ ,
2019; Kowalski and Bauman, 2020; Bauman and Kowalski, 2022b, but the
foundation of the downfolding approach is that the energy $E_{\mu}$ can be
obtained by diagonalizing an effective Hamiltonian $\bar{H}_{\rm ext}^{\rm
eff(DUCC)}$ in the corresponding active space (defined by projection operator
$P+Q_{\rm int}$, where $P$ and $Q_{\rm int}$ are projection operators onto the
reference function and electron-promoted determinants in the active space,
respectively)

$\bar{H}_{\mu}^{\rm eff(DUCC)}e^{\sigma_{\mu,{\rm
int}}}\ket{\Phi_{\mu}}=E_{\mu}e^{\sigma_{\mu,{\rm int}}}\ket{\Phi_{\mu}}\;,$
(14)

where

$\bar{H}_{\mu}^{\rm eff(DUCC)}=(P+Q_{\mu,{\rm int}})\bar{H}_{\mu,{\rm
ext}}^{\rm DUCC}(P+Q_{\mu,{\rm int}})$ (15)

and

$\bar{H}_{\mu,{\rm ext}}^{\rm DUCC}=e^{-\sigma_{\mu,{\rm
ext}}}He^{\sigma_{\mu,{\rm ext}}}\;.$ (16)

In practice, one constructs approximate many-body forms of $\bar{H}_{\mu,{\rm
ext}}^{\rm eff(DUCC)}$.

Three considerations go into the approximation of $\bar{H}_{\mu,{\rm
ext}}^{\rm eff(DUCC)}$: (1) the rank of many-body effects included in
$\bar{H}_{\rm ext}^{\rm eff(DUCC)}$, (2) the approximate representation of
$\sigma_{\mu,{\rm ext}}$ ($T_{\mu,{\rm ext}}$), and (3) the length of the
commutator expansion for $e^{-\sigma_{\rm ext}}He^{\sigma_{\rm ext}}$. In this
paper, I limit the effective Hamiltonian to one- and two-body elements. I also
employ traditional CC theory to source the approximate $T_{\rm ext}$ operators
that parameterize $\sigma_{\rm ext}$ ($\sigma_{\mu,{\rm ext}}\simeq
T_{\mu,{\rm ext}}-T^{\dagger}_{\mu,{\rm ext}}$). Finally, in a previous study,
we showed that approximating the expansion of $e^{-\sigma_{\rm
ext}}He^{\sigma_{\rm ext}}$ so as to be consistent through second-order
perturbation theory labeled DUCC(2), given by

$\bar{H}_{\mu,{\rm ext}}^{\rm DUCC(2)}\simeq H+[H_{N},\sigma_{\mu,{\rm
ext}}]+\frac{1}{2!}[[F_{N},\sigma_{\mu,{\rm ext}}],\sigma_{\mu,{\rm ext}}]\;,$
(17)

performs well in describing the system with the third-order consistent
expression labeled DUCC(3) and given by

$\displaystyle\bar{H}_{\mu,{\rm ext}}^{\rm DUCC(3)}\simeq$ $\displaystyle
H+[H_{N},\sigma_{\mu,{\rm ext}}]+\frac{1}{2!}[[H_{N},\sigma_{\mu,{\rm
ext}}],\sigma_{\mu,{\rm ext}}]+$ (18)
$\displaystyle\frac{1}{3!}[[[F_{N},\sigma_{\mu,{\rm ext}}],\sigma_{\mu,{\rm
ext}}],\sigma_{\mu,{\rm ext}}]\;,$

further improving upon the results. In Eqs. 17 and 18, $F_{N}$ and $H_{N}$ are
the normal-product Fock and Hamiltonian operators.

##  III  Results and Discussion

I investigated the doubly excited states of four molecules: H2, Methylene,
Formaldehyde, and Nitroxyl. For H2, the cc-pVTZ basis set was employed, while
aug-cc-pVDZ was used for Methylene, Formaldehyde, and Nitroxyl. The
$T_{\mu,{\rm ext}}$ amplitudes defining $\sigma_{\mu,{\rm ext}}$ are sourced
from traditional CCSD ($\sigma_{\mu,{\rm ext}}\simeq T^{\rm(CCSD)}_{\mu,{\rm
ext}}-T^{\rm(CCSD)\dagger}_{\mu,{\rm ext}}$). Any method that utilizes a non-
Aufbau reference determinant to describe an excited state is given the prefix
"ES-". Lastly, excitation energies computed as the difference between "ES-"
excited state energies and the canonical ground state energies using their
corresponding reference determinants are labeled with the prefix "$\Delta$-".
Core orbitals were frozen in all post-SCF calculations of Formaldehyde and
Nitroxyl. In the case of the DUCC methods, core orbitals were frozen after
forming the effective Hamiltonian from all-electron calculations.

###  III.1 H2

We start our discussion by investigating the
(1$\sigma_{g}$)${}^{2}\rightarrow$ (1$\sigma_{u}$)2 excitation along the
$H$–$H$ bond-breaking coordinate. In this case, CCSD is equivalent to FCI, so
using CCSD as a source of amplitudes to define $\sigma_{\mu,{\rm ext}}$
removes any concern about the role of higher-order excitations. An active
space constructed from the four lowest-energy orbitals was used for the
downfolding procedure, where the full space consists of 28 orbitals. The
results are shown in Fig. 1.

The $\Delta$-SCF is praised for providing excitation energies similar to FCI
near equilibrium, but that is because of fortuitous balanced errors in the
total energies for the ground and excited state, which is not the case for
other geometries along the potential energy surface. Diagonalizing the bare
Hamiltonian in the active space provides a much-needed improvement, especially
at stretched bond lengths, but it is not balanced for both states. When
correlation is downfolded with the DUCC(2) approach, total energies for the
ground state agree with FCI to within 0.16 eV along the whole potential energy
surface and improve upon the bare Hamiltonian for the excited state with the
exception of significantly compressed bond lengths. When the DUCC(3) method is
utilized, all energies are improved compared to the DUCC(2) approach, and one
gets both total and excitation energies that agree excellently with FCI
results by diagonalizing the effective Hamiltonian in an active space of only
four orbitals.

Figure 1:  Errors relative to FCI for the ground state (Top Panel),
[(1$\sigma_{g}$)${}^{2}\rightarrow$ (1$\sigma_{u}$)2] excited state (Middle
Panel), and the corresponding excitation energy (Bottom Panel) along the
$H$–$H$ bond-breaking coordinate. The equilibrium bond length is taken to be
0.7414 Å.

###  III.2 Methylene

The first excited state (1 ${}^{1}A_{1}$) and third excited state (2
${}^{1}A_{1}$) of Methylene have considerable mixing of configurations
$(1a_{1})^{2}(2a_{1})^{2}(1b_{1})^{2}(3a_{1})^{2}$ and
$(1a_{1})^{2}(2a_{1})^{2}(1b_{1})^{2}(1b_{2})^{2}$. The (1 ${}^{1}A_{1}$)
state can be computed using ground state methods as it is the lowest-energy
singlet state. The (2 ${}^{1}A_{1}$) state can then be calculated as an
excited state formed from a double excitation out of the (1 ${}^{1}A_{1}$)
state. The multiconfigurational character of the (1 ${}^{1}A_{1}$), while non-
negligible, is minor enough that ground-state methods such as CCSD can provide
an accurate description of the state. However, the mixing of configurations is
much more prominent in the (2 ${}^{1}A_{1}$). When calculated as a doubly
excited state from (1 ${}^{1}A_{1}$), one needs to include higher-than-double
excitations to capture this state accurately.

As shown in Fig. 2, while CCSD and CCSDT provide accurate results of the (1
${}^{1}A_{1}$) state, EOM-CCSD struggles to describe the (2 ${}^{1}A_{1}$)
state, with an error of 1.67 eV. This error vanishes with EOM-CCSDT, which is
nearly exact for this system. For the downfolding procedure, several active
spaces were explored consisting of 5-20 of the lowest-energy molecular orbital
of the Aufbau and non-Aufbau reference determinants (denoted as $n$), whereas
the full orbital space consists of 41 orbitals. The DUCC(2) approximation
substantially improves the total energies over the bare Hamiltonian in the
same size active space. Both states’ DUCC(2) energies are well-balanced,
leading to excitation energies that agree well with the benchmark EOM-CCSDTQ
result. However, there is still room for improvement in the total energies
with the DUCC(2) approach. Higher-order terms in the commutator expansion play
a crucial role in providing accurate total energies for the two states, as
shown with the DUCC(3) approach. For the (1 ${}^{1}A_{1}$) state, the DUCC(3)
approach provides total energies between CCSD and CCSDT, while for the (2
${}^{1}A_{1}$) state it reduces the errors to less than 0.44 eV for any active
space. The state-specific nature of the downfolding procedure means that the
errors are not always balanced. between given states, which is why the DUCC(3)
approach has larger errors in the excitation energy than the DUCC(2)
approximation. Still, there is a substantial improvement with the DUCC(3)
procedures over EOM-CCSD in describing this excitation with a steady
improvement to the EOM-CCSDTQ benchmark as the active-space size increases.

Figure 2:  Errors relative to CCSDTQ and EOMCCSDTQ for the (1 ${}^{1}A_{1}$)
state (Top Panel), (2 ${}^{1}A_{1}$) excited state (Middle Panel), and the
corresponding excitation energy (Bottom Panel) of the methylene molecule. The
$C-H$ bond length was 1.107Å, and the $H-C-H$ angle was 102.4 degrees.

###  III.3 Formaldehyde

Formaldehyde has a doubly excited state [(2${}^{1}B_{1}$)2 $\rightarrow$
(2${}^{1}B_{2}$)2] that has been troublesome for many low-level many-body
methods. The EOM-CCSD method overestimates the excitation energy by 4 eV. The
EOM-CCSDT method greatly improves upon this excitation energy, but one needs
to include quadruple excitations through EOM-CCSDTQ to get nearly exact
results. The ground state is well-described in the case of CCSD and CCSDT. It
is the excited state description that leads to errors in excitation energy.
When the non-Aufbau reference determinant is used, the ES-CCSD result provides
excited-state total energy comparable to canonical EOM-CCSDT, while the ES-
CCSDT method provides canonical EOM-CCSDTQ quality results. The agreement of
the ES-CCSD and ES-CCSDT methods reinforces the notion of using the
[(2${}^{1}B_{1}$)2 $\rightarrow$ (2${}^{1}B_{2}$)2] determinant as the
reference for the downfolding procedure.

For the downfolding procedure, an active space consisting of the 15 (13 after
freezing the core orbitals) lowest-energy molecular orbitals for both the
Aufbau and non-Aufbau reference determinants was utilized. The full
calculation involves 64 orbitals. Through the maximum overlap method, I
confirmed that all of the orbitals in the Aufbau determinant have
corresponding analogous orbitals in the non-Aufbau reference determinant and
that different orbitals are not introduced into the active space during the
SCF procedure. Illustrated in Fig. 3, the bare Hamiltonian provides excellent
excitation energy but at the cost of over 9 eV errors in both the ground and
excited state. With only 13 orbitals, the DUCC(2) approach greatly improved
both total energies and provided EOM-CCSDT quality excitation energy. The
DUCC(3) approach further improves the total energies, while a slight imbalance
in errors leads to a small increase in the error of the excitation energy.
Both the DUCC(2) and DUCC(3) approaches provide an impressive improvement
compared to EOM-CCSD in describing the excited state, with the DUCC(3) method
providing total energies for both states comparable to high-level CC
approaches.

Figure 3:  Errors relative to CCSDTQ and EOMCCSDTQ for the ground state (Top
Panel), [(2${}^{1}B_{1}$)2 $\rightarrow$ (2${}^{1}B_{2}$)2] excited state
(Middle Panel), and the corresponding excitation energy (Bottom Panel) of the
formaldehyde molecule. The equilibrium geometry was taken from Ref. Loos  _et
al._ , 2019.

###  III.4 Nitroxyl

Nitroxyl is a weak acid that has a low-lying [(7${}^{1}a^{\prime}$)2
$\rightarrow$ (2${}^{1}a^{\prime\prime}$)2] excited state. In a similar
fashion to Formaldehyde, the EOM-CCSD method overestimates the excitation
energy by 4 eV. The EOM-CCSDT method reduces the excitation energy error to
0.3 eV, and once again, quadruple excitations through EOM-CCSDTQ are needed to
get nearly exact results. For the EOM-CCSD and EOM-CCSDT approaches, the
errors in the excited state are approximately 10 times greater than the ground
state, which contributes to the excitation energy errors. However, when the
non-Aufbau reference determinant is used, once again, the ES-CCSD result is
comparable to canonical EOM-CCSDT, and the ES-CCSDT method provides canonical
EOM-CCSDTQ-level results. So the [(7${}^{1}a^{\prime}$)2 $\rightarrow$
(2${}^{1}a^{\prime\prime}$)2] reference determinant was used in the ground-
state DUCC pipeline.

An active space consisting of the 17 (15 after freezing the core orbitals)
lowest-energy molecular orbitals for both the Aufbau and non-Aufbau reference
determinants was used, whereas the full calculation contains 55 molecular
orbitals. Through the maximum overlap method, I confirmed that all of the
orbitals in the Aufbau determinant have corresponding analogous orbitals in
the non-Aufbau reference determinant. As shown in Fig. 4, diagonalizing the
truncated bare Hamiltonian gives nearly exact excitation energy, but ground
and excited states have total energy errors over 9 eV. Both the DUCC(2) and
DUCC(3) approaches greatly improve the total energies in the active space
compared to the bare Hamiltonian. However, his improvement is not always
balanced and can lead to excitation energies with undesired errors. Since
either approach significantly improves the EOM-CCSD total energy for the
excited state, the DUCC(2) and DUCC(3) approximations still provide better
excitation energy than EOM-CCSD despite the imbalance total energy errors.

Figure 4:  Errors relative to CCSDTQ and EOMCCSDTQ for the ground state (Top
Panel), [(7${}^{1}a^{\prime}$)2 $\rightarrow$ (2${}^{1}a^{\prime\prime}$)2]
excited state (Middle Panel), and the corresponding excitation energy (Bottom
Panel) of the nitroxyl molecule. The equilibrium geometry was taken from Ref.
Loos  _et al._ , 2019.

Compared to Formaldehyde, Nitroxyl exhibits a greater difference between CCSD
and CCSDT errors for the ground state and ES-CCSD and ES-CCSDT errors for the
excited state. Between ES-CCSD and ES-CCSDT, the energy difference is 0.66 eV.
Upon investigation, there is a $T_{2}$ amplitude in the ES-CCSD calculation
with a value of 0.61, which is less than 0.1 in the ES-CCSDT calculation.
Given the largest cluster amplitude fluctuation, we performed the downfolding
procedure with $T_{1}$ and $T_{2}$ amplitudes from the corresponding CCSDT and
ES-CCSDT calculations as terms with explicit $T_{3}$ contributions have not
been coded yet. Results in Fig. 5 show that higher-order excitations play a
non-negligible role in defining the effective Hamiltonian for Nitroxyl,
changing total energies by up to 2 eV. In the case of the excited state, we
notice a positive improvement in total energies, but the same cannot be said
for the ground state. However, for the time being, this story is left
incomplete because if triple excitations play such a significant role, it
stands to reason those explicit $T_{3}$ (and higher-order) contributions,
longer commutator expansions, and higher-order many-body elements of the
Hamiltonian can all endow further changes to the description of the ground and
excited states.

Figure 5:  Comparing the source of amplitudes defining the approximate
$\sigma_{\mu,{\rm ext}}$ operator. When CCSDT is used as a source of
amplitudes, the method is denoted with an asterisk *. Errors are relative to
CCSDTQ and EOMCCSDTQ for the ground state (Top Panel), [(7${}^{1}a^{\prime}$)2
$\rightarrow$ (2${}^{1}a^{\prime\prime}$)2] excited state (Middle Panel), and
the corresponding excitation energy (Bottom Panel) of the nitroxyl molecule.
The equilibrium geometry was taken from Ref. Loos  _et al._ , 2019.

###  III.5 Overall

Through the four examples, several trends can be observed. First, downfolding
procedures with non-Aufbau references can accurately describe excited states
in compact active spaces with substantial improvements over the active-space
bare Hamiltonian and even EOM-CCSD for the representative difficult cases
studied. Second is that higher-order terms in the commutator expansion play a
significant role in improving accuracies of total energies, as seen with the
difference between the DUCC(2) and DUCC(3) approaches. Third, since the
downfolding procedure is state state-specific, the description between two
states is not always balanced. It is important to reiterate that accurate
excitation energies in and method may be fortuitous at the expense of total
energies. The downfolding procedures in this paper aim to provide accurate
total energies and consequently excitation energies. To further improve upon
the total energies and provide a balanced description, the role of longer
commutator expansions, higher-order excitations, and higher body terms in the
downfolded Hamiltonian needs to be investigated.

As seen with methylene, the description of the two states systematically
improves as the active space size increases. The active spaces of the
remaining molecules represent system sizes well-suited for current and near-
term quantum computing architectures. Larger active spaces can be explored as
quantum computers grow in their quantum capacity. This means that the
approximations in the downfolding procedure are more befitting, and the
description of states becomes more accurate and balanced. One can imagine
reliably downfolding 100s or 1000s of orbitals to active spaces that are
magnitudes of order smaller, greatly expanding the envelope of problems one
can tackle with quantum computers.

##  IV Conclusion

In this paper, I explored how ground-state downfolding coupled cluster
techniques are efficient tools that can target excited states using non-Aufbau
reference determinants. The double unitary coupled cluster ansatz employed
produces state-specific hermitian effective Hamiltonians of reduced
dimensionality corresponding to an active space used to partition the
excitations in the ansatz. The final reduced dimensionality of each problem,
expressed by the downfolded Hamiltonian, represents system sizes amenable to
current and near-term quantum computing architecture.

We focused on doubly excited states of H2, Methylene, Formaldehyde, and
Nitroxyl, and demonstrated that downfolding techniques result in state-
specific effective Hamiltonians that, when diagonalized in their respective
active spaces, provide ground- and excited-state total energies (and therefore
excitation energies) comparable to high-level CC methods. The downfolding
procedures consistently improve the accuracy of total energies compared to the
bare Hamiltonian within the same active space size. The downfolding procedures
also provide excited state energies that outperform EOM-CCSD, leading to
better total and excitation energies. This technique of combining non-Aufbau
reference determinants with downfolding procedures allows for accurate and
reliable investigation of excited states with a significant reduction in
dimensionality. Together with excited-state quantum computing techniques and
algorithms, this technique greatly broadens the envelope of excited-state
quantum chemistry problems for quantum computers.

##  V Acknowledgement

This work was supported the Quantum Science Center (QSC), a National Quantum
Information Science Research Center of the U.S. Department of Energy (under
FWP 76213) and the Laboratory Directed Research and Development (LDRD) Program
at PNNL. This work used resources from the Pacific Northwest National
Laboratory (PNNL). PNNL is operated by Battelle for the U.S. Department of
Energy under Contract DE-AC05-76RL01830.

## References

  * Vogler and Kunkely (2001) A. Vogler and H. Kunkely, “Luminescent metal complexes: Diversity of excited states,” in  _Transition Metal and Rare Earth Compounds: Excited States, Transitions, Interactions I_, edited by H. Yersin (Springer Berlin Heidelberg, Berlin, Heidelberg, 2001) pp. 143–182.
  * González, Escudero, and Serrano-Andrés (2012) L. González, D. Escudero, and L. Serrano-Andrés, “Progress and challenges in the calculation of electronic excited states,” ChemPhysChem 13, 28–51 (2012).
  * Aspuru-Guzik  _et al._ (2005) A. Aspuru-Guzik, A. D. Dutoi, P. J. Love, and M. Head-Gordon, “Simulated quantum computation of molecular energies,” Science 309, 1704–1707 (2005).
  * Cao  _et al._ (2019) Y. Cao, J. Romero, J. P. Olson, M. Degroote, P. D. Johnson, M. Kieferová, I. D. Kivlichan, T. Menke, B. Peropadre, N. P. D. Sawaya, S. Sim, L. Veis, and A. Aspuru-Guzik, “Quantum chemistry in the age of quantum computing,” Chem. Rev. 119, 10856–10915 (2019).
  * McArdle  _et al._ (2020) S. McArdle, S. Endo, A. Aspuru-Guzik, S. C. Benjamin, and X. Yuan, “Quantum computational chemistry,” Rev. Mod. Phys. 92, 015003 (2020).
  * Claudino (2022) D. Claudino, “The basics of quantum computing for chemists,” Int. J. Quantum Chem. 122, e26990 (2022).
  * Nielsen and Chuang (2010) M. A. Nielsen and I. L. Chuang,  _Quantum computation and quantum information_ (Cambridge university press, 2010).
  * Kitaev (1995) A. Y. Kitaev, “Quantum measurements and the abelian stabilizer problem,” (1995), arXiv:quant-ph/9511026 [quant-ph] .
  * Bauman  _et al._ (2021) N. P. Bauman, H. Liu, E. J. Bylaska, S. Krishnamoorthy, G. H. Low, C. E. Granade, N. Wiebe, N. A. Baker, B. Peng, M. Roetteler, M. Troyer, and K. Kowalski, “Toward quantum computing for high-energy excited states in molecular systems: Quantum phase estimations of core-level states,” J. Chem. Theory Comput. 17, 201–210 (2021).
  * Parrish  _et al._ (2019a) R. M. Parrish, E. G. Hohenstein, P. L. McMahon, and T. J. Martínez, “Quantum computation of electronic transitions using a variational quantum eigensolver,” Phys. Rev. Lett. 122, 230401 (2019a).
  * Parrish  _et al._ (2019b) R. M. Parrish, E. G. Hohenstein, P. L. McMahon, and T. J. Martinez, “Hybrid quantum/classical derivative theory: Analytical gradients and excited-state dynamics for the multistate contracted variational quantum eigensolver,” (2019b), arXiv:1906.08728 [quant-ph] .
  * McClean  _et al._ (2016) J. R. McClean, J. Romero, R. Babbush, and A. Aspuru-Guzik, “The theory of variational hybrid quantum-classical algorithms,” New J. Phys. 18, 023023 (2016).
  * Tazi and Thom (2023) L. C. Tazi and A. J. W. Thom, “Folded spectrum vqe : A quantum computing method for the calculation of molecular excited states,” (2023), arXiv:2305.04783 [quant-ph] .
  * Kuroiwa and Nakagawa (2021) K. Kuroiwa and Y. O. Nakagawa, “Penalty methods for a variational quantum eigensolver,” Phys. Rev. Res. 3, 013197 (2021).
  * Greene-Diniz and Muñoz Ramo (2021) G. Greene-Diniz and D. Muñoz Ramo, “Generalized unitary coupled cluster excitations for multireference molecular states optimized by the variational quantum eigensolver,” Int. J. Quantum Chem. 121, e26352 (2021).
  * Selvarajan, Sajjan, and Kais (2022) R. Selvarajan, M. Sajjan, and S. Kais, “Variational quantum circuits to prepare low energy symmetry states,” Symmetry 14, 457 (2022).
  * McClean and Aspuru-Guzik (2015) J. R. McClean and A. Aspuru-Guzik, “Compact wavefunctions from compressed imaginary time evolution,” RSC Adv. 5, 102277–102283 (2015).
  * McArdle  _et al._ (2019) S. McArdle, T. Jones, S. Endo, Y. Li, S. C. Benjamin, and X. Yuan, “Variational ansatz-based quantum simulation of imaginary time evolution,” npj Quantum Inf. 5, 75 (2019).
  * Motta  _et al._ (2020a) M. Motta, C. Sun, A. T. Tan, M. J. O’Rourke, E. Ye, A. J. Minnich, F. G. Brandao, and G. K.-L. Chan, “Determining eigenstates and thermal states on a quantum computer using quantum imaginary time evolution,” Nat. Phys. 16, 205–210 (2020a).
  * Peeters and Devreese (1984) F. M. Peeters and J. T. Devreese, “Upper bounds for the free energy. a generalisation of the bogolubov inequality and the feynman inequality,” J. Phys. A 17, 625 (1984).
  * Soldatov (1995) A. V. Soldatov, “Generalized variational principle in quantum mechanics,” IInt. J. Mod. Phys. B 9, 2899–2936 (1995).
  * McClean  _et al._ (2017) J. R. McClean, M. E. Kimchi-Schwartz, J. Carter, and W. A. De Jong, “Hybrid quantum-classical hierarchy for mitigation of decoherence and determination of excited states,” Phys. Rev. A 95, 042308 (2017).
  * Urbanek  _et al._ (2020) M. Urbanek, D. Camps, R. Van Beeumen, and W. A. de Jong, “Chemistry on quantum computers with virtual quantum subspace expansion,” J. Chem. Theory Comput. 16, 5425–5431 (2020).
  * Ollitrault  _et al._ (2020) P. J. Ollitrault, A. Kandala, C.-F. Chen, P. K. Barkoutsos, A. Mezzacapo, M. Pistoia, S. Sheldon, S. Woerner, J. M. Gambetta, and I. Tavernelli, “Quantum equation of motion for computing molecular excitation energies on a noisy quantum processor,” Phys. Rev. Res. 2, 043140 (2020).
  * Kowalski and Bauman (2020) K. Kowalski and N. P. Bauman, “Sub-system quantum dynamics using coupled cluster downfolding techniques,” J. Chem. Phys 152, 244127 (2020).
  * Motta  _et al._ (2020b) M. Motta, T. P. Gujarati, J. E. Rice, A. Kumar, C. Masteran, J. A. Latone, E. Lee, E. F. Valeev, and T. Y. Takeshita, “Quantum simulation of electronic structure with a transcorrelated hamiltonian: improved accuracy with a smaller footprint on the quantum computer,” Phys. Chem. Chem. Phys. 22, 24270–24281 (2020b).
  * McArdle and Tew (2020) S. McArdle and D. P. Tew, “Improving the accuracy of quantum computational chemistry using the transcorrelated method,” arXiv preprint arXiv:2006.11181 (2020).
  * Takeshita  _et al._ (2020) T. Takeshita, N. C. Rubin, Z. Jiang, E. Lee, R. Babbush, and J. R. McClean, “Increasing the representation accuracy of quantum simulations of chemistry without extra quantum resources,” Physical Review X 10, 011004 (2020).
  * Bylaska  _et al._ (2021) E. J. Bylaska, D. Song, N. P. Bauman, K. Kowalski, D. Claudino, and T. S. Humble, “Quantum solvers for plane-wave hamiltonians: Abridging virtual spaces through the optimization of pairwise correlations,” Front. Chem. 9, 603019 (2021).
  * Liu, Li, and Yang (2022) J. Liu, Z. Li, and J. Yang, “Reducing circuit depth in adaptive variational quantum algorithms via effective hamiltonian theories,” J. Chem. Theory Comput. 18, 4795–4805 (2022).
  * Kowalski and Bauman (2022) K. Kowalski and N. P. Bauman, “Fock-space schrieffer–wolff transformation: Classically-assisted rank-reduced quantum phase estimation algorithm,” Appl. Sci. 13, 539 (2022).
  * Callahan, Lange, and Berkelbach (2021) J. M. Callahan, M. F. Lange, and T. C. Berkelbach, “Dynamical correlation energy of metals in large basis sets from downfolding and composite approaches,” J. Chem. Phys 154 (2021).
  * Fan  _et al._ (2023) Y. Fan, J. Liu, Z. Li, and J. Yang, “Quantum circuit matrix product state ansatz for large-scale simulations of molecules,” arXiv preprint arXiv:2301.06376 (2023).
  * Huang, Li, and Evangelista (2023) R. Huang, C. Li, and F. A. Evangelista, “Leveraging small-scale quantum computers with unitarily downfolded hamiltonians,” PRX Quantum 4, 020313 (2023).
  * Bauman  _et al._ (2019) N. P. Bauman, E. J. Bylaska, S. Krishnamoorthy, G. H. Low, N. Wiebe, C. E. Granade, M. Roetteler, M. Troyer, and K. Kowalski, “Downfolding of many-body Hamiltonians using active-space models: Extension of the sub-system embedding sub-algebras approach to unitary coupled cluster formalisms,” J. Chem. Phys 151, 014107 (2019).
  * Metcalf  _et al._ (2020) M. Metcalf, N. P. Bauman, K. Kowalski, and W. A. de Jong, “Resource-efficient chemistry on quantum computers with the variational quantum eigensolver and the double unitary coupled-cluster approach,” J. Chem. Theory Comput. 16, 6165–6175 (2020).
  * Bauman, Low, and Kowalski (2019) N. P. Bauman, G. H. Low, and K. Kowalski, “Quantum simulations of excited states with active-space downfolded Hamiltonians,” J. Chem. Phys 151, 234114 (2019).
  * Chládek  _et al._ (2021) J. Chládek, L. Veis, J. Pittner, K. Karol, _et al._ , “Variational quantum eigensolver for approximate diagonalization of downfolded hamiltonians using generalized unitary coupled cluster ansatz,” Quantum Sci. Technol. 6, 034008 (2021).
  * Bauman and Kowalski (2022a) N. P. Bauman and K. Kowalski, “Coupled cluster downfolding methods: The effect of double commutator terms on the accuracy of ground-state energies,” J. Chem. Phys 156, 094106 (2022a).
  * Bauman and Kowalski (2022b) N. P. Bauman and K. Kowalski, “Coupled cluster downfolding theory: Towards universal many-body algorithms for dimensionality reduction of composite quantum systems in chemistry and materials science,” Mater. Theory 6, 1–19 (2022b).
  * Gilbert, Besley, and Gill (2008) A. T. Gilbert, N. A. Besley, and P. M. Gill, “Self-consistent field calculations of excited states using the maximum overlap method (mom),” J. Phys. Chem. A 112, 13164–13171 (2008).
  * Triguero, Pettersson, and Ågren (1998) L. Triguero, L. Pettersson, and H. Ågren, “Calculations of near-edge x-ray-absorption spectra of gas-phase and chemisorbed molecules by means of density-functional and transition-potential theory,” Phys. Rev. B 58, 8097 (1998).
  * Ehlert, Gühr, and Saalfrank (2018) C. Ehlert, M. Gühr, and P. Saalfrank, “An efficient first principles method for molecular pump-probe nexafs spectra: Application to thymine and azobenzene,” J. Chem. Phys 149 (2018).
  * Evangelista, Shushkov, and Tully (2013) F. A. Evangelista, P. Shushkov, and J. C. Tully, “Orthogonality constrained density functional theory for electronic excited states,” J. Phys. Chem. A 117, 7378–7392 (2013).
  * Derricotte and Evangelista (2015) W. D. Derricotte and F. A. Evangelista, “Simulation of x-ray absorption spectra with orthogonality constrained density functional theory,” Phys. Chem. Chem. Phys. 17, 14360–14374 (2015).
  * Barca, Gilbert, and Gill (2018) G. M. Barca, A. T. Gilbert, and P. M. Gill, “Simple models for difficult electronic excitations,” J. Chem. Theory Comput. 14, 1501–1509 (2018).
  * Coester (1958) F. Coester, “Bound states of a many-particle system,” Nucl. Phys. 7, 421–424 (1958).
  * Coester and Kümmel (1960) F. Coester and H. Kümmel, “Short-range correlations in nuclear wave functions,” Nucl. Phys. 17, 477–485 (1960).
  * Čížek (1966) J. Čížek, “On the correlation problem in atomic and molecular systems. calculation of wavefunction components in ursell-type expansion using quantum-field theoretical methods,” J. Chem. Phys 45, 4256–4266 (1966).
  * Čížek (1969) J. Čížek, “On the use of the cluster expansion and the technique of diagrams in calculations of correlation effects in atoms and molecules,” Adv. Chem. Phys. , 35–89 (1969).
  * Čížek and Paldus (1971) J. Čížek and J. Paldus, “Correlation problems in atomic and molecular systems iii. rederivation of the coupled-pair many-electron theory using the traditional quantum chemical methodst,” Int. J. Quantum Chem. 5, 359–379 (1971).
  * Paldus, Čížek, and Shavitt (1972) J. Paldus, J. Čížek, and I. Shavitt, “Correlation problems in atomic and molecular systems. iv. extended coupled-pair many-electron theory and its application to the b h 3 molecule,” Phys. Rev. A 5, 50 (1972).
  * Živković and Monkhorst (2008) T. P. Živković and H. J. Monkhorst, “Analytic connection between configuration–interaction and coupled-cluster solutions,” J. Math. Phys. 19, 1007–1022 (2008).
  * Meissner, Balková, and Bartlett (1993) L. Meissner, A. Balková, and R. J. Bartlett, “Multiple solutions of the single-reference coupled-cluster method,” Chem. Phys. Lett. 212, 177–184 (1993).
  * Jankowski, Kowalski, and Jankowski (1994a) J. Jankowski, K. Kowalski, and P. Jankowski, “Applicability of single-reference coupled-cluster methods to excited states. a model study,” Chem. Phys. Lett. 222, 608–614 (1994a).
  * Jankowski, Kowalski, and Jankowski (1994b) K. Jankowski, K. Kowalski, and P. Jankowski, “Multiple solutions of the single-reference coupled-cluster equations. i. h4 model revisited,” Int. J. Quantum Chem. 50, 353–367 (1994b).
  * Jankowski, Kowalski, and Jankowski (1995) K. Jankowski, K. Kowalski, and P. Jankowski, “Multiple solutions of the single-reference coupled-cluster equations. ii. alternative reference states,” Int. J. Quantum Chem. 53, 501–514 (1995).
  * Kowalski and Jankowski (1998) K. Kowalski and K. Jankowski, “Towards complete solutions to systems of nonlinear equations of many-electron theories,” Phys. Rev. Lett. 81, 1195 (1998).
  * Jankowski and Kowalski (1999a) K. Jankowski and K. Kowalski, “Physical and mathematical content of coupled–cluster equations: Correspondence between coupled–cluster and configuration–interaction solutions,” J. Chem. Phys 110, 3714–3729 (1999a).
  * Jankowski and Kowalski (1999b) K. Jankowski and K. Kowalski, “Physical and mathematical content of coupled-cluster equations. ii. on the origin of irregular solutions and their elimination via symmetry adaptation,” J. Chem. Phys 110, 9345–9352 (1999b).
  * Jankowski and Kowalski (1999c) K. Jankowski and K. Kowalski, “Physical and mathematical content of coupled-cluster equations. iii. model studies of dissociation processes for various reference states,” J. Chem. Phys 111, 2940–2951 (1999c).
  * Piecuch, Kowalski, and Leszczyński (2000) P. Piecuch, K. Kowalski, and J. Leszczyński, “Computational chemistry: Reviews of current trends,” (2000).
  * Podeszwa and Stolarczyk (2002) R. Podeszwa and L. Z. Stolarczyk, “Multiple solutions of coupled-cluster equations for ppp model of [10] annulene,” Chem. Phys. Lett. 366, 426–432 (2002).
  * Podeszwa  _et al._ (2003) R. Podeszwa, L. Z. Stolarczyk, K. Jankowski, and K. Rubiniec, “Multiple solutions of coupled-cluster doubles equations for the pariser–parr–pople model of benzene,” Theor. Chem. Acc. 109, 309–315 (2003).
  * Mayhall and Raghavachari (2010) N. J. Mayhall and K. Raghavachari, “Multiple solutions to the single-reference ccsd equations for nih,” J. Chem. Theory Comput. 6, 2714–2720 (2010).
  * Mewes  _et al._ (2014) J.-M. Mewes, V. Jovanović, C. M. Marian, and A. Dreuw, “On the molecular mechanism of non-radiative decay of nitrobenzene and the unforeseen challenges this simple molecule holds for electronic structure theory,” Phys. Chem. Chem. Phys. 16, 12393–12406 (2014).
  * Lee, Small, and Head-Gordon (2019) J. Lee, D. W. Small, and M. Head-Gordon, “Excited states via coupled cluster theory without equation-of-motion methods: Seeking higher roots with application to doubly excited states and double core hole states,” J. Chem. Phys 151 (2019).
  * Paldus, Pylypow, and Jeziorski (1989) J. Paldus, L. Pylypow, and B. Jeziorski, “Spin-adapted multi-reference coupled cluster formalism including non-linear terms and its application to the h 4 model system,” in  _Many-Body Methods in Quantum Chemistry: Proceedings of the Symposium, Tel Aviv University 28–30 August 1988_ (Springer, 1989) pp. 151–170.
  * Geertsen, Rittby, and Bartlett (1989) J. Geertsen, M. Rittby, and R. J. Bartlett, “The equation-of-motion coupled-cluster method: Excitation energies of be and co,” Chem. Phys. Lett. 164, 57–62 (1989).
  * Comeau and Bartlett (1993) D. C. Comeau and R. J. Bartlett, “The equation-of-motion coupled-cluster method. applications to open-and closed-shell reference states,” Chem. Phys. Lett. 207, 414–423 (1993).
  * Stanton and Bartlett (1993) J. F. Stanton and R. J. Bartlett, “The equation of motion coupled-cluster method. a systematic biorthogonal approach to molecular excitation energies, transition probabilities, and excited state properties,” J. Chem. Phys 98, 7029–7039 (1993).
  * Kállay and Surján (2000) M. Kállay and P. R. Surján, “Computing coupled-cluster wave functions with arbitrary excitations,” J. Chem. Phys 113, 1359–1365 (2000).
  * Hirata, Nooijen, and Bartlett (2000) S. Hirata, M. Nooijen, and R. J. Bartlett, “High-order determinantal equation-of-motion coupled-cluster calculations for electronic excited states,” Chem. Phys. Lett. 326, 255–262 (2000).
  * Kowalski and Piecuch (2001a) K. Kowalski and P. Piecuch, “The active-space equation-of-motion coupled-cluster methods for excited electronic states: Full eomccsdt,” J. Chem. Phys 115, 643–651 (2001a).
  * Kowalski and Piecuch (2001b) K. Kowalski and P. Piecuch, “Excited-state potential energy curves of ch+: a comparison of the eomccsdt and full eomccsdt results,” Chem. Phys. Lett. 347, 237–246 (2001b).
  * Kucharski  _et al._ (2001) S. A. Kucharski, M. Włoch, M. Musiał, and R. J. Bartlett, “Coupled-cluster theory for excited electronic states: The full equation-of-motion coupled-cluster single, double, and triple excitation method,” J. Chem. Phys 115, 8263–8266 (2001).
  * Hirata (2004) S. Hirata, “Higher-order equation-of-motion coupled-cluster methods,” J. Chem. Phys 121, 51–59 (2004).
  * Kállay and Gauss (2004) M. Kállay and J. Gauss, “Calculation of excited-state properties using general coupled-cluster and configuration-interaction models,” J. Chem. Phys 121, 9257–9269 (2004).
  * Loos  _et al._ (2019) P.-F. Loos, M. Boggio-Pasqua, A. Scemama, M. Caffarel, and D. Jacquemin, “Reference energies for double excitations,” J. Chem. Theory Comput. 15, 1939–1956 (2019).


# G-RCN: Optimizing the Gap between Classification and Localization Tasks for
Object Detection

Yufan Luo, Li Xiao
Institute of Computing Technology, Chinese Academy of Science
<EMAIL_ADDRESS>Corresponding Author

###### Abstract

Multi-task learning is widely used in computer vision. Currently, object
detection models utilize shared feature map to complete classification and
localization tasks simultaneously. By comparing the performance between the
original Faster R-CNN and that with partially separated feature maps, we show
that: (1) Sharing high-level features for the classification and localization
tasks is sub-optimal; (2) Large stride is beneficial for classification but
harmful for localization; (3) Global context information could improve the
performance of classification. Based on these findings, we proposed a paradigm
called Gap-optimized region based convolutional network (G-RCN), which aims to
separating these two tasks and optimizing the gap between them. The paradigm
was firstly applied to correct the current ResNet protocol by simply reducing
the stride and moving the Conv5 block from the head to the feature extraction
network,which brings 3.6 improvement of $AP_{70}$ on the PASCAL VOC dataset
and 1.5 improvement of $AP$ on the COCO dataset for ResNet50. Next, the new
method is applied on the Faster R-CNN with backbone of VGG16,ResNet50 and
ResNet101, which brings above 2.0 improvement of $AP_{70}$ on the PASCAL VOC
dataset and above 1.9 improvement of $AP$ on the COCO dataset. Noticeably, the
implementation of G-RCN only involves a few structural modifications, with no
extra module added.

## Introduction

The development of deep learning substantially improves the ability of
computer to perform the image analysis, especially when recognizing and
detecting objects. In recent years, the study of convolutional neural
networks(?; ?; ?; ?; ?; ?) promotes the acquisition of super-human level
recognition ability(?) on ImageNet(?) dataset and obtains 0.62 at $AP_{50}$
(?) on COCO(?) detection dataset respectively. Meanwhile, the correlation
between recognition and detection tasks has been caught to transfer feature
extraction networks from ImageNet recognition task to various detection
tasks(?). And also very deep feature extraction networks such as ResNet(?)
have been proven to bring performance boosting in both recognition and
detection tasks simultaneously, which indicates the similarity between
recognition and detection tasks.

However, although deep neural networks have achieved the super-human level
performance in 1000 classes of object recognition task, the detection networks
have not obtained comparable performance to human in 80 classes of object
detection tasks, which gives the evidence that there exists gap between
recognition and detection tasks. Object detection requires powerful
classification performance together with capacity of precise localization to
determine one in infinite candidate positions. Intuitively, classification
task needs more abstract and comprehensive information to tell whether the
object has some specific features, for example, the network may need to judge
whether an object has hands when recognizing a human. On the contrary,
localization task prefers detailed, location sensitive feature information,
for example, we need to extract the specific location information of a finger
in order to accurately localize a human. However, the widely used detection
networks, including both one-stage(?; ?) and two-stage models(?; ?; ?), employ
the paradigm with shared feature extraction neural networks which extract
features that contribute to both the classification and localization tasks
simultaneously. As shown later, this will be proven as a sub-optimal strategy,
because of the huge gap between the classification and localization tasks.

Several researches have been proposed to optimize this gap. For example, (?)
claims that localization and classification require translational variant
features and translational invariant features respectively, and proposes to
generate position-sensitive score map to solve this distinction. (?)
constructs different classification head and localization head to perform
localization and classification tasks separately. (?) attributes the
performance drop of the detection tasks to the large scale variation between
objects, and then presents a novel training method directed against the scale
mismatching during task transferring. But none of them discuss where the gap
exactly is and propose a pertinent solution.

This work aims at exploring and optimizing the gap between the classification
and localization tasks. We firstly separated the last block of the
convolutional layers for the classification and localization tasks to
demonstrate that sharing the high-level features for the classification and
localization tasks is sub-optimal. By introducing the attention mechanism to
obtain a more abstract feature map as well as reducing the stride of the
convolutional layers respectively, we further demonstrated that: (1) Large
stride is beneficial for classification but harmful for localization; (2)
Global context information could improve the performance of classification.
Based on the observations, we proposed a novel paradigm, which we refer as
Gap-optimized region based convolutional network(G-RCN) to separate the
classification and localization tasks and optimize the gap between them.
Noticeably, only the last few convolutional layers need to be separated
therefore the separation brings very little parameter growth.

The G-RCN was applied on the Faster R-CNN with backbone of VGG16, ResNet50 and
ResNet101, and tested on the PASCAL VOC and COCO dataset. The implementation
of G-RCN brings significant performance improvement for all the backbones,
with above 2.0 improvement of $AP_{70}$ on the PASCAL VOC dataset and above
1.9 improvement of $AP$ on the COCO dataset. We further explored that current
protocol which places the Conv5 block of the ResNet as the head to improve
performance is misleading, and correct it by reducing the stride and moving
the Conv5 block back to the feature extraction network. Noticeably, the simple
modification brings 3.6 improvement of $AP_{70}$ on the PASCAL VOC dataset and
1.5 improvement of $AP$ on the COCO dataset for ResNet50. Furthermore, since
the conflict between classification and localization tasks is universal, we
expect our paradigm to be applied to all the other state-of-the-art object
detection models.

## Related Work

R-CNN based detection models(?; ?; ?) bring standard paradigm of two stage
detection networks which is widely used in practice. Recent improvements on
object detection can mainly be summarized as three directions. The first
direction was to obtain more powerful representation by adding extra
structures to extract additional information or enhanced features. Among them,
methods of adding context information(?; ?; ?) are in the primary direction.
For example, (?) obtains more discriminative and diverse features by adding
appearance features of different regions around targets and enhancing the
representation ability by multi-task learning with segmentation task. Besides,
the Mask R-CNN(?) for the object detection task also uses the auxiliary
information from segmentation task to achieve performance improvement. The
popularity of the attention model(?; ?) not only produces the Relation
Network(?) which is the current best use of the context information on the
object detection network, but also brings stronger feature representation and
leads to performance boosting on detection, segmentation and classification
tasks(?).

Another way to improve network performance is by making better use of the
training data. Hard example mining(?) is representative of this approach by
learning from difficult samples. Also, some studies try to control the
learning procedure by designing appropriate loss function of the network. For
example, focal loss(?) prevents the learning bias of the network by designing
a new cross-entropy function directed against imbalance between
positive/negative samples and simple/hard samples.

The third way relies on improving the expressive capacity of the feature
extraction network. For example, the DCN(?) enables the network to extract
more focused features by using deformable convolutional layers and deformable
ROI pooling layers. The FPN(?) develops a top-down architecture and builds
literal connections that allow each layer of the feature extraction network to
have precise detailed information and rich semantic information
simultaneously. SE-Net(?) utilizes the attention mechanism by assigning
different importance weights to different channels. ResNet(?),Inception-
ResNet(?) and ResNetXt(?) construct deeper thus more powerful feature
extraction networks and improve performance in both classification and
detection tasks.

Our work is also related to DetNet(?), which designs a backbone that is
specifically for detection problems. Besides, (?) suggests to use unshared
parameter of heads while we suggest to use unshared feature extraction
network.

## Exploring the Gap between Classification and Localization

In this section, we demonstrated that the widely used detection paradigm which
performs classification and localization tasks simultaneously is suboptimal.
To that end, we conducted several experiments based on the Faster R-CNN (?)
with backbone of VGG16(?) as the baseline and several experiments was done to
explore the gap. The super-parameters and settings were set the same as (?) in
all the experiments unless announced specially. The modified architectures
were all trained and validated based on the COCO 2014 detection datasets.
Based on the findings, we modified the paradigm slightly and provide
significant performance improvements without bells and whistles. As the
development is based on the general region-based object detection paradigm, it
can be extend to apply to other state-of-the-art object detection models.

### Separation Improves Performance

Figure 1: Models used to explore the gap between classification and
localization. F represents backbone and R represents head.

We firstly conducted several experiments to prove that there are clear gaps
between classification and localization tasks. We separated classification and
localization tasks in different parts of detection models to observe the
performance of each corresponding modified architecture.

| AP | $AP_{50}$ | $AP_{75}$
---|---|---|---
Faster R-CNN | 21.3 | 42.0 | 19.8
detect-F | 21.8 | 41.9 | 20.7
detect-RF | 21.9 | 41.9 | 20.9
detect-S | 21.3 | 41.6 | 19.9
Table 1: Model performances to demonstrate the existence of gap between
classification and localization.VGG16 is adopted as backbone.The structures of
different models are shown in Figure 1.

Three different models are developed to demonstrate the improvement of
performance when separating the classification and localization tasks. All
filters are initialized by the pre-trained network on ImageNet, including
those filters that are separated for the two tasks. As shown in Figure 1, the
detect-F is constructed by using two different conv5 blocks for classification
and localization tasks, and generates anchor proposals for the two tasks
separately. The detect-RF adopts two independent multi-layer perceptions in
the head for classification and localization respectively. The detect-S is
constructed as a comparison with detect-RF, which adds another head but not
separate the two tasks, and therefore it has the same level of parameters as
the detect-RF.

It is worth mention that in detect-F and detect-RF, the original RPN is
firstly applied on the feature maps of the classification branches to generate
anchors and predict a score for each anchor, and then the top 2000 anchors is
selected and passed to the head R to perform classification. At the same time,
the selected anchors are mapped onto the feature maps of the localization
branches to perform localization.

Results are summarized in Table 1. Different from other related researches(?),
we not only demonstrated the existence of gap between classification and
localization, but also experimentally prove that the separation of these two
tasks can be applied on the feature map level to improve performance.
Interestingly, the detect-F has the similar performance as detect-RF, but only
has a few convolutional layers more than the Faster R-CNN. On the contrary,
the detect-S does not bring any performance improvement. This eliminates the
argument that the performance boosting of the detect-RF is brought by the
dramatic increasing of parameters. It is worth mention that the separation of
other convolutional blocks does not bring additional improvement, which
suggests that classification and localization are both beneficial for low-
level and middle-level feature extraction while brings distinction on the
high-level features.

In the rest of this article, we will discuss what each task needs and where is
the gap, and then proceed to further modify the detection paradigm aiming at
each task.

### Global context on detection

The influence of context information on detection remains a mystery. We
suggest that rich global context information, which adds more auxiliary and
diversified information, can improve the classification significantly. On the
contrary, as with intuition, rich global context information may confuse the
localization process and bring no improvement. In the below experiments, we
extract global context information based on (?) and enhance object features by
combining the context information of the whole image.

The backbone generates feature map $G$ by filtering the whole image, and then
the ROI pooling layer crops activations lay inside proposals and generate
activation map $P$ with a fixed size. In our experiments, the VGG16 backbone
is used and thus the dimensions of the feature maps are 512. As some global
context information may not be relevant or useful , we adopted an attention
procedure to select information that is beneficial for the proposal $P$ from
the global context information $G$. In our experiments, $P$ which is the
result of 7*7 ROI pooling of proposal constitutes a collection of 49 queries,
each query is represented by a 512-dimensional vector. Then we perform $G$
with a 14*14 ROI pooling to obtain $V$ which constitutes a set of 196 key-
value pairs (key and value are the same things here), each key-value pair is
also a 512-dimension vector. We used an attention model similar to (?), which
came from the multi-head dot-product attention module(?) to filter out noise
of the global context information.

$P=P+softmax(\frac{W_{P}PV^{T}W_{K}^{T}}{\sqrt{k}})W_{V}V$ (1)

| AP | $AP_{50}$ | $AP_{75}$
---|---|---|---
Faster R-CNN | 21.3 | 42.0 | 19.8
context/cls | 22.1 | 42.9 | 20.9
context/loc | 21.4 | 41.4 | 20.1
Table 2: Model performances to demonstrate the impact of global context
information on object detection. (context/cls): adding global context
information for classification. (context/loc): adding global context
information for localization.

Among them, $W_{P}$ , $W_{K}$ and $W_{V}$ are matrices obtained through
learning. As the channels of $V$ and $P$ are corresponding one-by-one, we
discarded $W_{V}$ and observed slightly performance improvement. The distances
between vectors represent the relativities of the semantic features.
Therefore, the computation is equivalent to calculating the correlation
between each part of the region proposal $P$ and each part of the original
feature map $G$, and then enhancing the proposal representation with the
semantic information of the image according to the degree of correlations. In
order to investigate the impact of context information on classification and
localization tasks, we separately added the global context information on the
two tasks. The results are summarized in Table 2. As shown, the performance
improvement support our suggestions that the performance improvement is mainly
reflected on classification.

### Stride is essential in localization

| AP | $AP_{50}$ | $AP_{75}$ | $AP_{s}$ | $AP_{m}$ | $AP_{l}$
---|---|---|---|---|---|---
detect-F | 21.8 | 41.9 | 20.7 | 5.7 | 24.0 | 34.3
-pool/cls | 21.0 | 39.8 | 20.2 | 6.3 | 23.0 | 31.1
-pool/loc | 22.9 | 42.5 | 22.7 | 7.1 | 25.9 | 34.6
conv(2,2,2)/loc | 21.4 | 41.7 | 20.1 | 5.8 | 23.4 | 33.7
pool(2,2,1)/loc | 23.0 | 42.4 | 22.9 | 7.2 | 26.3 | 34.6
Table 3: Model performance to demonstrate the impact of global context on
detection. (-pool/cls): remove the last pooling layer on classification
branch; (-pool/loc): remove the last pooling layer on localization branch;
conv(2,2,2)/loc: replace the last pooling layer on localization branch with a
2*2 convolutional layer with a stride of 2; pool(2,2,1)/loc: change the stride
of the last pooling layer on localization branch from 2 to 1.

Pooling layers are widely used in the convolutional neural networks. Whether
in classification, segmentation or detection tasks, the pooling layer serves
as an important component of the feature extraction network, which plays an
essential role in reducing resolution, decreasing parameters, and transmitting
primary information. Nevertheless, the pooling layer brings defects including
loss of suboptimal information and inaccurate alignment. ResNet(?) suggests to
use 2*2 convolution kernel with stride of 2 to replace pooling layer for
keeping information while reducing the resolution of feature maps. Hinton(?)
proposes a new design of capsule network to replace the pooling layers.
However, currently there is no research quantitatively study the impact of
pooling on object detection. We believe that pooling brings different
influence on classification and localization tasks. Pooling is important for
the classification task, as it enlarges receptive fields and integrates
information of the target. On the contrary, the localization task needs
smaller strides thus needs fewer pooling layers, as the local deviation caused
by the pooling layer will greatly influence the localization accuracy,
especially when a high IOU threshold is applied. Therefore, we construct
several experiments to prove that the number of pooling layers in current
detection networks are more than what they need.

In order to study the impact of pooling on classification and localization
tasks separately, we adopted detect-F as the baseline to separate the two
tasks since it brings very little parameter growth. The conv5 block of the
VGG16 backbone is split for the two tasks. The details are summarized in Table
3. For comparison, the last pooling layers are removed either from the
classification branch(-pool/cls) or from localization branch(-pool/loc) in the
backbone. To ensure the representation capacity of the network, the
convolutional layers following the removed pooling layer are remained.

Figure 2: Paradigm of the Gap-optimized Region Based Convolutional
Network(G-RCN). (cls) represents classification branch, (loc) represents
localization branch. F represents backbone and R represents head.

Results show that the pooling layer improves the performance of the
classification branch, but on the contrary, it brings performance degradation
to the localization branch. Interestingly, we observed that when the pooling
layer in the localization branch is removed, not only small objects, but also
medium and large objects have performance improvement. On the contrary, there
is a significant performance drop when the pooling layer in the classification
branch is removed. Therefore, the improvement is not due to decreasing of
resolution.

Furthermore, to prove that this performance change is not caused by loss of
information during pooling, we adopted a 2*2 convolution kernel with stride of
2 (cov(2,2,2)/loc) and a 2*2 pooling layer with stride of 1 (pool(2,2,1)/loc)
to replace the pooling layer in the localization branch as comparison
experiments. Interestingly, the 2*2 convolution kernel with stride of 2 has a
performance drop even it does not lose information like pooling. On the
contrary, a 2*2 pooling layer with stride of 1 significantly improves the
performance even it still loses information.

We suggested that this phenomenon is caused by the deviation in localization.
Intuitively, the stride determines the distance between two adjacent receptive
fields on the feature map. Therefore, the network is not sensible to any
changes within the stride, small stride will directly brings localization
improvement. This is reflected in the phenomenon that performance improvement
of the localization network after removing the pooling layer is much more
significant on $AP_{75}$ than that on $AP_{50}$, and also the removal of the
pooling layer can bring the detection performance improvement for all scales
of objects. On the contrary, classification prefers larger stride with larger
receptive fields to integrate global information.

## Gap-optimized Region Based Convolutional Network

In this section, we combined our discovery in the above section and developed
a new paradigm from the general region based object detection models. We
achieved this by a new construct that we refer to as Gap-optimized region
based convolutional network(G-RCN), the detailed structure is shown in Figure
2. As the stride of the detection model denotes the smallest grain can be
recognized, our goal is to limit the stride of detection models for more
precise localization. We achieve this without adding any additional modules or
information but decreasing the stride in the localization branch in
traditional detection models.

### ResNet-det protocol

The implementation of ResNet-based detection models originally followed the
protocol as in (?), in which all layers of conv5 block of ResNet are adopted
as the head and attach to the RoI pooling layer. The backbone is constituted
by all the layers in the first four blocks and generate feature maps shared
for classification and localization. And also, there are experiments (?)
demonstrating that utilizing the conv5 block in the head can lead to better
performance than putting the Conv5 block into the feature extraction network
and using shared MLP in the head. Based on Section(cite), we think this
protocol is misleading as the localization performance will drop dramatically
if put the conv5 layers forward directly. Obviously, placing the Conv5 block
on the feature extraction network leads to more powerful and abstract feature
representation which is beneficial for classification. The performance drop is
mainly caused by the stride on the conv5 block which degrades the localization
accuracy, and can be improved easily.

We modified the ResNet-based protocol for detection(named ResNet-det) which
simply changes the stride of the first convolutional layer of the Conv5 block
from 2 to 1 and moves the block from the head back to the feature extraction
network. The head is added back with two fully-connected layers similar with
(?). This modification will be proved to bring a significant improvement in
the experimental part, though it is a small change.

### Faster R-CNN

We also applied G-RCN on the Faster R-CNN with various backbones, we firstly
separate the classification and localization tasks from current paradigm since
sharing the two tasks is proven to be sub-optimal. To that end, we split these
two tasks in the feature extraction network. As multi-task learning is helpful
for learning the low-level features, we start splitting at the last few layers
of our backbone. On the classification branch, an attention procedure is added
to update the feature map according to the correlation between a local part
and the global context of the feature map as the global context information is
beneficial for classification. On the contrary, since localization requires
more detailed information and larger stride usually leads to larger deviation,
the stride of the localization branch is reduced(denoted as conv_loc block).
Finally, both classification and localization branches share a common head to
prevent a significant increase of parameters.

## Dataset and Evaluation

Method |  $AP_{70}$ |  aero |  bike |  bird |  boat |  bottle |  bus |  car |  cat |  chair |  cow |  table |  dog |  horse |  mbike |  persn |  plant |  sheep |  sofa |  train |  tv
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
VGG16(Baseline) |  55.8 |  59.0 |  60.0 |  51.6 |  39.0 |  43.9 |  75.3 |  68.6 |  66.0 |  36.4 |  59.0 |  53.9 |  61.6 |  67.8 |  58.1 |  54.3 |  17.6 |  58.8 |  59.2 |  62.5 |  63.8
VGG16 -pool/loc |  56.6 |  60.2 |  63.2 |  50.0 |  41.2 |  45.3 |  69.1 |  70.4 |  68.5 |  39.6 |  63.1 |  49.5 |  60.8 |  67.2 |  58.8 |  54.8 |  22.2 |  59.2 |  57.3 |  65.2 |  67.2
VGG16 -pool/loc + context/cls (G-RCN) |  58.1 |  60.9 |  60.8 |  51.8 |  41.2 |  43.5 |  73.7 |  70.4 |  66.0 |  40.3 |  73.4 |  53.5 |  62.5 |  65.6 |  62.5 |  56.4 |  25.1 |  63.6 |  60.5 |  64.2 |  66.5
ResNet101(Baseline) |  60.6 |  67.6 |  67.7 |  58.7 |  44.3 |  46.2 |  76.8 |  69.1 |  77.0 |  40.3 |  62.7 |  51.9 |  65.2 |  68.7 |  66.8 |  57.3 |  30.5 |  57.4 |  71.4 |  66.9 |  65.5
ResNet101(G-RCN) |  63.0 |  66.0 |  67.1 |  61.5 |  52.2 |  49.3 |  77.1 |  76.1 |  74.1 |  42.1 |  70.3 |  53.8 |  74.0 |  68.7 |  63.2 |  58.3 |  30.3 |  66.6 |  68.6 |  66.8 |  73.1
ResNet50(Baseline) |  55.9 |  58.3 |  58.6 |  44.8 |  40.3 |  40.0 |  76.2 |  68.5 |  70.5 |  34.5 |  61.8 |  54.0 |  61.8 |  67.5 |  57.0 |  54.6 |  23.4 |  58.0 |  60.9 |  64.0 |  63.3
ResNet50(G-RCN) |  57.9 |  66.7 |  57.4 |  53.7 |  46.8 |  43.9 |  76.1 |  69.1 |  73.9 |  33.2 |  66.7 |  51.3 |  62.3 |  64.5 |  64.3 |  56.1 |  27.2 |  60.7 |  57.8 |  63.3 |  63.1
ResNet50-det(Baseline) |  59.5 |  61.5 |  66.1 |  56.9 |  44.3 |  43.5 |  75.9 |  69.8 |  73.1 |  38.9 |  61.3 |  58.8 |  63.7 |  68.1 |  66.4 |  56.9 |  23.6 |  62.7 |  66.4 |  66.8 |  64.7
Table 4: Performance of G-RCN on the PASCAL VOC dataset. Baseline:Faster R-CNN
framework without separation of classification and localization.(-pool/loc):
separate the two tasks and remove the last pooling layer on the localization
branch; (context/cls): adding global context information on classification
branch. ResNet50-det reduces the stride and moves the Conv5 block back to the
feature extraction network.

Datasets and Metrics We evaluated our proposed paradigm on the COCO 2014
detection dataset(?) which contains 80 categories and PASCAL VOC 2007+2012
datasets(?) which contains 20 categories. On the COCO dataset, we trained our
image on the training set of 83,000 images and tested the performance on the
validation set of 40,000 images. we adopted standard COCO metrics including
AP, $AP_{50}$, $AP_{75}$, $AP_{small}$, $AP_{medium}$, $AP_{large}$ to
evaluate model performances. Small, medium, large objects are defined as whose
size is less than 32*32, more than 32*32 and less than 96*96, more than 96*96
respectively. On the PASCAL VOC dataset, following the protocol in (?), we
used the VOC 2007 trainval dataset of 5,000 images and VOC 2012 trainval
dataset of 11,000 images for training, and tested models on VOC 2007 test set
of 5,000 images. Since objects on the VOC dataset are relatively large and
easy to detect, we used $AP_{70}$ as the evaluation metric to highlight the
performance improvement. As is commonly practiced, we adopted pretrained
backbone of VGG and ResNet to verify the generalizability of the G-RCN. The
results on the PASCAL VOC dataset is shown in Table 4 and the results on the
COCO dataset is shown in Table 2.

Method | AP | $AP_{50}$ | $AP_{75}$ | $AP_{small}$ | $AP_{medium}$ | $AP_{large}$
---|---|---|---|---|---|---
VGG16(Baseline) | 21.3 | 42.0 | 19.8 | 5.7 | 23.2 | 33.3
VGG16 -pool/loc | 22.9 | 42.5 | 22.7 | 7.1 | 25.9 | 34.5
VGG16 -pool/loc + context/cls (G-RCN) | 23.3 | 43.4 | 22.9 | 7.8 | 26.5 | 34.6
ResNet101(Baseline) | 22.8 | 43.7 | 21.7 | 5.9 | 24.6 | 37.0
ResNet101 (G-RCN) | 25.3 | 44.8 | 25.8 | 7.7 | 28.6 | 39.4
ResNet50(Baseline) | 20.8 | 40.9 | 19.3 | 5.4 | 22.7 | 33.4
ResNet50(G-RCN) | 22.7 | 42.0 | 22.5 | 6.9 | 26.1 | 35.4
ResNet50-det(Baseline) | 22.3 | 43.2 | 21.0 | 5.9 | 24.2 | 35.6
Table 5: Performance of G-RCN on the COCO dataset. Baseline: Faster R-CNN
framework without separation tasks.(-pool/loc): separate the two tasks and
remove the last pooling layer on localization branch; (context/cls): adding
global context information on classification branch. ResNet50-det reduces the
stride and moves the Conv5 block back to the feature extraction network.222New
results will be updated soon.

Implementation Details We implemented our models and conduct experiments on
Tensorflow(?). The shorter side of input image is resized to 600 pixels during
preprocessing. Each mini-batch contains 1 image, the RPN is trained with
batchsize of 256 and Fast R-CNN is trained with 128 Region of interest (ROI)s
per batch. SGD is adopted with a momentum of 0.9. We tested our models with
NMS of 0.3 and ROIs as 300 per image. For COCO dataset, the learning rate is
set as 0.001 for the first 240k iterations and 0.0001 for the rest 80k
iterations, totally 320k iterations. Anchors with five sizes and three aspect
ratios are adopted. For PASCAL VOC dataset, the learning rate is set as 0.001
for the first 150k iterations and 0.0001 for the rest 30k iterations, totally
180k iterations. Anchors with three sizes and three aspect ratios are adopted.

### G-RCN with VGG backbone

We firstly applied G-RCN on the Faster R-CNN with VGG16 backbone. Same as that
in the above section, the separation starts at the conv5 block with the last
pooling layer on the localization branch removed (-pool/loc). An attention
procedure is then applied on the classification branch to add the global
context information on the feature maps (+context/cls). As shown in Table 4
and Table 2 , the implementation of G-RCN receives 2.3 $AP_{70}$ improvement
on the PASCAL VOC and 2.0 $AP$ improvement on the COCO dataset. Interestingly,
the performance boosting on the COCO dataset is mainly contributed by
separation the two tasks and removing the pooling layer, while the performance
boosting on the PASCAL VOC dataset is mainly due to adding the global context
information. This may because that there are more small objects on the COCO
dataset, which are more sensitive to the stride. On the contrary, the objects
on the PASCAL VOC dataset are relatively large, therefore the overall
performance may more depends on classification.

### G-RCN with ResNet backbone

In this section, we adopted the G-RCN on the Faster R-CNN with RestNet50,
ResNet101 backbone, and proposed a way to separate these two tasks. Separation
for ResNet101 starts at the last 6 bottlenecks in the conv4 block, and the
conv5 block remains to be used in the head as original. The first
convolutional layer of the first bottleneck in conv4 block, which originally
has a stride of 2, is modified to a stride equal to 1 for the localization
branch. But both classification and localization branches share the same
kernels of the first 17 bottlenecks of the conv4 block. Similarly, the
separation for ResNet50 applies on the last 2 bottlenecks of the Conv4 block
and the stride of the first convolutional layer of the Cov4 block is reduced
from 2 to 1. Detailed architectures will be provided in the supplementary
material.

As shown in Table 4 and Table 2, the implementation of G-RCN on ResNet101
receives 2.4 $AP_{70}$ improvement on the PASCAL VOC dataset and 2.5 $AP$
improvement on the COCO dataset. The G-RCN also brings 2.0 $AP_{70}$
improvement on the PASCAL VOC dataset and 1.9 $AP$ improvement on the COCO
dataset for ResNet50.

### Evaluation of ResNet-det

_ResNet-det_ follows the modification we proposed above, we performed RoI
Align with an output size of $7\times 7$ rather than $14\times 14$ to save
memory footprint on the conv5 layers which modified the stride to 1 for
advancing localization performance(these two pooling brings similar
performance). Surprisingly, as shown in Table 4 and Table 2, ResNet50+ brings
3.6 $AP_{70}$ improvement on PASCAL VOC dataset and 1.5 $AP$ improvement COCO
dataset. We also attempt to replace the fc layers with two convolution layers
or conv5 block of ResNet but observed no improvement.

## Conclusion

In this work, we presented an analysis to separate the classification and
localization tasks for the region based object detection model and study the
effect on the two tasks separately. we explored that: (1) Sharing the high-
level features for the classification and localization tasks is sub-optimal;
(2) Large stride is beneficial for classification but harmful for
localization; (3) Global context information could improve the performance of
classification. Based on the analysis, a new paradigm(G-RCN) was proposed to
separate the classification and localization tasks and optimizing the gap
between them. Experimental results on the PASCAL VOC dataset and COCO dataset
demonstrated the significant performance improvement in object detection when
applying G-RCN. We further explored that current protocol which places the
Conv5 block of ResNet as the head is misleading, one can greatly improve the
performance by simply reducing the stride and moving the Conv5 block back to
the feature extraction network. In the future, we would like to further
improve the ResNet based object detection model by combining this strategy
with the implementation of G-RCN, we would also like to extend the G-RCN to
apply on the other models such as YOLO, FPN. Furthermore, there may also
exists conflict between classification and segmentation tasks, we would also
like to explore that.

## References

  * [Abadi et al.(2016)Abadi, Barham, Chen, Chen, Davis, Dean, Devin, Ghemawat, Irving, Isard, et al.] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.  Tensorflow: A system for large-scale machine learning.  In _12th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 16)_, pp. 265–283, 2016.
  * [Bell et al.(2016)Bell, Lawrence Zitnick, Bala, and Girshick] Bell, S., Lawrence Zitnick, C., Bala, K., and Girshick, R.  Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2874–2883, 2016.
  * [Cheng et al.(2018)Cheng, Wei, Shi, Feris, Xiong, and Huang] Cheng, B., Wei, Y., Shi, H., Feris, R., Xiong, J., and Huang, T.  Revisiting rcnn: On awakening the classification power of faster rcnn.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 453–468, 2018.
  * [Dai et al.(2016)Dai, Li, He, and Sun] Dai, J., Li, Y., He, K., and Sun, J.  R-fcn: Object detection via region-based fully convolutional networks.  In _Advances in neural information processing systems_ , pp. 379–387, 2016.
  * [Dai et al.(2017)Dai, Qi, Xiong, Li, Zhang, Hu, and Wei] Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., and Wei, Y.  Deformable convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 764–773, 2017.
  * [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  2009\.
  * [Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and Zisserman] Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ , 88(2):303–338, 2010.
  * [Gidaris & Komodakis(2015)Gidaris and Komodakis] Gidaris, S. and Komodakis, N.  Object detection via a multi-region and semantic segmentation-aware cnn model.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 1134–1142, 2015.
  * [Girshick(2015)] Girshick, R.  Fast r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 1440–1448, 2015.
  * [Girshick et al.(2014)Girshick, Donahue, Darrell, and Malik] Girshick, R., Donahue, J., Darrell, T., and Malik, J.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 580–587, 2014.
  * [He et al.(2016)He, Zhang, Ren, and Sun] He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * [He et al.(2017)He, Gkioxari, Dollár, and Girshick] He, K., Gkioxari, G., Dollár, P., and Girshick, R.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2961–2969, 2017.
  * [Hu et al.(2018a)Hu, Gu, Zhang, Dai, and Wei] Hu, H., Gu, J., Zhang, Z., Dai, J., and Wei, Y.  Relation networks for object detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3588–3597, 2018a.
  * [Hu et al.(2018b)Hu, Shen, and Sun] Hu, J., Shen, L., and Sun, G.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 7132–7141, 2018b.
  * [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] Krizhevsky, A., Sutskever, I., and Hinton, G. E.  Imagenet classification with deep convolutional neural networks.  In _Advances in neural information processing systems_ , pp. 1097–1105, 2012.
  * [Li et al.(2018)Li, Peng, Yu, Zhang, Deng, and Sun] Li, Z., Peng, C., Yu, G., Zhang, X., Deng, Y., and Sun, J.  Detnet: A backbone network for object detection.  _arXiv preprint arXiv:1804.06215_ , 2018.
  * [Lin et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick] Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pp. 740–755. Springer, 2014.
  * [Lin et al.(2017a)Lin, Dollár, Girshick, He, Hariharan, and Belongie] Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2117–2125, 2017a.
  * [Lin et al.(2017b)Lin, Goyal, Girshick, He, and Dollár] Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2980–2988, 2017b.
  * [Liu et al.(2016)Liu, Anguelov, Erhan, Szegedy, Reed, Fu, and Berg] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A. C.  Ssd: Single shot multibox detector.  In _European conference on computer vision_ , pp. 21–37. Springer, 2016.
  * [Peng et al.(2018)Peng, Xiao, Li, Jiang, Zhang, Jia, Yu, and Sun] Peng, C., Xiao, T., Li, Z., Jiang, Y., Zhang, X., Jia, K., Yu, G., and Sun, J.  Megdet: A large mini-batch object detector.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 6181–6189, 2018.
  * [Redmon et al.(2016)Redmon, Divvala, Girshick, and Farhadi] Redmon, J., Divvala, S., Girshick, R., and Farhadi, A.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 779–788, 2016.
  * [Ren et al.(2015)Ren, He, Girshick, and Sun] Ren, S., He, K., Girshick, R., and Sun, J.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _Advances in neural information processing systems_ , pp. 91–99, 2015.
  * [Sabour et al.(2017)Sabour, Frosst, and Hinton] Sabour, S., Frosst, N., and Hinton, G. E.  Dynamic routing between capsules.  In _Advances in neural information processing systems_ , pp. 3856–3866, 2017.
  * [Shrivastava et al.(2016)Shrivastava, Gupta, and Girshick] Shrivastava, A., Gupta, A., and Girshick, R.  Training region-based object detectors with online hard example mining.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 761–769, 2016.
  * [Simonyan & Zisserman(2014)Simonyan and Zisserman] Simonyan, K. and Zisserman, A.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * [Singh & Davis(2018)Singh and Davis] Singh, B. and Davis, L. S.  An analysis of scale invariance in object detection snip.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3578–3587, 2018.
  * [Szegedy et al.(2015)Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, and Rabinovich] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1–9, 2015.
  * [Szegedy et al.(2017)Szegedy, Ioffe, Vanhoucke, and Alemi] Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. A.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017\.
  * [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pp. 5998–6008, 2017.
  * [Xie et al.(2017)Xie, Girshick, Dollár, Tu, and He] Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1492–1500, 2017.
  * [Xu et al.(2015)Xu, Ba, Kiros, Cho, Courville, Salakhudinov, Zemel, and Bengio] Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y.  Show, attend and tell: Neural image caption generation with visual attention.  In _International conference on machine learning_ , pp. 2048–2057, 2015.