# Detection of Cosmic Magnification via Galaxy Shear - Galaxy Number Density
Correlation
from HSC Survey Data

Xiangkun Liu<EMAIL_ADDRESS>South-Western Institute for Astronomy Research,
Yunnan University, Kunming 650500, P.R.China  Dezi Liu  South-Western
Institute for Astronomy Research, Yunnan University, Kunming 650500, P.R.China
Zucheng Gao  School of Physics, Peking University, Beijing 100871, P.R.China
Chengliang Wei  Purple Mountain Observatory, Chinese Academy of Sciences,
Nanjing, 210023, P.R.China  Guoliang Li  Purple Mountain Observatory, Chinese
Academy of Sciences, Nanjing, 210023, P.R.China  School of Astronomy and Space
Science, University of Science and Technology of China, Hefei, 230026,
P.R.China  Liping Fu  The Shanghai Key Lab for Astrophysics, Shanghai Normal
University, 100 Guilin Road, Shanghai 200234, P.R.China  Toshifumi Futamase
Department of Astrophysics and Meteorology, Kyoto Sangyo University, Kita-ku,
Kyoto 603-8555, Japan  Zuhui Fan<EMAIL_ADDRESS>South-Western Institute
for Astronomy Research, Yunnan University, Kunming 650500, P.R.China

###### Abstract

We propose a novel method to detect cosmic magnification signals by cross-
correlating foreground convergence fields constructed from galaxy shear
measurements with background galaxy positional distributions, namely shear-
number density correlation. We apply it to the Hyper Suprime-Cam Subaru
Strategic Program (HSC-SSP) survey data. With 27 non-independent data points
and their full covariance, $\chi_{0}^{2}\approx 34.1$ and $\chi_{T}^{2}\approx
24.0$ with respect to the null and the cosmological model with the parameters
from HSC shear correlation analyses in Hamana  _et al._ [1], respectively. The
Bayes factor of the two is $\log_{10}B_{T0}\approx 2.2$ assuming equal model
probabilities of null and HSC cosmology, showing a clear detection of the
magnification signals. Theoretically, the ratio of the shear-number density
and shear-shear correlations can provide a constraint on the effective
multiplicative shear bias $\bar{m}$ using internal data themselves. We
demonstrate the idea with the signals from our HSC-SSP mock simulations and
rescaling the statistical uncertainties to a survey of $15000\deg^{2}$. For
two-bin analyses with background galaxies brighter than $m_{lim}=23$, the
combined analyses lead to a forecasted constraint of $\sigma(\bar{m})\sim
0.032$, $2.3$ times tighter than that of using the shear-shear correlation
alone. Correspondingly, $\sigma(S_{8})$ with
$S_{8}=\sigma_{8}(\Omega_{\mathrm{m}}/0.3)^{0.5}$ is tightened by $\sim 2.1$
times. Importantly, the joint constraint on $\bar{m}$ is nearly independent of
cosmological parameters. Our studies therefore point to the importance of
including the shear-number density correlation in weak lensing analyses, which
can provide valuable consistency tests of observational data, and thus to
solidify the derived cosmological constraints.

##  I Introduction

Weak gravitational lensing effects (WL), mainly consisting of cosmic shear and
magnification, probe directly the matter distribution of the Universe, and are
uniquely important in cosmological studies [2, 3, 4, 5, 6].

Cosmic shear induces tiny image distortions on distant galaxies. By accurately
measuring their shapes, shear signals can be extracted statistically. This has
been the main stream of current weak lensing studies. With large surveys, such
as the on-going Dark Energy Survey [DES, 7], the Kilo-Degree Survey [KiDS, 8],
and the Hyper Suprime-Cam Subaru Strategic Program survey[HSC-SSP, 9], the
derived cosmological constraints are improving significantly [e.g., 10, 11,
12, 1].

Cosmic magnification, on the other hand, leads to slight changes of the galaxy
spatial distribution due both to the lensing magnification of galaxy flux and
the lensing increase of solid angle [13, 14]. In principle, cosmic
magnification can be extracted from photometric surveys without the need of
shape measurements. There have been extensive theoretical investigations on
the complementarity of cosmic magnification and shear and how to accurately
reconstruct cosmic magnification power spectrum purely from the spatial
distribution of galaxies [e.g., 15, 16, 17, 18, 19, 20], where the galaxy
intrinsic clustering and the bias are the main concerns. Observational
detections of cosmic magnification using galaxy distributions have been
reported [e.g., 21, 22, 23].

Currently, most of the studies concentrate either on cosmic shear or on cosmic
magnification. Combined analyses employing both sides of the signals are done
for clusters of galaxies [24], but are still lacking on cosmic scales. In this
paper, we put forward a method to cross-correlate foreground shear and
background position of galaxies. This not only can lead to clean detections of
cosmic magnification signals without being affected by galaxy bias, but also
can, by combining with shear-shear correlation, provide important consistency
tests of observational data themselves and further constrain the shear
measurement bias. Validated by mock simulations, we apply the analyses to HSC-
SSP data, and show cross-correlation signals with high significance. We also
forecast the potential constraint on the multiplicative bias parameter $m$ of
shear measurement from future large surveys.

The paper is organized as follows. In Sec. II, we present our cross-
correlation methodology theoretically. Sec. III describes the HSC-SSP data
used in our analyses. Sec. IV contains details of the cross-correlation
analyses. In Sec. V, we show the results measured from our HSC mock
simulations and from observations. The cosmological potential of combining the
shear-shear and shear-number density correlations is shown in Sec. VI. Summary
and discussions are given in Sec. VII.

##  II Method

In our analyses, we cross-correlate the foreground convergence field
constructed from the shear measurements and the spatial distribution of
background galaxies (hereafter, shear-number density correlation).
Specifically, we have

$\omega_{ij}(\theta)=\langle{\kappa_{i}(\hat{\mathbf{n}})}\delta_{j}(\hat{\mathbf{n}}^{\prime},m_{lim})\rangle_{\theta},$
(1)

where $\theta$ is the angular separation of the two directional vectors
$\hat{\mathbf{n}}$, $\hat{\mathbf{n}}^{\prime}$, $\kappa_{i}$ is the
foreground lensing convergence field from the galaxy shear measurements, and
$\delta_{j}$ is the fluctuation field of background galaxy number count with
the limiting magnitude $m_{lim}$. The latter is given by

$\delta_{j}(\hat{\mathbf{n}}^{\prime},m_{lim})=\delta_{gj}(\hat{\mathbf{n}}^{\prime},m_{lim})+\delta_{{\mu}j}(\hat{\mathbf{n}}^{\prime},m_{lim}),$
(2)

where $\delta_{gj}$ describes the intrinsic clustering of the galaxy sample,
which is related the density fluctuation field with a bias factor involved.
The second term $\delta_{{\mu}j}$ is the fluctuation arising from the lensing
magnification effect.

Assuming the foreground and the background are well separated, the intrinsic
clustering term does not contribute to the correlation of Eqn.(1), and
therefore eliminating the impact of the galaxy bias. For $\delta_{{\mu}j}$, it
is [e.g., 2, 23]

$\delta_{{\mu}j}(\hat{\mathbf{n}}^{\prime},m_{lim})=\mu^{\alpha(m_{lim})-1}-1\approx
2\kappa_{j}(\hat{\mathbf{n}}^{\prime})[\alpha(m_{lim})-1],$ (3)

where
$\alpha(m_{lim})=2.5\mathrm{d}[\mathrm{log}_{10}N(m_{lim})]/{\mathrm{d}m_{lim}}$
is the local slope of the galaxy cumulative number count $N(m_{lim})$ in log
scale, and $\mu$ is the lensing magnification. The second expression in
Eqn.(3) is the approximation in the weak lensing limit with $\kappa\ll 1$ and
$\mu\approx 1+2\kappa$. We then have

$\displaystyle\omega_{ij}(\theta)$ $\displaystyle=\langle
2[\alpha(m_{lim})-1]\kappa_{i}(\hat{\mathbf{n}})\kappa_{j}(\hat{\mathbf{n}}^{\prime})\rangle_{\theta}$
(4)
$\displaystyle=2[\alpha(m_{lim})-1]\frac{1}{2\pi}\int_{0}^{\infty}l\mathrm{d}lP_{\kappa}^{ij}(l)J_{0}(l\theta),$

where $J_{0}$ is the zero-th order Bessel function and $P_{\kappa}^{ij}$ is
the convergence cross power spectrum between $i$ and $j$ given by

$P_{\kappa}^{ij}(l)=\int{\mathrm{d}w\frac{q_{i}(w)q_{j}(w)}{f_{K}^{2}(w)}}P_{\delta}\left(\frac{l}{f_{K}(w)},w\right),$
(5)

with

$q_{i(j)}(w)=\frac{3H_{0}^{2}\Omega_{\mathrm{m}}}{2a(w)c^{2}}\int_{w}^{w_{H}}\mathrm{d}w^{\prime}p_{i(j)}(w^{\prime})\frac{f_{K}(w^{\prime}-w)}{f_{K}(w^{\prime})}.$
(6)

Here $w$, $f_{K}(w)$, $P_{\delta}$ and $p_{i(j)}(w)$ are the comoving radial
distance, comoving angular diameter distance, 3-dimensional matter power
spectrum, and the radial distribution calculated from the redshift
distribution for $i$-th ($j$-th) redshift bin, respectively.
$\Omega_{\mathrm{m}}$, $H_{0}$, $a$ and $c$ are the present matter density
parameter, Hubble constant, cosmic scale factor, and the speed of light,
respectively. We note that the shear-number density correlation here is, in
some sense, the inverse of the galaxy-galaxy lensing analyses which correlate
foreground galaxy positions with background shear signals.

For the shear-shear correlation between $i$ and $j$, we have [10]

$\xi_{+ij}(\theta)=\langle\gamma_{ti}(\hat{\mathbf{n}})\gamma_{tj}(\hat{\mathbf{n}}^{\prime})+\gamma_{\times
i}(\hat{\mathbf{n}})\gamma_{\times
j}(\hat{\mathbf{n}}^{\prime})\rangle=\frac{1}{2\pi}\int_{0}^{\infty}l\mathrm{d}lP_{\kappa}^{ij}(l)J_{0}(l\theta).$
(7)

It is seen from Eqns.(4) and (7) that theoretically, $\omega_{ij}$ and
$\xi_{+ij}$ have exactly the same cosmology dependence with the ratio of the
two being a constant $2[\alpha(m_{lim})-1]$, independent of $\theta$.
Observationally, the shear measurements often involve a multiplicative bias
and an additive bias. While the additive bias can be estimated from the data,
the determination of the multiplicative bias $m$ needs extra calibrations.
Assuming that we have no knowledge of $m$, then the two correlations
calculated from the observational data should be, to a very good
approximation, $\hat{\omega}_{ij}=(1+\bar{m}_{i})\omega_{ij}$ and
$\hat{\xi}_{+ij}=(1+\bar{m}_{i})(1+\bar{m}_{j})\xi_{+ij}$, where $\bar{m}_{i}$
and $\bar{m}_{j}$ are the effective multiplicative bias of the shear sample
$i$ and shear sample $j$, respectively. Then

$\hat{\omega}_{ij}(\theta)/\hat{\xi}_{+ij}(\theta)=2[\alpha(m_{lim})-1]/(1+\bar{m_{j}}).$
(8)

Thus the ratio of Eqn.(8) provides a means to calibrate $m$ in a cosmology-
independent way using the observational data themselves if $\alpha$ can be
determined accurately.

In the following, we apply the cross-correlation analyses to the HSC-SSP data.

##  III Observational data

HSC-SSP is a large imaging survey with weak lensing cosmology as a major
science driver [9]. In our analyses, we use data from first and second data
release PDR1 and PDR2 [25, 26], and the first-year shear catalog (S16A) [27].

To implement the cross correlation analyses, we divide galaxies into low- and
high-redshift bins, with $0.2\leq z_{p}\leq 0.7$ and $1.0\leq z_{p}\leq 1.5$,
respectively, where $z_{p}$ is the best-fit value of the photometric redshift
(photo-z) of a galaxy [28, 29]. For the low-redshift bin, we use the S16A
shear catalog to construct the convergence field, adopting the selection
criteria listed in Table 4 of Mandelbaum  _et al._ [27]. The sample’s redshift
distribution is shown in blue in the left panel of Figure 1 by stacking the
photo-z distribution of individual galaxies.

Figure 1: Left: Normalized redshift distributions of the foreground galaxy
sample and background PDR2 galaxy subsamples with different magnitude cuts.
Right: $i$-band differential galaxy number density of PDR2 wide (red) in
comparison with that of HSC PDR1 COSMOS UltraDeep catalog (blue). Circles and
squares are for the full and high redshift samples, respectively.

For the high-redshift bin, we take the photometric data from PDR2. To ensure
the data quality and avoid introducing significant photometric bias, besides
the common flag criteria in Aihara  _et al._ [25], two extra cuts are applied:
the $i$-band flux signal-to-noise (S/N) ratio $\geq 5$ and
iblendness_abs_flux, a measure of how strongly an object is blended, $\leq
10^{-0.375}$. We further take i_extendedness_value = 1 at $i<24.5$ to remove
the impact of point sources [25]. In addition, we make a cross match in
position between S16A and PDR2, and exclude those galaxies with a small
position difference ($\leq 0.5$ arcsec) but a large magnitude difference
($\geq 0.5$), which amount to $\sim 3\%$ of the whole sample. For
magnification detections, we calculate the magnitude of each galaxy by
${mag}=\mathrm{i\\_cmodel\\_mag}-a_{i}$ with $a_{i}$ the Galactic extinction
correction in $i$-band given in the PDR2 catalog. The right panel of Figure 1
shows the magnitude distributions. By comparing with the HSC COSMOS UltraDeep
data [25], we find that the high-redshift sample is about 80% and 65% complete
up to $mag\approx 23.5$ and $24.5$, respectively. For magnification analyses,
we then construct different subsamples with ${mag}\leq 22.5$, $\leq 23.0$, and
$\leq 23.5$, respectively. The impact of the incompleteness is tested with our
mock simulations to be described later in the paper. The redshift
distributions of these subsamples are shown in different colors in the left
panel of Figure 1.

It is seen that the low- and high-redshift samples have only a small fraction
of redshift overlap. We numerically check the contamination from the galaxy-
convergence intrinsic correlation in the overlapped region. It is about
$1-2\%$ to $\omega_{ij}/\\{2[\alpha(m_{lim})-1]\\}$, negligible in the
analyses here. For future high precision studies, we should investigate this
contamination carefully.

To perform the analyses, we select 52 overlapped fields between S16A and PDR2,
with 40 having an area of $1.5\times 1.5\deg^{2}$ each, and 12 with $1.0\times
1.0\deg^{2}$ each cropped from partially-overlapped tracts. The total area is
$\sim 100\deg^{2}$. For the low-redshift shear sample, the weighted number
density is $n_{g}\sim 8.0~{}\mathrm{arcmin}^{-2}$. For the high-redshift PDR2
data, $n_{g}\sim 0.28$, $0.77$, and $1.69~{}\mathrm{arcmin}^{-2}$, for
$m_{lim}=22.5$, $23.0$, and $23.5$, respectively. We estimate the slope
$\alpha$ by bootstrapping the 52 fields to generate 1000 sets of data. The
average $\alpha$ for the three $m_{lim}$ is $2.48$, $1.97$, and $1.48$
calculated with the interval of $\Delta m=0.01$ around $m_{lim}$. The
corresponding $1\sigma$ uncertainty is $0.052$, $0.030$, and $0.018$. We test
by using $\Delta m=0.05$, and the $\alpha$ value differences are less than
$1\%$.

For the corresponding shear-shear correlation analyses, S16A is used for both
low- and high-redshift samples. We apply the same $m_{lim}$ to construct high-
redshift shear subsamples. Their redshift distributions are nearly identical
to the ones shown in the left panel of Figure 1.

##  IV Cross-correlation analyses

We first reconstruct the convergence fields from the low-redshift shear sample
with the same procedures detailed in Liu  _et al._ [30], which are summarized
here. (1) The smoothed shear fields are calculated following Oguri  _et al._
[31]. Specifically,

$\displaystyle\langle\bm{\epsilon}\rangle(\bm{\theta})$
$\displaystyle=\frac{\sum_{k}w_{k}W_{\theta_{\mathrm{G}}}(\bm{\theta_{k}-\theta})\bm{e}_{k}}{2\sum_{k}w_{k}(1-e_{\mathrm{rms},k}^{2})(1+m_{k})W_{\theta_{\mathrm{G}}}(\bm{\theta_{k}-\theta})}$
(9)
$\displaystyle-\frac{\sum_{k}w_{k}W_{\theta_{\mathrm{G}}}(\bm{\theta_{k}-\theta})\bm{c}_{k}}{\sum_{k}w_{k}(1+m_{k})W_{\theta_{\mathrm{G}}}(\bm{\theta_{k}-\theta})}$

where $\bm{e}_{k}$ is the two-component ellipticity of galaxy $k$, and
$w_{k}$, $\bm{c}_{k}$, $m_{k}$ and $e_{\mathrm{rms},k}$ are the corresponding
weight, additive and multiplicative biases of shear measurements, and the
intrinsic ellipticity dispersion per component. We take the Gaussian smoothing
kernel with $W_{\theta_{\rm G}}(\bm{\theta})=1/({\pi\theta_{\rm
G}^{2}})\exp{(-{|\bm{\theta}|^{2}}/{\theta_{\rm G}^{2}})}$ and $\theta_{\rm
G}=1.5$ arcmin. (2) From $\langle\bm{\epsilon}\rangle(\bm{\theta})$, we
reconstruct the convergence field $K_{\rm GRID}$ sampled on $1024\times 1024$
pixels for each of the 52 fields. (3) We generate smoothed filling-factor maps
$f$ from the galaxy spatial distributions of the shear sample, and exclude
regions with $f<0.6$ [30]. The outermost $5\theta_{\mathrm{G}}$ pixels in each
of the four sides of a map are also removed to reduce the boundary effects.
(4) We rotate each of the galaxies randomly, and reconstruct the noise part
$N_{\mathrm{GRID}}$ of the convergence fields following steps (1) and (2). In
total, 30 sets of $N_{\mathrm{GRID}}$ are generated for each field.

For the high-redshift galaxies from PDR2, we apply the same mask and boundary
exclusion criteria as those of the foreground convergence fields. The
remaining galaxies make up the background subsamples $G_{\mathrm{L}}$ with
different $m_{lim}$. We also generate 30 sets of random samples
$G_{\mathrm{R}}$ from the galaxies in each field by randomly populating them
spatially.

Figure 2:  Results from the simulated mocks with $\tilde{\omega}_{ij}$ and
$\tilde{\xi}_{+ij}$ shown in blue and red, respectively. The black solid and
dashed lines are the corresponding theoretical predictions. The bottom right
panel shows $\tilde{\xi}_{+ij}$ vs.
$\tilde{\omega}_{ij}/\\{2[\alpha(m_{lim})-1]\\}$ with the theoretical
expectation of 1:1 shown as the solid line.

An estimator for the shear-number density cross-correlation is then given by

$\displaystyle\tilde{\omega}_{ij}(\theta)$ $\displaystyle=(\langle
K_{\mathrm{GRID}}G_{\mathrm{L}}\rangle-\langle
K_{\mathrm{GRID}}G_{\mathrm{R}}\rangle)$ (10) $\displaystyle-(\langle
N_{\mathrm{GRID}}G_{\mathrm{L}}\rangle-\langle
N_{\mathrm{GRID}}G_{\mathrm{R}}\rangle).$

The calculations are done using
Athena111http://www.cosmostat.org/software/athena [32]. Considering the size
of each selected field, a total of 9 bins with logarithmic bin-width of
$\Delta\mathrm{log_{10}}\theta\sim 0.173$ are chosen with the central $\theta$
ranging from 1.21 to 29.65 arcmin. Note that in model calculations, the kernel
$W_{G}$ applied to the foreground needs to be included in Eqn.(4) and also in
Eqn.(7).

To estimate the shear-shear correlation between low- and high-redshift bins,
we construct low-redshift shear fields $\Gamma_{\mathrm{GRID}}$ on pixels
directly from the smoothed shear fields $\langle\bm{\epsilon}\rangle$. For
high redshift part, we build shear subsamples $\bm{\gamma_{j}}$ with
$m_{lim}=22.5,23.0$, and $23.5$ from S16A. The cosmic shear two-point
correlations are then calculated over all preserved pixel-galaxy pairs ($ab$)
within some bin around $\theta$ using
$\tilde{\xi}_{+ij}(\theta)=\langle\Gamma_{t\mathrm{GRID}}\gamma_{tj}+\Gamma_{\times\mathrm{GRID}}\gamma_{\times
j}\rangle=\sum_{ab}{w_{b}[\Gamma_{1\mathrm{GRID}}(\bm{\theta_{a}})\gamma_{1j}(\bm{\theta_{b}})+\Gamma_{2\mathrm{GRID}}(\bm{\theta_{a}})\gamma_{2j}(\bm{\theta_{b}})]}/\sum_{ab}w_{b}$,
where
$\bm{\gamma_{j}}(\bm{\theta_{b}})=\frac{1}{(1+\hat{m})}[\frac{\bm{e_{b}}}{2\mathcal{R}}-\bm{c_{b}}]$,
with an ensemble correction of multiplicative bias $\hat{m}$ and reponsivity
factor $\mathcal{R}$ applied following [27].

Figure 3:  Correlation results from HSC-SSP observational data.

##  V HSC mock validation and observational results

Before applying to data, we validate our analyses by building HSC-SSP mocks
from ray-tracing simulations described in Liu  _et al._ [33]. We have 59
lensing-plane outputs between $z=3$ and $z=0$, and each consists of 96 maps of
$3.5\times 3.5\deg^{2}$ each.

With these lensing maps, we create HSC mocks as follows. (1) For high-redshift
PDR2 mocks, the galaxy positions and magnitudes from PDR2 of the 52 fields are
adopted. Because of the independence between the original HSC galaxy
positions/magnitudes and our simulated lensing signals, we regard them as the
unlensed quantities. The redshift $z_{p}$ is also kept. (2) For shear mocks,
we preserve the position, magnitude, redshift $z_{p}$, and shear related
quantities of each galaxy from S16A. We randomly rotate them to eliminate the
original shear signals in the data. Otherwise the signals embedded in the HSC
data will contaminate the mock simulation results of both shear-number density
and shear-shear correlations. (3) We pad the mock galaxies into our simulated
maps, and calculate the lensing signals at their positions by interpolating
(in redshift and angular position) the grid signals on lensing maps. For high
redshift PDR2 mocks, we obtain the deflection angle and magnification $\mu$
for each galaxy, and adjust its position and magnitude (adding a factor of
2.5$\log_{10}(1/\mu)$) accordingly. For shear mocks, we follow the procedures
of Oguri  _et al._ [31] including the shear bias from S16A to our simulated
lensing signal of each galaxy. (4) We do different paddings of (3), and
generate 20 mocks for each of the 52 fields. (5) We randomly select one from
the 20 for each field and compose one set of HSC mock with 52 fields. In total
200 sets of mock data are generated. (6) The same convergence reconstruction
and correlation estimates are done as for the observational data. (7) The
covariance matrices are estimated from the 200 mocks. The results are about
the same as those calculated from 400 mocks.

In Figure 2, we show the results averaged over the 200 mocks. The blue and red
symbols are for the shear-number density and shear-shear correlations,
respectively. The solid and dashed lines are the corresponding model
predictions with the listed $\alpha(m_{lim})$ values. The last panel shows
$\tilde{\xi}_{+ij}$ vs. $\tilde{\omega}_{ij}/\\{2[\alpha(m_{lim})-1]\\}$ with
the theoretical expectation of 1:1 black line noting that we correct for the
shear bias in calculating the correlations as shown in Eqns.(9) and (10) and
$\tilde{\xi}_{+ij}(\theta)$. The mock results agree well with that of the
model, validating our analyese.

Our mock settings implicitly assume the completeness of the high-redshift PDR2
sample, which is not true for the real data. Ideally, to evaluate the effect
of incompleteness, we should generate mock data from deeper surveys, and do
analyses for the selected incomplete samples. This is infeasible here because
of the small coverage of the HSC UltraDeep survey. We thus assess the impact
by assuming an artificial incompleteness function to select galaxies from our
high-redshift PDR2 mocks. Our results show, consistent with other studies,
that the magnification signals from incomplete samples follow the predictions
with $\alpha$ calculated from the underlying complete sample[34]. For our
specific mocks, with the completeness fraction of $\sim 80\%$ and $\sim 70\%$,
the $\alpha$ value estimated from the incomplete samples is lower by $\sim
4\%$ and $\sim 8\%$, respectively, than that from the complete sample. The
effect of such changes on the shear-number density correlation is well within
the statistical uncertainties here. In follow-up studies, we will investigate
in detail the impact of incompleteness on future high precision analyses.

The results from the observational data are shown in Figure 3 with symbols and
lines the same as those in Figure 2, except that the model is $\Omega_{\rm
m}=0.346$ and $\sigma_{8}=0.749$ [1].

The observed shear-number density correlations follow the expected trend of
varying $\alpha$, and are consistent with the theoretical predictions. With 27
non-independent data points and the full covariance, $\chi_{T}^{2}=24.0$ and
$\chi_{0}^{2}=34.1$ with respect to the model and to the null, respectively.
The corresponding p-value of $p(\geq\chi_{T}^{2})$ is $0.63$, four times
higher than $p(\geq\chi_{0}^{2})=0.16$. Assuming Gaussian likelihood and equal
probability of the considered cosmological model and the null, the Bayes
factor is
$\log_{10}B_{T0}=\log_{10}\\{\exp[-0.5(\chi^{2}_{T}-\chi^{2}_{0})]\\}\approx
2.2$, showing a solid detection of the signals [35].

##  VI Cosmological potentials

With the shear multiplicative bias $m$ corrected according to that given in
S16A, we see from Figure 3 that
$\tilde{\omega}_{ij}/\\{2[\alpha(m_{lim})-1]\\}$ and $\tilde{\xi}_{+ij}$
follow each other well showing the consistency of the HSC shear and
magnification signals, and the residual shear bias after correction is
insignificant within the error bars.

Without the bias information (or it is incorrect), the ratio of the two
correlations is given in Eqn.(8) containing the factor of $(1+\bar{m}_{j})$.
If $\alpha(m_{lim})$ can be well determined, the combination of the two can be
used to calibrate $\bar{m}_{j}$ using observational data themselves,
independent of cosmology. To demonstrate the idea, we perform a forecast of
cosmological constraints by combining the shear-number density and shear-shear
correlations in comparison with that using the shear-shear correlation alone.

To avoid the instability problem, we do not directly calculate the ratio.
Instead, we perform analyses by combining the data vectors of
$\tilde{\omega}_{ij}$ and $\tilde{\xi}_{+ij}$. Specifically, we consider the
same two redshift bins as we did for the HSC analyses, but scale the effective
survey area from $\sim 60\deg^{2}$ to $15000\deg^{2}$. For the high redshift
bin, we use the subsample with $m_{lim}=23$. How to combine different samples
with different $m_{lim}$ to enhance the constraining power will be our future
tasks. The fiducial values of the correlation signals are taken from the mock
results, and the covariance matrices are scaled by the considered area. The
number densities of different galaxy samples are assumed to be the same as
those of HSC analyses.

We consider three parameters $(\Omega_{\rm m},\sigma_{8},\bar{m})$ assuming
the same $\bar{m}$ for low- and high-redshift shear samples. The value of
$\alpha(m_{lim})=1.96$ from our mocks, and its uncertainty is estimated from
the HSC data by bootstrap different fields and then scaled to 15000
$\deg^{2}$, which gives $\sigma_{\alpha}\approx 0.002$.

The Fisher forecast results are shown in Figure 4, where the blue and red
parts are for the case with $\tilde{\xi}_{+ij}$ only and that of combining
$\tilde{\xi}_{+ij}$ and $\tilde{\omega}_{ij}$, respectively. By including
$\tilde{\omega}_{ij}$, the constraint on $\bar{m}$ do improve significantly,
and the degeneracies with $\Omega_{\rm m}$ and $\sigma_{8}$ are largely
eliminated. Consequently, the constraints on the cosmological parameters are
tightened, as presented in Table 1. With SS and SP for shear-shear and shear-
number density correlations and $S_{8}=\sigma_{8}(\Omega_{\rm m}/0.3)^{0.5}$,
the improvement on $\bar{m}$ and $S_{8}$ is $2.3$ and $2.1$ times,
respectively.

The combination of the two correlations can also potentially reveal scale-
dependent shear measurement bias or other systematics because then the ratio
of the two should have a scale dependence. In the analyses here, because of
the relatively large error bars, such a dependence is not seen clearly for HSC
data. We will explore this potential in our follow-up studies with respect to
future observations.

Figure 4:  Fisher forecast for the constraints on ($\Omega_{\mathrm{m}}$, $\sigma_{8}$, $\bar{m}$) with the survey area of $15000\deg^{2}$ and $m_{lim}=23.0$ for the high-redshift sample. Blue and red are, respectively, the results from $\tilde{\xi}_{+ij}$ only and that of combining $\tilde{\xi}_{+ij}$ and $\tilde{\omega}_{ij}$. Table 1: Forecast constraints for a $15000\deg^{2}$ survey | SS only | SS & SP
---|---|---
$\Omega_{\mathrm{m}}$ ($68\%$CL) | 0.032 | 0.027
$\sigma_{8}$ ($68\%$CL) | 0.059 | 0.051
$\bar{m}$ ($68\%$CL) | 0.075 | 0.032
$S_{8}$ ($68\%$CL) | 0.043 | 0.020

##  VII Summary and Discussion

In this study, we propose the shear-number density correlation analyses and
apply to HSC data. The cosmic magnification signals are clearly detected with
no galaxy bias involved. Combining with the shear-shear correlation allows us
to check the data consistency internally. Our results show that the cosmic
shear and magnification signals from HSC are consistent with each other within
error bars. We further explore the potential of including shear-number density
correlations in cosmological studies, revealing significant improvements of
the constraints on $\bar{m}$ and $S_{8}$. Thus for future large weak lensing
surveys, we advocate to include this information in addition to the shear-
shear, galaxy clustering and galaxy-galaxy lensing analyses to enhance the
cosmological gains. For that, this paper provides a proof of concept. We will
carry out more detailed studies about different effects, such as sample
incompleteness, dust effect, photo-z errors and intrinsic alignments. We will
also investigate analyses methods to increase the signal-to-noise ratio of
shear-number density correlations and how to inlcude them in multi-bin
tomographic studies.

###### Acknowledgements.

The Hyper Suprime-Cam (HSC) collaboration includes the astronomical
communities of Japan and Taiwan, and Princeton University. The HSC
instrumentation and software were developed by the National Astronomical
Observatory of Japan (NAOJ), the Kavli Institute for the Physics and
Mathematics of the Universe (Kavli IPMU), the University of Tokyo, the High
Energy Accelerator Research Organization (KEK), the Academia Sinica Institute
for Astronomy and Astrophysics in Taiwan (ASIAA), and Princeton University.
Funding was contributed by the FIRST program from Japanese Cabinet Office, the
Ministry of Education, Culture, Sports, Science and Technology (MEXT), the
Japan Society for the Promotion of Science (JSPS), Japan Science and
Technology Agency (JST), the Toray Science Foundation, NAOJ, Kavli IPMU, KEK,
ASIAA, and Princeton University. This paper makes use of software developed
for the Large Synoptic Survey Telescope. We thank the LSST Project for making
their code available as free software at http://dm.lsst.org. The Pan-STARRS1
Surveys (PS1) have been made possible through contributions of the Institute
for Astronomy, the University of Hawaii, the Pan-STARRS Project Office, the
Max-Planck Society and its participating institutes, the Max Planck Institute
for Astronomy, Heidelberg and the Max Planck Institute for Extraterrestrial
Physics, Garching, The Johns Hopkins University, Durham University, the
University of Edinburgh, Queen’s University Belfast, the Harvard-Smithsonian
Center for Astrophysics, the Las Cumbres Observatory Global Telescope Network
Incorporated, the National Central University of Taiwan, the Space Telescope
Science Institute, the National Aeronautics and Space Administration under
Grant No. NNX08AR22G issued through the Planetary Science Division of the NASA
Science Mission Directorate, the National Science Foundation under Grant No.
AST-1238877, the University of Maryland, and Eotvos Lorand University (ELTE)
and the Los Alamos National Laboratory. The analyses of this paper are based
on data collected at the Subaru Telescope and retrieved from the HSC data
archive system, which is operated by Subaru Telescope and Astronomy Data
Center at National Astronomical Observatory of Japan. We thank the stimulating
discussions with Pengjie Zhang, Jun Zhang and Martin Kilbinger. The
calculations of this study are partly done on the Yunnan University Astronomy
Supercomputer. X.K.L. acknowledges the supports from NSFC of China under grant
11803028 and YNU grant C176220100008. Z.H.F. is supported by NSFC of China
under grants 11933002, U1931210 and 11653001. Z.H.F. and X.K.L. are also
supported by a grant of CAS Interdisciplinary Innovation Team. D.Z.L.
acknowledges the support from the China Postdoctoral Science Foundation under
the grant 2019M663582. C.L.W. acknowledges the support from NSFC of China
under grant 11903082. G.L.L. is supported by NSFC of China under grants
11673065, U1931210, and 11273061. LPF acknowledges the support from NSFC
grants 11722326, 11673018 and 11933002, STCSM grant 18590780100 and 188014066,
and the Dawn Program 19SG41 and the Innovation Program 2019-01-07-00-02-E00032
of SMEC. T.F. is supported by Grant-in-Aids for Scientific Research from JSPS
(No.20 K03937), and also acknowledges the SWIFAR visiting fellow program. We
also acknowledge the support from ISSI/ISSI-BJ International Team Program -
Weak gravitational lensing studies from space missions.

## References

  * Hamana  _et al._ [2020] T. Hamana, M. Shirasaki, S. Miyazaki, C. Hikage, M. Oguri, S. More, R. Armstrong, A. Leauthaud, R. Mandelbaum, H. Miyatake, A. J. Nishizawa, M. Simet, M. Takada, H. Aihara, J. Bosch, Y. Komiyama, R. Lupton, H. Murayama, M. A. Strauss, and M. Tanaka, Cosmological constraints from cosmic shear two-point correlation functions with HSC survey first-year data, PASJ 72, 16 (2020), arXiv:1906.06041 [astro-ph.CO]
  * Bartelmann and Schneider [2001] M. Bartelmann and P. Schneider, Weak gravitational lensing, Phys. Rep. 340, 291 (2001), arXiv:astro-ph/9912508 [astro-ph]
  * Hoekstra and Jain [2008] H. Hoekstra and B. Jain, Weak Gravitational Lensing and Its Cosmological Applications, Annual Review of Nuclear and Particle Science 58, 99 (2008), arXiv:0805.0139 [astro-ph]
  * Van Waerbeke  _et al._ [2010] L. Van Waerbeke, H. Hildebrandt, J. Ford, and M. Milkeraitis, Magnification as a Probe of Dark Matter Halos at High Redshifts, ApJ 723, L13 (2010), arXiv:1004.3793 [astro-ph.CO]
  * Fu and Fan [2014] L.-P. Fu and Z.-H. Fan, Probing the dark side of the Universe with weak gravitational lensing effects, Research in Astronomy and Astrophysics 14, 1061-1120 (2014)
  * Kilbinger [2015] M. Kilbinger, Cosmology with cosmic shear observations: a review, Reports on Progress in Physics 78, 086901 (2015), arXiv:1411.0115 [astro-ph.CO]
  * Dark Energy Survey Collaboration  _et al._ [2016] Dark Energy Survey Collaboration, T. Abbott, F. B. Abdalla, J. Aleksić, S. Allam, A. Amara, D. Bacon, E. Balbinot, M. Banerji, K. Bechtol, A. Benoit-Lévy, G. M. Bernstein, E. Bertin, J. Blazek, C. Bonnett, S. Bridle, D. Brooks, R. J. Brunner, E. Buckley-Geer, D. L. Burke, G. B. Caminha, D. Capozzi, J. Carlsen, A. Carnero-Rosell, M. Carollo, M. Carrasco-Kind, J. Carretero, F. J. Castander, L. Clerkin, T. Collett, C. Conselice, M. Crocce, C. E. Cunha, C. B. D’Andrea, L. N. da Costa, T. M. Davis, S. Desai, H. T. Diehl, J. P. Dietrich, S. Dodelson, P. Doel, A. Drlica-Wagner, J. Estrada, J. Etherington, A. E. Evrard, J. Fabbri, D. A. Finley, B. Flaugher, R. J. Foley, P. Fosalba, J. Frieman, J. García-Bellido, E. Gaztanaga, D. W. Gerdes, T. Giannantonio, D. A. Goldstein, D. Gruen, R. A. Gruendl, P. Guarnieri, G. Gutierrez, W. Hartley, K. Honscheid, B. Jain, D. J. James, T. Jeltema, S. Jouvel, R. Kessler, A. King, D. Kirk, R. Kron, K. Kuehn, N. Kuropatkin, O. Lahav, T. S. Li, M. Lima, H. Lin, M. A. G. Maia, M. Makler, M. Manera, C. Maraston, J. L. Marshall, P. Martini, R. G. McMahon, P. Melchior, A. Merson, C. J. Miller, R. Miquel, J. J. Mohr, X. Morice-Atkinson, K. Naidoo, E. Neilsen, R. C. Nichol, B. Nord, R. Ogando, F. Ostrovski, A. Palmese, A. Papadopoulos, H. V. Peiris, J. Peoples, W. J. Percival, A. A. Plazas, S. L. Reed, A. Refregier, A. K. Romer, A. Roodman, A. Ross, E. Rozo, E. S. Rykoff, I. Sadeh, M. Sako, C. Sánchez, E. Sanchez, B. Santiago, V. Scarpine, M. Schubnell, I. Sevilla-Noarbe, E. Sheldon, M. Smith, R. C. Smith, M. Soares-Santos, F. Sobreira, M. Soumagnac, E. Suchyta, M. Sullivan, M. Swanson, G. Tarle, J. Thaler, D. Thomas, R. C. Thomas, D. Tucker, J. D. Vieira, V. Vikram, A. R. Walker, R. H. Wechsler, J. Weller, W. Wester, L. Whiteway, H. Wilcox, B. Yanny, Y. Zhang, and J. Zuntz, The Dark Energy Survey: more than dark energy - an overview, MNRAS 460, 1270 (2016), arXiv:1601.00329 [astro-ph.CO]
  * de Jong  _et al._ [2013] J. T. A. de Jong, G. A. Verdoes Kleijn, K. H. Kuijken, and E. A. Valentijn, The Kilo-Degree Survey, Experimental Astronomy 35, 25 (2013), arXiv:1206.1254 [astro-ph.CO]
  * Aihara  _et al._ [2018a] H. Aihara, N. Arimoto, R. Armstrong, S. Arnouts, N. A. Bahcall, S. Bickerton, J. Bosch, K. Bundy, P. L. Capak, J. H. H. Chan, M. Chiba, J. Coupon, E. Egami, M. Enoki, F. Finet, H. Fujimori, S. Fujimoto, H. Furusawa, J. Furusawa, T. Goto, A. Goulding, J. P. Greco, J. E. Greene, J. E. Gunn, T. Hamana, Y. Harikane, Y. Hashimoto, T. Hattori, M. Hayashi, Y. Hayashi, K. G. Hełminiak, R. Higuchi, C. Hikage, P. T. P. Ho, B.-C. Hsieh, K. Huang, S. Huang, H. Ikeda, M. Imanishi, A. K. Inoue, K. Iwasawa, I. Iwata, A. T. Jaelani, H.-Y. Jian, Y. Kamata, H. Karoji, N. Kashikawa, N. Katayama, S. Kawanomoto, I. Kayo, J. Koda, M. Koike, T. Kojima, Y. Komiyama, A. Konno, S. Koshida, Y. Koyama, H. Kusakabe, A. Leauthaud, C.-H. Lee, L. Lin, Y.-T. Lin, R. H. Lupton, R. Mandelbaum, Y. Matsuoka, E. Medezinski, S. Mineo, S. Miyama, H. Miyatake, S. Miyazaki, R. Momose, A. More, S. More, Y. Moritani, T. J. Moriya, T. Morokuma, S. Mukae, R. Murata, H. Murayama, T. Nagao, F. Nakata, M. Niida, H. Niikura, A. J. Nishizawa, Y. Obuchi, M. Oguri, Y. Oishi, N. Okabe, S. Okamoto, Y. Okura, Y. Ono, M. Onodera, M. Onoue, K. Osato, M. Ouchi, P. A. Price, T.-S. Pyo, M. Sako, M. Sawicki, T. Shibuya, K. Shimasaku, A. Shimono, M. Shirasaki, J. D. Silverman, M. Simet, J. Speagle, D. N. Spergel, M. A. Strauss, Y. Sugahara, N. Sugiyama, Y. Suto, S. H. Suyu, N. Suzuki, P. J. Tait, M. Takada, T. Takata, N. Tamura, M. M. Tanaka, M. Tanaka, M. Tanaka, Y. Tanaka, T. Terai, Y. Terashima, Y. Toba, N. Tominaga, J. Toshikawa, E. L. Turner, T. Uchida, H. Uchiyama, K. Umetsu, F. Uraguchi, Y. Urata, T. Usuda, Y. Utsumi, S.-Y. Wang, W.-H. Wang, K. C. Wong, K. Yabe, Y. Yamada, H. Yamanoi, N. Yasuda, S. Yeh, A. Yonehara, and S. Yuma, The Hyper Suprime-Cam SSP Survey: Overview and survey design, PASJ 70, S4 (2018a), arXiv:1704.05858 [astro-ph.IM]
  * Kilbinger  _et al._ [2013] M. Kilbinger, L. Fu, C. Heymans, F. Simpson, J. Benjamin, T. Erben, J. Harnois-Déraps, H. Hoekstra, H. Hildebrandt, T. D. Kitching, Y. Mellier, L. Miller, L. Van Waerbeke, K. Benabed, C. Bonnett, J. Coupon, M. J. Hudson, K. Kuijken, B. Rowe, T. Schrabback, E. Semboloni, S. Vafaei, and M. Velander, CFHTLenS: combined probe cosmological model comparison using 2D weak gravitational lensing, MNRAS 430, 2200 (2013), arXiv:1212.3338 [astro-ph.CO]
  * Hildebrandt  _et al._ [2017] H. Hildebrandt, M. Viola, C. Heymans, S. Joudaki, K. Kuijken, C. Blake, T. Erben, B. Joachimi, D. Klaes, L. Miller, C. B. Morrison, R. Nakajima, G. Verdoes Kleijn, A. Amon, A. Choi, G. Covone, J. T. A. de Jong, A. Dvornik, I. Fenech Conti, A. Grado, J. Harnois-Déraps, R. Herbonnet, H. Hoekstra, F. Köhlinger, J. McFarland, A. Mead, J. Merten, N. Napolitano, J. A. Peacock, M. Radovich, P. Schneider, P. Simon, E. A. Valentijn, J. L. van den Busch, E. van Uitert, and L. Van Waerbeke, KiDS-450: cosmological parameter constraints from tomographic weak gravitational lensing, MNRAS 465, 1454 (2017), arXiv:1606.05338 [astro-ph.CO]
  * Abbott  _et al._ [2018] T. M. C. Abbott, F. B. Abdalla, A. Alarcon, J. Aleksić, S. Allam, S. Allen, A. Amara, J. Annis, J. Asorey, S. Avila, D. Bacon, E. Balbinot, M. Banerji, N. Banik, W. Barkhouse, M. Baumer, E. Baxter, K. Bechtol, M. R. Becker, A. Benoit-Lévy, B. A. Benson, G. M. Bernstein, E. Bertin, J. Blazek, S. L. Bridle, D. Brooks, D. Brout, E. Buckley-Geer, D. L. Burke, M. T. Busha, A. Campos, D. Capozzi, A. Carnero Rosell, M. Carrasco Kind, J. Carretero, F. J. Castander, R. Cawthon, C. Chang, N. Chen, M. Childress, A. Choi, C. Conselice, R. Crittenden, M. Crocce, C. E. Cunha, C. B. D’Andrea, L. N. da Costa, R. Das, T. M. Davis, C. Davis, J. De Vicente, D. L. DePoy, J. DeRose, S. Desai, H. T. Diehl, J. P. Dietrich, S. Dodelson, P. Doel, A. Drlica-Wagner, T. F. Eifler, A. E. Elliott, F. Elsner, J. Elvin-Poole, J. Estrada, A. E. Evrard, Y. Fang, E. Fernandez, A. Ferté, D. A. Finley, B. Flaugher, P. Fosalba, O. Friedrich, J. Frieman, J. García-Bellido, M. Garcia-Fernandez, M. Gatti, E. Gaztanaga, D. W. Gerdes, T. Giannantonio, M. S. S. Gill, K. Glazebrook, D. A. Goldstein, D. Gruen, R. A. Gruendl, J. Gschwend, G. Gutierrez, S. Hamilton, W. G. Hartley, S. R. Hinton, K. Honscheid, B. Hoyle, D. Huterer, B. Jain, D. J. James, M. Jarvis, T. Jeltema, M. D. Johnson, M. W. G. Johnson, T. Kacprzak, S. Kent, A. G. Kim, A. King, D. Kirk, N. Kokron, A. Kovacs, E. Krause, C. Krawiec, A. Kremin, K. Kuehn, S. Kuhlmann, N. Kuropatkin, F. Lacasa, O. Lahav, T. S. Li, A. R. Liddle, C. Lidman, M. Lima, H. Lin, N. MacCrann, M. A. G. Maia, M. Makler, M. Manera, M. March, J. L. Marshall, P. Martini, R. G. McMahon, P. Melchior, F. Menanteau, R. Miquel, V. Miranda, D. Mudd, J. Muir, A. Möller, E. Neilsen, R. C. Nichol, B. Nord, P. Nugent, R. L. C. Ogando, A. Palmese, J. Peacock, H. V. Peiris, J. Peoples, W. J. Percival, D. Petravick, A. A. Plazas, A. Porredon, J. Prat, A. Pujol, M. M. Rau, A. Refregier, P. M. Ricker, N. Roe, R. P. Rollins, A. K. Romer, A. Roodman, R. Rosenfeld, A. J. Ross, E. Rozo, E. S. Rykoff, M. Sako, A. I. Salvador, S. Samuroff, C. Sánchez, E. Sanchez, B. Santiago, V. Scarpine, R. Schindler, D. Scolnic, L. F. Secco, S. Serrano, I. Sevilla-Noarbe, E. Sheldon, R. C. Smith, M. Smith, J. Smith, M. Soares-Santos, F. Sobreira, E. Suchyta, G. Tarle, D. Thomas, M. A. Troxel, D. L. Tucker, B. E. Tucker, S. A. Uddin, T. N. Varga, P. Vielzeuf, V. Vikram, A. K. Vivas, A. R. Walker, M. Wang, R. H. Wechsler, J. Weller, W. Wester, R. C. Wolf, B. Yanny, F. Yuan, A. Zenteno, B. Zhang, Y. Zhang, J. Zuntz, and Dark Energy Survey Collaboration, Dark Energy Survey year 1 results: Cosmological constraints from galaxy clustering and weak lensing, Phys. Rev. D 98, 043526 (2018), arXiv:1708.01530 [astro-ph.CO]
  * Blandford and Narayan [1992] R. D. Blandford and R. Narayan, Cosmological applications of gravitational lensing., ARA&A 30, 311 (1992)
  * Bartelmann and Narayan [1995] M. Bartelmann and R. Narayan, Gravitational lensing and the mass distribution of clusters, in  _Dark Matter_, American Institute of Physics Conference Series, Vol. 336, edited by S. S. Holt and C. L. Bennett (1995) pp. 307–319, arXiv:astro-ph/9411033 [astro-ph]
  * Menard and Bartelmann [2002] B. Menard and M. Bartelmann, Cosmological information from quasar-galaxy correlations induced by weak lensing, A&A 386, 784 (2002), astro-ph:0203163 [astro-ph.CO]
  * Menard  _et al._ [2003] B. Menard, M. Bartelmann, and Y. Mellier, Measuring Omega0 with higher-order quasar-galaxy correlations induced by weak lensing, A&A 409, 411 (2003)
  * Menard  _et al._ [2010] B. Menard, R. Scranton, M. Fukugita, and G. Richards, Measuring the galaxy-mass and galaxy-dust correlations through magnification and reddeing, MNRAS 405, 1025 (2010), arXiv:0902.4240 [astro-ph.CO]
  * van Waerbeke [2010] L. van Waerbeke, Shear and magnification: cosmic complementarity, MNRAS 401, 2093 (2010), arXiv:0906.1583 [astro-ph.CO]
  * Yang  _et al._ [2017] X. Yang, J. Zhang, Y. Yu, and P. Zhang, Weak-lensing Power Spectrum Reconstruction by Counting Galaxies. I. The ABS Method, ApJ 845, 174 (2017), arXiv:1703.01575 [astro-ph.CO]
  * Zhang  _et al._ [2018] P. Zhang, X. Yang, J. Zhang, and Y. Yu, Weak-lensing Power Spectrum Reconstruction by Counting Galaxies. II. Improving the ABS Method with the Shift Parameter, ApJ 864, 10 (2018), arXiv:1807.00443 [astro-ph.CO]
  * Morrison  _et al._ [2012] C. B. Morrison, R. Scranton, B. Ménard, S. J. Schmidt, J. A. Tyson, R. Ryan, A. Choi, and D. M. Wittman, Tomographic magnification of Lyman-break galaxies in the Deep Lens Survey, MNRAS 426, 2489 (2012), arXiv:1204.2830 [astro-ph.CO]
  * Ford  _et al._ [2014] J. Ford, H. Hildebrandt, L. Van Waerbeke, T. Erben, C. Laigle, M. Milkeraitis, and C. B. Morrison, Cluster magnification and the mass-richness relation in CFHTLenS, MNRAS 439, 3755 (2014), arXiv:1310.2295 [astro-ph.CO]
  * Garcia-Fernandez  _et al._ [2018] M. Garcia-Fernandez, E. Sanchez, I. Sevilla-Noarbe, E. Suchyta, E. M. Huff, E. Gaztanaga, Aleksić, J. , R. Ponce, F. J. Castander, B. Hoyle, T. M. C. Abbott, F. B. Abdalla, S. Allam, J. Annis, A. Benoit-Lévy, G. M. Bernstein, E. Bertin, D. Brooks, E. Buckley-Geer, D. L. Burke, A. Carnero Rosell, M. Carrasco Kind, J. Carretero, M. Crocce, C. E. Cunha, C. B. D’Andrea, L. N. da Costa, D. L. DePoy, S. Desai, H. T. Diehl, T. F. Eifler, A. E. Evrard, E. Fernandez, B. Flaugher, P. Fosalba, J. Frieman, J. García-Bellido, D. W. Gerdes, T. Giannantonio, D. Gruen, R. A. Gruendl, J. Gschwend, G. Gutierrez, D. J. James, M. Jarvis, D. Kirk, E. Krause, K. Kuehn, N. Kuropatkin, O. Lahav, M. Lima, N. MacCrann, M. A. G. Maia, M. March, J. L. Marshall, P. Melchior, R. Miquel, J. J. Mohr, A. A. Plazas, A. K. Romer, A. Roodman, E. S. Rykoff, V. Scarpine, M. Schubnell, R. C. Smith, M. Soares-Santos, F. Sobreira, G. Tarle, D. Thomas, A. R. Walker, W. Wester, and DES Collaboration, Weak lensing magnification in the Dark Energy Survey Science Verification data, MNRAS 476, 1071 (2018)
  * Umetsu  _et al._ [2014] K. Umetsu, E. Medezinski, M. Nonino, J. Merten, M. Postman, M. Meneghetti, M. Donahue, N. Czakon, A. Molino, S. Seitz, D. Gruen, D. Lemze, I. Balestra, N. Benítez, A. Biviano, T. Broadhurst, H. Ford, C. Grillo, A. Koekemoer, P. Melchior, A. Mercurio, J. Moustakas, P. Rosati, and A. Zitrin, CLASH: Weak-lensing Shear-and-magnification Analysis of 20 Galaxy Clusters, ApJ 795, 163 (2014), arXiv:1404.1375 [astro-ph.CO]
  * Aihara  _et al._ [2018b] H. Aihara, R. Armstrong, S. Bickerton, J. Bosch, J. Coupon, H. Furusawa, Y. Hayashi, H. Ikeda, Y. Kamata, H. Karoji, S. Kawanomoto, M. Koike, Y. Komiyama, D. Lang, R. H. Lupton, S. Mineo, H. Miyatake, S. Miyazaki, T. Morokuma, Y. Obuchi, Y. Oishi, Y. Okura, P. A. Price, T. Takata, M. M. Tanaka, M. Tanaka, Y. Tanaka, T. Uchida, F. Uraguchi, Y. Utsumi, S.-Y. Wang, Y. Yamada, H. Yamanoi, N. Yasuda, N. Arimoto, M. Chiba, F. Finet, H. Fujimori, S. Fujimoto, J. Furusawa, T. Goto, A. Goulding, J. E. Gunn, Y. Harikane, T. Hattori, M. Hayashi, K. G. Hełminiak, R. Higuchi, C. Hikage, P. T. P. Ho, B.-C. Hsieh, K. Huang, S. Huang, M. Imanishi, I. Iwata, A. T. Jaelani, H.-Y. Jian, N. Kashikawa, N. Katayama, T. Kojima, A. Konno, S. Koshida, H. Kusakabe, A. Leauthaud, C.-H. Lee, L. Lin, Y.-T. Lin, R. Mandelbaum, Y. Matsuoka, E. Medezinski, S. Miyama, R. Momose, A. More, S. More, S. Mukae, R. Murata, H. Murayama, T. Nagao, F. Nakata, M. Niida, H. Niikura, A. J. Nishizawa, M. Oguri, N. Okabe, Y. Ono, M. Onodera, M. Onoue, M. Ouchi, T.-S. Pyo, T. Shibuya, K. Shimasaku, M. Simet, J. Speagle, D. N. Spergel, M. A. Strauss, Y. Sugahara, N. Sugiyama, Y. Suto, N. Suzuki, P. J. Tait, M. Takada, T. Terai, Y. Toba, E. L. Turner, H. Uchiyama, K. Umetsu, Y. Urata, T. Usuda, S. Yeh, and S. Yuma, First data release of the Hyper Suprime-Cam Subaru Strategic Program, PASJ 70, S8 (2018b), arXiv:1702.08449 [astro-ph.IM]
  * Aihara  _et al._ [2019] H. Aihara, Y. AlSayyad, M. Ando, R. Armstrong, J. Bosch, E. Egami, H. Furusawa, J. Furusawa, A. Goulding, Y. Harikane, C. Hikage, P. T. P. Ho, B.-C. Hsieh, S. Huang, H. Ikeda, M. Imanishi, K. Ito, I. Iwata, A. T. Jaelani, R. Kakuma, K. Kawana, S. Kikuta, U. Kobayashi, M. Koike, Y. Komiyama, X. Li, Y. Liang, Y.-T. Lin, W. Luo, R. Lupton, N. B. Lust, L. A. MacArthur, Y. Matsuoka, S. Mineo, H. Miyatake, S. Miyazaki, S. More, R. Murata, S. V. Namiki, A. J. Nishizawa, M. Oguri, N. Okabe, S. Okamoto, Y. Okura, Y. Ono, M. Onodera, M. Onoue, K. Osato, M. Ouchi, T. Shibuya, M. A. Strauss, N. Sugiyama, Y. Suto, M. Takada, Y. Takagi, T. Takata, S. Takita, M. Tanaka, T. Terai, Y. Toba, H. Uchiyama, Y. Utsumi, S.-Y. Wang, W. Wang, and Y. Yamada, Second data release of the Hyper Suprime-Cam Subaru Strategic Program, PASJ 71, 114 (2019), arXiv:1905.12221 [astro-ph.IM]
  * Mandelbaum  _et al._ [2018] R. Mandelbaum, H. Miyatake, T. Hamana, M. Oguri, M. Simet, R. Armstrong, J. Bosch, R. Murata, F. Lanusse, A. Leauthaud, J. Coupon, S. More, M. Takada, S. Miyazaki, J. S. Speagle, M. Shirasaki, C. Sifón, S. Huang, A. J. Nishizawa, E. Medezinski, Y. Okura, N. Okabe, N. Czakon, R. Takahashi, W. R. Coulton, C. Hikage, Y. Komiyama, R. H. Lupton, M. A. Strauss, M. Tanaka, and Y. Utsumi, The first-year shear catalog of the Subaru Hyper Suprime-Cam Subaru Strategic Program Survey, PASJ 70, S25 (2018), arXiv:1705.06745 [astro-ph.CO]
  * Tanaka  _et al._ [2018] M. Tanaka, J. Coupon, B.-C. Hsieh, S. Mineo, A. J. Nishizawa, J. Speagle, H. Furusawa, S. Miyazaki, and H. Murayama, Photometric redshifts for Hyper Suprime-Cam Subaru Strategic Program Data Release 1, PASJ 70, S9 (2018), arXiv:1704.05988 [astro-ph.GA]
  * Nishizawa  _et al._ [2020] A. J. Nishizawa, B.-C. Hsieh, M. Tanaka, and T. Takata, Photometric Redshifts for the Hyper Suprime-Cam Subaru Strategic Program Data Release 2, arXiv e-prints , arXiv:2003.01511 (2020), arXiv:2003.01511 [astro-ph.GA]
  * Liu  _et al._ [2015] X. Liu, C. Pan, R. Li, H. Shan, Q. Wang, L. Fu, Z. Fan, J.-P. Kneib, A. Leauthaud, L. Van Waerbeke, M. Makler, B. Moraes, T. Erben, and A. Charbonnier, Cosmological constraints from weak lensing peak statistics with Canada-France-Hawaii Telescope Stripe 82 Survey, MNRAS 450, 2888 (2015), arXiv:1412.3683 [astro-ph.CO]
  * Oguri  _et al._ [2018] M. Oguri, S. Miyazaki, C. Hikage, R. Mandelbaum, Y. Utsumi, H. Miyatake, M. Takada, R. Armstrong, J. Bosch, Y. Komiyama, A. Leauthaud, S. More, A. J. Nishizawa, N. Okabe, and M. Tanaka, Two- and three-dimensional wide-field weak lensing mass maps from the Hyper Suprime-Cam Subaru Strategic Program S16A data, PASJ 70, S26 (2018), arXiv:1705.06792 [astro-ph.CO]
  * Schneider  _et al._ [2002] P. Schneider, L. van Waerbeke, M. Kilbinger, and Y. Mellier, Analysis of two-point statistics of cosmic shear. I. Estimators and covariances, A&A 396, 1 (2002), arXiv:astro-ph/0206182 [astro-ph]
  * Liu  _et al._ [2014] X. Liu, Q. Wang, C. Pan, and Z. Fan, Mask Effects on Cosmological Studies with Weak-lensing Peak Statistics, ApJ 784, 31 (2014), arXiv:1304.2873 [astro-ph.CO]
  * Hildebrandt [2016] H. Hildebrandt, Observational biases in flux magnification measurements, MNRAS 455, 3943 (2016), arXiv:1511.01352 [astro-ph.GA]
  * Raftery [1995] A. E. Raftery, Bayesian model selection in social research (with discussion), Sociological Methodology 25, 111 (1995)


# Adversarial Seeded Sequence Growing for Weakly-Supervised Temporal Action
Localization

Chengwei Zhang1+,Yunlu Xu2,Zhanzhan Cheng23∗,Yi Niu2,Shiliang Pu2,Fei
Wu3,Futai Zou1 1Shanghai Jiao Tong University, Shanghai, China
<EMAIL_ADDRESS>2Hikvision Research Institute, China
<EMAIL_ADDRESS>3Zhejiang University,
Hangzhou, China<EMAIL_ADDRESS>
(2019)

###### Abstract.

Temporal action localization is an important yet challenging research topic
due to its various applications. Since the frame-level or segment-level
annotations of untrimmed videos require amounts of labor expenditure, studies
on the weakly-supervised action detection have been springing up. However,
most of existing frameworks rely on Class Activation Sequence (CAS) to
localize actions by minimizing the video-level classification loss, which
exploits the most discriminative parts of actions but ignores the minor
regions. In this paper, we propose a novel weakly-supervised framework by
adversarial learning of two modules for eliminating such demerits.
Specifically, the first module is designed as a well-designed Seeded Sequence
Growing (SSG) Network for progressively extending seed regions (namely the
highly reliable regions initialized by a CAS-based framework) to their
expected boundaries. The second module is a specific classifier for mining
trivial or incomplete action regions, which is trained on the shared features
after erasing the seeded regions activated by SSG. In this way, a whole
network composed of these two modules can be trained in an adversarial manner.
The goal of the adversary is to mine features that are difficult for the
action classifier. That is, erasion from SSG will force the classifier to
discover minor or even new action regions on the input feature sequence, and
the classifier will drive the seeds to grow, alternately. At last, we could
obtain the action locations and categories from the well-trained SSG and the
classifier. Extensive experiments on two public benchmarks THUMOS’14 and
ActivityNet1.3 demonstrate the impressive performance of our proposed method
compared with the state-of-the-arts.

Temporal Action Localization, Video Understanding, Weak Supervision

+Zhang partially did this work during an internship in Hikvision Research
Institute.

*Corresponding author.

††journalyear: 2019††copyright: acmcopyright††conference: Proceedings of the
27th ACM International Conference on Multimedia; October 21–25, 2019; Nice,
France††booktitle: Proceedings of the 27th ACM International Conference on
Multimedia (MM ’19), October 21–25, 2019, Nice, France††price: 15.00††doi:
10.1145/3343031.3351044††isbn: 978-1-4503-6889-6/19/10††submissionid:
1001††ccs: Understanding multimedia content Media interpretation

##  1\. Introduction

Temporal action localization, also called action detection, is to localize the
temporal locations of actions as well as identify action categories from
untrimmed videos, which is a fundamental and challenging problem in video
understanding. Many existing works (Shou et al., 2016; Zhao et al., 2017; Shou
et al., 2017; Xu et al., 2017; Yang et al., 2018; Chao et al., 2018; Lin et
al., 2018; Alwassel et al., 2018; Buch et al., 2017; Dai et al., 2017; Richard
and Gall, 2016) make efforts to address this problem in a strong-supervised
manner, where these algorithms rely on fully labeled data (_e.g_. actions
annotated with precise starting and ending frames). However, untrimmed videos
are usually very long, so manually annotating action locations usually seems
time-consuming and expensive in real applications.

Figure 1. Seed-Grow Mechanism for Action Localization.

Above issues motivate researchers to weakly-supervised temporal actions
detection only using video-level labels (i.e., action categories). Actually,
weakly-supervised temporal action detection is similar to object instance
detection in image with only image-level annotations (i.e., object
categories). Inspired by Class Activation Map (CAM) (Zhou et al., 2016) used
in object detection, STPN (Nguyen et al., 2018) introduces the one-dimension
extension of the CAM named as Temporal-CAM, also called Class Activation
Sequence (CAS) in recent works (Shou et al., 2018), to locate the temporal
activating regions by conducting the classification task with only video-level
labels.

Following the CAS-based classification framework, (Shou et al., 2018)
attempted to design an Outer-Inner-Contrastive(OIC) loss to find salient
intervals, and (Paul et al., 2018) addressed the co-occurrence problem in
action detection for capturing more discriminative patterns. Recently, work
(Xu et al., 2019) focused on the relation learning among actions via an RNN
architecture and the CAS mechanism. All the above CAS-based works are designed
for weakly-supervised action localization, and achieve good performance,
especially on the evaluation of low IoUs (e.g. IoU=0.1 or 0.2).

However, the CAS-based action detector usually localizes actions in untrimmed
videos at the most discriminative action interval, which often appears in
action response peak and results in the failure on the evaluation of high
localization precision (See Figure 1). That is, such CAS-based actions
detectors tend to fall into two essential issues: 1) Poor performance on the
evaluation of long-duration actions due to the peak response problem caused by
CAS. For example, only few discrete regions of Instance 1 are activated by
CAS, which directly results in the poor results on the long-duration action
detection. 2) Missing of trivial or indiscriminative actions as the case of
Instance 2 missed in CAS results.

Here, we focus our attention on conquering above issues. With extensive
observations, we find that though CAS tends to generate sparse activating
peaks on action regions, these peaks provide important cues for mining salient
parts of actions or indiscriminative actions. Therefore, an intuitive idea is
that to mine more reliable action regions by referring to the estimated action
cues, termed as seeds. Inspired by the seed-grow mechanism in image
segmentation tasks (Kolesnikov and Lampert, 2016), we adapt it into the
temporal action localization tasks. Differently, we design the following two
complementary manners of grow in the seeded sequences.

  * •

We treat these activated peaks as seeds indicating important action cues and
then extend the time durations for separated seeds to their boundaries,
denoted as the first manner of grow.

  * •

Simultaneously, we erase the activated peaks from shared feature regions and
further conduct a self-adaptive classifier for mining potential trivial or
indiscriminative actions, denoted as the second manner of grow.

Above two procedures should be trained in an adversarial manner. On the one
hand, erasing seeded regions of SSG will force the classifier to mine the less
discriminative action regions from the feature regions. On the other hand, the
classifier will also drive the seeds to grow, alternately.

In this paper, we propose a new weakly-supervised action detection framework
called Adversarial Seeded Sequence Growing (ASSG) by adversarial learning of a
Seed Sequence Growing (SSG) network and a self-adaptive action classification
network. Specifically, the SSG is responsible for learning independent
temporal heatmaps (corresponding to the action occurring probability
distribution) for each action category respectively. The module takes in the
most discriminative regions from CAS as the initial seeds, and progressively
expands the seeded regions to neighborhood in a self-guided way. The action
classifier devotes to exploiting the trivial missing or incomplete instances.
It first erases the high-confidence regions from the SSG and thus has to find
new reliable parts by the supervision of video-level class annotations. It
worth noting that the module adjusts the training parameters of the shared
feature maps with the SSG without any additional learning parameters, so the
classifier can further promote the expanding (namely growing) of seeded
regions. Consequently, these two module are trained in an adversarial manner
and jointly contributes to the iteratively growing of reliable regions. Then
the final results, i.e., the action locations and their categories, can be
obtained from the well-trained SSG and the classifier.

The main contributions of our paper are summarized as follows:

(1) We propose an end-to-end weakly-supervised action detection approach
integrating SSG network and a specific video-level classifier for mining
indiscriminative action locations. To the best of our knowledge, this is the
first work to introduce the seed-grow mechanism in temporal action detection.

(2) We train two modules in an adversarial manner, which not only helps grow
action occurring durations and also mines trivial or indiscriminative actions.

(3) Extensive experiments demonstrate that our method achieves impressive
performance on the challenging THUMOS’14 (Jiang et al., 2014) and
ActivityNet1.3 (Heilbron et al., 2015) datasets, especially on the evaluation
of high IoUs which is more valuable than that in low IoUs.

##  2\. Related Work

###  2.1. Temporal Action Localization.

Temporal action localization aims at identifying the temporal action
intervals. According to the utilized supervision information in model
training, we divide existing methods into two categories: the fully-supervised
based and the weakly-supervised based.

Fully-Supervised Action Detection. Most existing works generally train action
detection model with frame-wisely action annotations, i.e., each action is
annotated with category as well as its starting and ending position. In early
time, sliding windows strategy (Oneata et al., 2014) with a well-trained
action classifier is the traditional solution for temporal detection.
Afterwards, the proposal-based (Shou et al., 2016; Zhao et al., 2017; Xu et
al., 2017) methods were developed for effectively narrowing down the number of
candidate instances. Specifically, S-CNN (Shou et al., 2016) used a multi-
stage CNN for temporal action localization with extracted robust video feature
representation. SSN (Zhao et al., 2017) applied a watershed temporal
actionness grouping algorithm (TAG) (Heilbron et al., 2015) for generating
action regions and then designed the structured temporal pyramid classifiers
for identifying actions’ categories and their localization. Inspired by object
detection, SSAD (Lin et al., 2017), R-C3D (Xu et al., 2017) and TAL-Net (Chao
et al., 2018) were proposed to detect one-dimensional actions by generalizing
2D spatial proposal mechanism to 1D temporal proposal form. Recently, a
special boundary sensitive network (BSN) (Lin et al., 2018) attempted to
locate temporal boundaries and further integrated them into action proposals.
In addition, some frame-level segmentation networks (Shou et al., 2017; Yang
et al., 2018) were also developed to generate more precise action
localizations by conducting the one-dimensional semantic segmentations task.
However, all above works rely on frame-wisely action annotations, which are
usually impractical in real applications due to the amounts of labor
expenditure.

Weakly-Supervised Action Detection. Recently, action detection with only
video-level labels has been studied. UntrimmedNet (Wang et al., 2017)
introduced an end-to-end model for learning only single-label action
categories as well as localizations, which is the first action detection
approach without using frame-wise labels. STPN(Nguyen et al., 2018) adopted an
attention module to identify a sparse subset of key action segments in a
video, and fused the key segments into action regions via adaptive temporal
pooling. Similarly, AutoLoc (Shou et al., 2018) directly learned the
boundaries using a novel Outer-Inner-Contrastive (OIC) loss to provide the
desired segment-level supervision, and W-TALC (Paul et al., 2018) introduced
the Co-Activity Similarity Loss and jointly optimized it with the cross-
entropy loss for weakly-supervised detecting temporal actions. The recent
state-of-the-art STAR (Xu et al., 2019) focuses on the relationship learning
among multiple actions, which exploits recurrent networks for assembling
expected action instances into high-level feature representation, and then
predicted class labels and locations for each action category step-by-step.
Most of them apply the attention mechanism integrating the Class Activation
Sequence(CAS) mechanism to capture the most discriminative temporal regions,
while they ignore the long-duration of action occurring problem or the missing
of trivial action regions. Therefore, previous methods usually falls into poor
performance on the evaluation of high-IoU localization. Fortunately, recent
work (Zhong et al., 2018) found the contradiction between classifier and
detector and thus designed a step-by-step erasion approach to train the one-
by-one classifiers. This provides us the inspiration to further mine the less
discriminative features from a complete video iteratively.

###  2.2. Weakly-Supervised Object Localization.

Weakly supervised object localization methods locate target objects using
convolutional classification networks. The widely-used Class Activation Map
(CAM) (Zhou et al., 2016) can be used to discover the spatial distribution of
discriminative parts, while they can but find very small and noncontinuous
regions of the entire objects. To handle with the weakness, Hide-and-seek
(Singh and Yong, 2017) tried to force the model to see different parts of the
image and focused on multiple relevant parts of the object beyond just the
most discriminative one. It is implemented by randomly masking different
regions of training images in each training epoch. Erasing-based approaches
(Wei et al., 2017) are proposed to mine the complementary object regions other
than the former most discriminative parts, and then use the fused results as
the final object localization. Furthermore, semantic segmentation methods
(Kolesnikov and Lampert, 2016; Huang et al., 2018) treat the salient parts of
object as seeds, and then iteratively expand the regions to a definite object
boundary. These explorations based on image-level annotations have exemplified
the weakly supervised object detection, which could provide valuable
experiences to the temporal action localization.

Figure 2. The proposed ASSG architecture. (a) Encoded segmental features from
video inputs. (b) The SSG module, with CAS-based results as initial seeds,
iteratively extending the high reliable action or background regions using
growing rules, which learns the category of reliable input segments by a
seeding loss. (c) Erasion of the seeding regions from the shared feature map
from SSG. (d) An action classifier, which uses self-adapted pooling (SAP) for
feature aggregation into the final class confidence with the cross-entropy
loss.

##  3\. Methodology

###  3.1. Overview

Given an untrimmed video, we traditionally trim the sequence into $N$
segments, and encode each segment as a $K$-dimensional feature vector with a
pre-trained two-stream video feature extractor. That is,
$X=\\{x_{t}\\}_{t=1}^{N},x_{t}\in\mathbb{R}^{K}$. Our goal is to localize all
action instances in videos via an action detection model trained with only
video-level annotations. Here, the annotation of each video is defined as
$Y=\\{y_{i}\\}_{i=1}^{M}$, $y_{i}\in\\{1,2,\cdots,\mathcal{C}\\}$, where $M$
is the number of categories in this single video and
$\\{1,2,\cdots,\mathcal{C}\\}$ means the set of action categories.

For this purpose, we design an end-to-end action detection framework composed
of two adversarial parts: 1) the Seeded Sequence Growing (SSG) module for
extending action occurring durations with pre-fetched initial seeds, which is
detailed in Subsection 3.2, and 2) the self-adaptive action classifier for
further exploiting the missing or incomplete instances, which is detailed in
Subsection 3.3. The two modules are integrated into an entire framework and
trained in an adversarial manner, as shown in Figure 2.

###  3.2. Seeded Sequence Growing

The SSG module first uses the reliable weakly-supervised results as initial
seeds to generate reliable supervision of sparse discriminative regions. Then
it progressively increases the seeded temporal regions.

####  3.2.1. Initial Seeds.

Foreground seeds can be generated from CAS-based networks (Nguyen et al.,
2018; Shou et al., 2018; Xu et al., 2019). The sparse and high reliable action
regions are the peaks in the CAS activations by a relatively high threshold.

Background refers to those non-action occurring durations. Assuming that a
background region is likely to appear between two action durations when it
comes to a scene change, while action occurring at a time always has the
consistent shot motion, we utilize the saliency detection (Wang et al., 2017)
to capture shot changes as the probable background seeds.

Here, we formalize the initializing seeds as
$\mathcal{S}=\\{\mathcal{S}_{c}\\}_{c=0}^{C}$, where $c=0$ represents the
background, and $c\in\\{1,\cdots,\mathcal{C}\\}$ represents each action
category respectively.

####  3.2.2. Backbone Network.

The SSG network is learning independent temporal heatmaps for each action
category. It is designed to progressively label more reliable action locations
from the initial seeds $\mathcal{S}$.

Concretely, the backbone of SSG module first stacks two temporal convolutional
layers (striding filters along time dimension), in which each temporal
convolutional layer follows the setting as {filters=512, kernel size=1 ,
stride=1}. A ReLU layer follows each temporal convolution. On top of the SSG
is also a temporal convolution layer but for producing the _class heatmaps_
for each temporal segments
$\mathcal{H}=\\{\mathcal{H}_{c,t}|c\in\\{0,\cdots,C\\},t\in\\{1,\cdots,N\\}\\}$,
in which $\mathcal{H}_{c,t}$ means the class $c$ probability distribution of
the $t$-th segment in the video.

####  3.2.3. Growing Strategy.

Inspired by the growing strategy of dynamic supervision (Huang et al., 2018),
we propose a one-dimension growing policy to dynamically enlarge the seeded
sequences on each independent class heatmap $\mathcal{H}_{c,t}$. Once
initialized, the current action regions are grown by expanding those seeds
$\mathcal{S}$ to neighboring unlabeled locations $\mathcal{N}(\mathcal{S})$
via the growing criterion $\mathcal{G}$:

(1)
$\mathcal{G}(\mathcal{H}_{c,t},\mathcal{S}_{c},\theta_{g}^{c})=\begin{cases}True,&l\in\mathcal{N}(\mathcal{S}_{c}),~{}\mathcal{H}_{c,t}\geq\theta_{g}^{c}{}\\\
&\text{and}~{}c=\mathop{\arg\max\limits_{c^{\prime}}}\mathcal{H}_{c^{\prime},t}\\\
False,&otherwise.\end{cases},$

where $\theta_{g}$ is the pre-set _growing threshold_ value respectively for
each action class and the background. Here, we use a simple definition of
$\mathcal{N}(\mathcal{S})$ to be the set of locations next to each seed in
$\mathcal{S}$. Considering this criterion, if $\mathcal{G}$ is true, we label
the category of the $t$-th segment with class $c$ as newly added supervision
regions. Iteratively, we motivate the activations of both the action occurring
and the background durations on heatmaps alternately with the grown
supervision.

In practice, since the co-occurrence locations can not be assigned to two
different classes when conducting the seed-grow mechanism in original temporal
segmentation framework, we generate separated seeds for each action class
(including the background) and expand the seeded regions respectively. That
is, the SSG predicts individual action occurring regions one-by-one with
growing policy for each class.

###  3.3. Self-Adaptive Action Classifier

This module is designed to mine the relatively long or trivial actions, which
shares the feature map with SSG. It first erases the most discriminative part
dynamically activated by SSG, and then predicts the action class by directly
_aggregating_ the output of shared feature maps into classification confidence
scores. In this way, the classifier adaptively updates the shared parameters
supervised by video class annotations without adding extra parameters.

To be specific, in the first step, we _extract_ the foreground feature maps
from the entire maps $\mathcal{F}$ in SSG, and then _erase_ the highly
activated regions to generate the remaining feature maps
$\grave{\mathcal{F}}$. As $\mathcal{F}$ contains mixed activations of the
foreground and the background, we need to draw only the foreground features
for classification. Therefore, a pair of opposite ReLU activations is designed
to generate $\mathcal{F}$, which forces the foreground or background seeds to
grow in the positive or negative activation parts respectively. Naturally, the
classifier can obtain the foreground features by a ReLU layer. The erasion is
simply implemented by thresholding on the activation values.

The next step is the _aggregation_ of feature maps. The common _aggregating_
approach like global max-poling (GMP) (Oneata et al., 2014) or global average-
pooling (GAP) (Zhou et al., 2016) is not suitable for this task, since the
former ignores too many less discriminative regions and in the later case, the
global feature will be overwhelmed by the large-scale occupied background
segments. For the purpose of inspiring full potential of the classifier, we
design a Self-Adaptive Pooling (SAP) approach for straightforward video class
prediction. The SAP is to re-balance the weights of segments with an off-the-
shelf attention weights $A_{c,t}(X)$ for class $c$ at temporal location $t$,
which could be achieved from the learned feature maps shared with the SSG as

(2) $\displaystyle SAP(X)$
$\displaystyle=\sum_{t=1}^{N}A_{\ast,t}(X)\cdot\grave{\mathcal{H}}_{\ast,t}.$

Here, $\cdot$ is the dot product operation of two scalars. Similarly to
$\mathcal{H}$, $\grave{\mathcal{H}}$ here is the activation distribution after
erasion. The self-adaptive weighted aggregation of $\grave{\mathcal{H}}$ over
the entire $N$ temporal segments results in the entire video-level class
distribution.

In the equation, the attention weights $A_{c,t}(X)$ can be formulated by

(3)
$A_{c,t}(X)=\frac{e^{(\sum_{i=1}^{|f_{c,t}(X)|}f_{c,t}^{i}(X))}}{\sum_{t=1}^{N}e^{(\sum_{i=1}^{|f_{c,t}(X)|}f_{c,t}^{i}(X))}},$

where $f(\cdot)$ represents the mapping functions for the foreground features
$\grave{\mathcal{F}}$ from the network inputs. $|f_{c,t}(X)|$ represents the
feature dimensions at each location on $\mathcal{F}$.

Note that, assuming the highly activated regions are likely to be the
interested action occurrences, $A(X)$ is the self-adaptive weight directly
computed from the feature map without any extra explicit attention modules, .

###  3.4. Training of ASSG

The two modules (i.e.,the SSG and the action classifier) are integrated into a
whole network and trained in an adversarial manner.

####  3.4.1. Seeded Sequence Growing Loss.

$T_{c}$ is a set of temporal locations that are identified as action class
$c$. Here, the seeding loss $L_{seed}$ is defined as:

(4)
$\begin{split}L_{seed}=&-\frac{1}{\sum_{c\in{[0,C]}}|T_{c}|}\sum_{c\in{[0,C]}}\sum_{t\in{T_{c}}}log\mathcal{H}_{c,t}\\\
\end{split}$

The SSG learns the parameter by optimizing the seeding loss function
$L_{seed}$, which encourages the networks to match reliable localization cues
$T_{c}$, including the foreground ($c\in{[1,C]}$) and the background ($c=0$),
but is agnostic about the rest of the locations.

####  3.4.2. Action Classification Loss.

The classification loss is defined as the cross-entropy loss over multiple
categories by

(5)
$\begin{split}L_{class}&=-\sum_{c=1}^{C}\hat{y}_{c}\log{SAP(\hat{y}_{c}|X)},\\\
\end{split}$

where $\hat{y}_{c}$ represents the ground truth of the action category.

####  3.4.3. End-to-end Training.

The whole network is trained in an adversarial manner. By erasing the seeded
regions activated by SSG, the classifier branch poses a more difficult task to
discover minor or even new action regions. Alternately, the classifier will
also boost the seeds growing and generate more reliable regions. The training
procedures are illustrated in Algorithm 1.

Algorithm 1 Framework of ASSG.

1:Training data, $\mathcal{V}=\\{\mathcal{X},y_{c}\\}$; growing threshold,
$\theta_{g}$; adversarial threshold $\theta_{a}$; sequence length $N$; total
categories $C$; initial seeds $\mathcal{S}$

2:Enhanced CAS heatmaps $\mathcal{H}$; prediction label $c$;

3:Initialize $\mathcal{H}$;

4:Initialize the shared deep feature map $\mathcal{F}$;

5:while training not converge do

6: for $t=0\to N-1$ do

7: for $c=0\to C$ do

8: if $\mathcal{G}(\mathcal{H}_{c,t}(\mathcal{X}),\mathcal{S},\theta_{g}^{c})$
then

9: the location at t-th segment is labeled as c;

10: else

11: the location at t-th segment keeps unlabeled state;

12: end if

13: end for

14: end for

15: Update $\mathcal{H}$ with the seeding loss $L_{seed}$.

16: Obtain high reliable regions: $U_{c}$ = {$t|\mathcal{H}_{c,t}>\theta_{a}$
};

17: Erase clips at temporal location in sets $\\{U_{c}\\}_{c=1}^{C}$ for each
category from ${\mathcal{F}}$ to obtain $\grave{\mathcal{F}}$;

18: Calculate SAP($\mathcal{X}$);

19: Update $\grave{\mathcal{F}}$,$\mathcal{F}$ (sharing parameters) with the
cross-entropy loss $L_{class}$.

20: Compute updated $\mathcal{H}$;

21:end while

###  3.5. Location Prediction

We use the predicted heatmap by the SSG module to generate temporal action
proposals. As $\mathcal{H}_{c,t}$ denoted in Subsection 3.2.2 indicates the
probability in the class heatmaps. Here, we fuse the separately trained two-
stream network predictions. For each class c in the corresponding heatmap,
each proposal [$N_{start},N_{end}$] is assigned to a score by:

(6)
$\sum_{t=N_{start}}^{N_{end}}\frac{[\lambda\cdot\mathcal{H}_{c,t}^{RGB}+(1-\lambda)\cdot\mathcal{H}_{c,t}^{flow}]}{N_{end}-N_{start}+1}$

in which we fuse the probability values of RGB and optical flow streams by the
modality ratio $\lambda$. For final detection, we perform non-maximum
suppression (NMS) among temporal proposals of each class by removing highly
overlapped ones.

##  4\. Experiments

###  4.1. Datasets and Evaluation

Datasets. THUMOS’14 (Jiang et al., 2014) is a popular dataset for action
localization task, which consists of 20 action classes, and only its
validation and testing set contains temporal annotations. Since the training
set contains no temporal annotations, the fully-supervised algorithms use the
212 untrimmed videos in validation set to train the network and the 200
testing videos to evaluate the algorithm. To facilitate comparisons, we follow
this conventional protocol but without using the temporal annotations for
training. ActivityNet v1.3 (Heilbron et al., 2015) covers 200 activity
classes. We also use the 10,024 training set videos without temporal
annotations for training and 4926 validation videos for validating. In this
section, we report our results on both THUMOS’14 and ActivityNet v1.3.

Evalution Metrics. In action localization task, mean Average Precision (mAP)
is used as evaluation metrics, where Average Precision (AP) is calculated on
each action class when the prediction is classified correctly and its temporal
overlap Intersection over Union (IoU) with the ground truth segments exceeding
the evaluation threshold. The ablation study are performed on the THUMOS’14
dataset and evaluated with average mAP (Ave-mAP) by calculating the multiple
overlap IoU with thresholds varying from 0.1 to 0.5. The overall performance
compared with the state-of-the-arts is evaluated with average mAP from 0.1 to
0.5 on THUMOS’14 and 0.5 to 0.95 on Activitynet v1.3.

###  4.2. Implementation Details

We implement our algorithm using Caffe (Jia et al., 2014). For comparison, we
employ the common procedures described in (Nguyen et al., 2018; Xu et al.,
2019) to uniformly sample 400 segments from each video. For extracting visual
features, then we use the two-stream I3D network described in (Carreira and
Zisserman, 2017) pre-trained on Kinetics dataset (Kay et al., 2017). For the
CAS-based network, we realize the ST-GradCAM with the pre-defined parameters
described in (Xu et al., 2019) as our default setting. Note that we train the
ST-GradCAM without a specific sub-module for repetition alignment, which needs
additional annotations of action frequency. Our proposed network is trained by
Adam optimizer with initial learning rate $10^{-4}$ on both streams. For the
growing threshold $\theta_{g}$, we set both foreground and background
threshold as 0.99. Besides, the earsing threshold $\theta_{a}$ is set to 0.4
on both datasets. In location prediction, we empirically set $\lambda$ to 0.3.

Figure 3. Visualization of action localization by ASSG network. Temporal
confidence distribution (the predicted heatmap in SSG) of the action class
Hanmmer Throw is denoted in red curve and the prediction of the background is
in blue curve. The yellow locations are selected samples of enhanced detection
compared with the common CAS results.

(a) The growing threshold $\theta_{g}$

(b) The erasing threshold $\theta_{a}$

Figure 4. Performance with different thresholds values. Figure (a) shows mAP
evaluation of different $\theta_{g}$ choices, where $\theta_{gf}$ and
$\theta_{gb}$ means the foreground and background threshold respectively.
Figure (b) shows mAP evaluation of different $\theta_{a}$ choices.

###  4.3. Ablation Study

We conduct ablation study on two pre-defined thresholds, effect of feature
aggregation methods and different modules. Qualitative evaluation is also
provided in this section.

####  4.3.1. Thresholds.

We pre-define two thresholds in our ASSG framework: $\theta_{g}$ for deciding
to label the segment or not in the seeded sequence growing rule, and
$\theta_{a}$ for erasing the high activated regions from SSG in the
classification branch, respectively.

For the growing threshold $\theta_{g}$, we could set different thresholds for
different classes and background respectively. For convenience, we set the
same threshold number for all the actions as foreground threshold
$\theta_{gf}$ and another background threshold $\theta_{gb}$. Then we fix the
$\theta_{gb}=0.99$ and vary the $\theta_{gf}$ from 0.8 to 0.95 with a step
size 0.05, vice verse. As shown in Figure 4(a), the choice of different
threshold values has small influence on the final results, which is convenient
and robust for network training.

For the adversarial threshold $\theta_{a}$, it strikes a balance between the
adversarial training of two modules. We set the threshold from 0.3 to 0.7 and
observe that the localization performance is boosted when the threshold
$\theta_{a}$ = 0.4 as shown in Figure 4(b). Larger value or smaller would
introduce more background noises and thus slightly influence the minor region
mining.

####  4.3.2. Feature Aggregation.

In the self-adaptive action classifier module, we aggregate the shared
features directly into classification scores without any additional learning
parameters. Instead of the common GMP and GAP approach, we introduce an
aggregation method called SAP in Subsection 3.3. Here we discuss the different
aggregation types used in our classifier in Table 1.

We conclude that the low performance of GAP is mainly because the foreground
features are overwhelmed by the background segments, while GMP ignores too
many regions. Our proposed SAP achieves best performance at all IoUs compared
with the existing GMP and GAP, which verifies the effectiveness of our well-
designed self-adaptive weighted aggregation method SAP.

Table 1. Evaluation of different aggregation methods in terms of mAP@IoU on
THUMOS’14.

Methods | 0.1 | 0.2 | 0.3 | 0.4 | 0.5
---|---|---|---|---|---
GMP | 52.8 | 46.4 | 37.6 | 29.0 | 18.4
GAP | 50.6 | 45.1 | 36.5 | 27.9 | 17.5
SAP | 60.1 | 54.6 | 45.1 | 34.3 | 22.4

####  4.3.3. Architecture and Modules.

We study the effects of different modules in the entire framework. The CAS,
CAS w/ SSG (short in SSG), and the CAS w/ SSG w/ classifier (short in ASSG)
results are shown in Table 2. The SSG module boosts the evaluation of the top
reported CAS average mAP (Xu et al., 2019) from 24.4% to 34.2%. And the
additional classification branch for adversarial training jointly denoted as
ASSG further increases the mAP value of SSG by 9.3%, definitely a large
margin. Each of the two modules plays an important role in improving the
detection results.

Table 2. Evaluation of different modules in ASSG on THUMOS’14.

Reported CAS (Xu et al., 2019) | $\surd$ | $\surd$ | $\surd$
---|---|---|---
SSG (CAS w/ SSG) |  | $\surd$ | $\surd$
ASSG (CAS w/ SSG w/ classifier) |  |  | $\surd$
Ave-mAP(%) | 24.4 | 34.2 | 43.5

####  4.3.4. Qualitative Evaluation.

Visualization results are shown in Figure 3. We randomly select a test video
in THUMOS’14 dataset (including the action Hammer Throw) for the trained
proposed ASSG network initialized with the reimplemented CAS baselines (Xu et
al., 2019). For further demonstrating the performance of ASSG, we
qualitatively analyze the effective of the growing mechanism in different
parts of the entire network.

  * •

The seeds are thresholded results from reimplemented top CAS-based network (Xu
et al., 2019), which effectively detects the most discriminative parts in the
videos by a recognition network and fails in the evaluation of high quality
detection shown as the missing regional or entire action durations highlighted
in yellow on top line. For instance, in the first region denoted in yellow
rectangles, only the start and end point are activated and in the later two
yellow-filled parts, the entire action instances are missing.

  * •

We can obviously find the SSG detections expand to some less discriminative
parts, which could lead to better detection proposals than the direct CAS
result. The improvement verifies the effectiveness of seed-grow mechanism
introduced into the SSG for temporal action detection.

  * •

Finally, the entire proposed network ASSG leads to a more satisfying
localization results as the action and complementary background regions
interconnect each other tightly. When compared with the single SSG module, we
also find that the independent prediction of each foreground class and the
background respectively in ASSG holds more confidence (i.e. the prediction
probabilities are much higher than the single SSG). We attribute the advantage
to the adversarial training of the two modules, which makes it more difficult
for the classifier to identify the video class and thus effectively motivates
it to mine more minor regions and to expand the small seeded region to its
precise boundaries.

###  4.4. State-of-the-Art Comparisons

We compare our model with the state-of-the-art weakly-supervised and fully-
supervised methods on THUMOS’14 and ActivityNet1.3 benchmarks. Table 3 and
Table 4 summarize the results.

Table 3. Comparison with state-of-the-arts on THUMOS’14.

Method | Label | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7
---|---|---|---|---|---|---|---|---
Richard et al. (Richard and Gall, 2016) | strong | 39.7 | 35.7 | 30.0 | 23.2 | 15.2 | – | –
S-CNN (Shou et al., 2016) | strong | 47.7 | 43.5 | 36.3 | 28.7 | 19.0 | 10.3 | 5.3
CDC (Shou et al., 2017) | strong | – | – | 40.1 | 29.4 | 23.3 | 13.1 | 7.9
Gao et al. (Gao et al., 2017) | strong | 54.0 | 50.9 | 44.1 | 34.9 | 25.6 | 19.1 | 9.9
Xu et al. (Xu et al., 2017) | strong | 54.5 | 51.5 | 44.8 | 35.6 | 28.9 | – | –
SSN (Zhao et al., 2017) | strong | 66.0 | 59.4 | 51.9 | 41.0 | 29.8 | 19.6 | 10.7
SSAD (Lin et al., 2017) | strong | 50.1 | 47.8 | 43.0 | 35.0 | 24.6 | – | –
TPC (Yang et al., 2018) | strong | – | – | 44.1 | 37.1 | 28.2 | 20.6 | 12.7
TALNet (Chao et al., 2018) | strong | 59.8 | 57.1 | 53.2 | 48.5 | 42.8 | – | –
Alwasssel et al.(Alwassel et al., 2018) | strong | 49.6 | 44.3 | 38.1 | 28.4 | 19.8 | – | –
BSN (Lin et al., 2018) | strong | – | – | 53.5 | 45.0 | 36.9 | 33.8 | 20.8
UntrimmedNet (Wang et al., 2017) | weak | 44.4 | 37.7 | 28.2 | 21.1 | 13.7 | – | –
Hide-and-Seek (Singh and Yong, 2017) | weak | 36.4 | 27.8 | 19.5 | 12.7 | 6.8 | – | –
Zhong et al. (Zhong et al., 2018) | weak | 45.8 | 39.0 | 31.1 | 22.5 | 15.9 | – | –
AutoLoc (Shou et al., 2018) | weak | – | – | 35.8 | 29.0 | 21.2 | 13.4 | 5.8
W-TALC (Paul et al., 2018) | weak | 55.2 | 49.6 | 40.1 | 31.1 | 22.8 | – | 7.6
STPN (Nguyen et al., 2018) | weak | 52.0 | 44.7 | 35.5 | 25.8 | 16.9 | 9.9 | 4.3
STAR (Xu et al., 2019) | weak | 68.8 | 60.0 | 48.7 | 34.7 | 23.0 | – | –
STPN-CAS w/ ASSG | weak | 55.6 | 49.5 | 41.1 | 31.5 | 20.9 | 13.7 | 5.9
STAR-CAS w/ ASSG | weak | 65.6 | 59.4 | 50.4 | 38.7 | 25.4 | 15.0 | 6.6

Results on THUMOS’14. The comparison results between the proposed ASSG and
other state-of-the-art models are shown in Table 3. As ASSG is a framework
enhancing the CAS detections, its seeds can be initialized by various CAS-
based models. In this perspective, we conduct ASSG based on two typical
structure of CAS-based networks, respectively with CAS results from STPN
(Nguyen et al., 2018) as initial seeds (denoted as STPN-CAS w/ ASSG) and CAS
results from a constrained STAR (Xu et al., 2019) (excluding the specific sub-
module with additional action frequency annotation) as initial seeds (denoted
as STAR-CAS w/ ASSG). For a fair comparison, we also follow their prediction
operations to fuse attention weights with the CAS by generating proposals
separately.

We find that STPN-CAS w/ ASSG improves the performance by a large margin
compared to the original STPN result, similarly in STAR-CAS w/ ASSG compared
with STAR. The STAR-CAS w/ ASSG outperforms all other weakly-supervised
methods with multiple overlap IoU thresholds varied from 0.3 to 0.6. For
instance, when the IoU threshold used in evaluation is set to 0.5, ASSG
network improves the state-of-the-art results from 23.0% to 25.4%. It is
noting that at IoU threshold of 0.1 and 0.2, our ASSG still achieves
impressive performance, which surpasses all other methods except for STAR (Xu
et al., 2019). While STAR used additional action frequency annotations
(different from all existed weakly-supervised works), and it also reported
results without frequency annotations, i.e., avg-mAP ranging from 0.1 to 0.5
is 44.0%, which falls behind our ASSG 47.9% by a 3.9%. In specific, we
attribute the effectiveness of our approach above CAS-based results (Nguyen et
al., 2018; Shou et al., 2018; Paul et al., 2018; Xu et al., 2019) (especially
in higher IoU thresholds) to the ability of ASSG to mine less discriminative
action regions, which results in more precise boundaries and completeness of
detection.

We also compare the results with the fully-supervised methods. The performance
of our weakly-supervised model with only video-level class annotations in
training, even achieves comparable results with the state-of-the-art strong-
supervised ones, by only 0.4% behind the TALNet (Chao et al., 2018) at IoU
threshold of 0.1 and 3.1% behind the BSN (Lin et al., 2018) at IoU threshold
of 0.3.

Results on ActivityNet1.3. Table 4 shows the results on ActivityNet1.3
dataset. Note that, the dataset characteristics differ largely from those in
THUMOS’14 that many videos in ActivityNet1.3 are including relatively long
action durations per instance. Therefore, THUMOS’14 is a better dataset for
evaluating most action localization methods, which is also claimed in (Chao et
al., 2018). Since our designed framework is to expand the action regions from
the initial small and sparse reliable regions, connecting all the discrete
parts into a unified long-duration action poses a great challenge.

Results show that ASSG gets better overall performance than all the existing
weakly-supervised results by increasing the mAP at IoU threshold of 0.5 and
0.75 from 31.1% to 32.3% and 18.8% to 20.1% respectively. We do not care about
the mAP at IoU threshold of 0.95 since the precision is relatively low, which
makes little sense in current situations. Although the performance gain is
smaller compared to that in THUMOS’14, the improvement performance also
verifies the common effectiveness on both datasets.

Table 4. Comparison with state-of-the-arts on ActivityNet1.3.

Method | Label | 0.5 | 0.75 | 0.95
---|---|---|---|---
Singh et al. (Singh and Cuzzolin, 2016) | strong | 34.5 | – | –
CDC (Shou et al., 2017) | strong | 45.3 | 26.0 | 0.2
SSN (Xiong et al., 2017) | strong | 39.1 | 23.5 | 5.5
SSAD (Lin et al., 2017) | strong | 49.0 | 32.9 | 7.9
Chao et al. (Chao et al., 2018) | strong | 38.2 | 18.3 | 1.3
BSN (Lin et al., 2018) | strong | 52.5 | 33.5 | 8.9
STPN (Nguyen et al., 2018) | weak | 29.3 | 16.9 | 2.6
STAR (Xu et al., 2019) | weak | 31.1 | 18.8 | 4.7
STAR-CAS w/ ASSG | weak | 32.3 | 20.1 | 4.0

##  5\. Conclusion

By observing the weakness of CAS-based approach that only the most
discriminative parts can be detected, we extend the seed-grow mechanism to our
weakly-supervised temporal action detection. We design a framework of two
modules, an SSG and an action classifier respectively, which jointly help
small and sparse action occurring durations grow. The two modules are trained
in an adversarial manner. The operation of erasing seeded regions forces the
classifier to handle with a more difficult task by focusing on less
discriminative regions. Alternately, the classifier drives the seeds to grow.
Extensive experiments demonstrate that our ASSG achieves superior performance
on the challenging THUMOS’14 above all other weakly-supervised methods,
especially on the evaluation of high IoUs, and has impressive results on the
ActivityNet1.3 datasets as well.

##  6\. Acknowledgement

Chengwei Zhang and Futai Zou were partially supported by National Key Research
and Development Program of China (No.2017 YFB0802300), NSFC-Zhejiang Joint
Fund for the Integration of Industrialization and Informatization (No.
U1509219), and National Key Research and Development Program of China (No.
2018YFB 0803500). THANKS to all members in DAVAR lab.

## References

  * (1)
  * Alwassel et al. (2018) Humam Alwassel, Fabian Caba Heilbron, and Bernard Ghanem. 2018\.  Action Search: Spotting Actions in Videos and Its Application to Temporal Action Localization. In _ECCV_. 251–266.
  * Buch et al. (2017) Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. 2017\.  SST: Single-Stream Temporal Action Proposals. In _CVPR_. 6373–6382.
  * Carreira and Zisserman (2017) Joao Carreira and Andrew Zisserman. 2017.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In _CVPR_. 4724–4733.
  * Chao et al. (2018) Yu Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar. 2018\.  Rethinking the Faster R-CNN Architecture for Temporal Action Localization. In _CVPR_. 2933–2942.
  * Dai et al. (2017) Xiyang Dai, Bharat Singh, Guyue Zhang, Larry S. Davis, and Yan Qiu Chen. 2017.  Temporal Context Network for Activity Localization in Videos. In _ICCV_. 5727–5736.
  * Gao et al. (2017) Jiyang Gao, Zhenheng Yang, and Ram Nevatia. 2017.  Cascaded Boundary Regression for Temporal Action Detection.  _CoRR_ abs/1705.01180 (2017).
  * Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015.  ActivityNet: A large-scale video benchmark for human activity understanding. In _CVPR_. 961–970.
  * Huang et al. (2018) Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, and Jingdong Wang. 2018.  Weakly-Supervised Semantic Segmentation Network with Deep Seeded Region Growing. In _CVPR_. 7014–7023.
  * Jia et al. (2014) Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross B Girshick, Sergio Guadarrama, and Trevor Darrell. 2014\.  Caffe: Convolutional Architecture for Fast Feature Embedding.  _ACM MM_ , 675–678.
  * Jiang et al. (2014) Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar. 2014.  THUMOS Challenge: Action Recognition with a Large Number of Classes.  http://crcv.ucf.edu/THUMOS14/.
  * Kay et al. (2017) Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017.  The Kinetics Human Action Video Dataset.  _CoRR_ abs/1705.06950 (2017).
  * Kolesnikov and Lampert (2016) Alexander Kolesnikov and Christoph H. Lampert. 2016.  Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation. In _ECCV_.
  * Lin et al. (2017) Tianwei Lin, Xu Zhao, and Zheng Shou. 2017.  Single Shot Temporal Action Detection. In _ACM MM_. 988–996.
  * Lin et al. (2018) Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. 2018.  BSN: Boundary Sensitive Network for Temporal Action Proposal Generation. In _ECCV_. 3–19.
  * Nguyen et al. (2018) Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han. 2018\.  Weakly Supervised Action Localization by Sparse Temporal Pooling Network. In _CVPR_. 6752–6761.
  * Oneata et al. (2014) Dan Oneata, Jakob J Verbeek, and Cordelia Schmid. 2014\.  Efficient Action Localization with Approximately Normalized Fisher Vectors. In _CVPR_. 2545–2552.
  * Paul et al. (2018) Sujoy Paul, Sourya Roy, Amit K Roy Chowdhury, and Amit K. 2018\.  W-TALC: Weakly-supervised Temporal Activity Localization and Classification. In _ECCV_. 563–579.
  * Richard and Gall (2016) Alexander Richard and Juergen Gall. 2016.  Temporal Action Detection Using a Statistical Language Model. In _CVPR_. 3131–3140.
  * Shou et al. (2017) Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang. 2017\.  CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos. In _CVPR_. 5734–5743.
  * Shou et al. (2018) Zheng Shou, Hang Gao, Lei Zhang, Kazuyuki Miyazawa, and Shih-Fu Chang. 2018.  AutoLoc: Weakly-supervised Temporal Action Localization in Untrimmed Videos. In _ECCV_. 154–171.
  * Shou et al. (2016) Zheng Shou, Dongang Wang, and Shih Fu Chang. 2016.  Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs. In _CVPR_. 1049–1058.
  * Singh and Cuzzolin (2016) Gurkirt Singh and Fabio Cuzzolin. 2016.  Untrimmed Video Classification for Activity Detection: submission to ActivityNet Challenge.  _CoRR_ abs/1607.01979 (2016).
  * Singh and Yong (2017) Krishna Kumar Singh and Jae Lee Yong. 2017.  Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-Supervised Object and Action Localization. In _ICCV_. 3544–3553.
  * Wang et al. (2017) Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool. 2017\.  UntrimmedNets for Weakly Supervised Action Recognition and Detection. In _CVPR_. 6402–6411.
  * Wei et al. (2017) Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming Ming Cheng, Zhao Yao, and Shuicheng Yan. 2017\.  Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. In _CVPR_.
  * Xiong et al. (2017) Yuanjun Xiong, Yue Zhao, Limin Wang, Dahua Lin, and Xiaoou Tang. 2017.  A Pursuit of Temporal Accuracy in General Activity Detection.  _CoRR_ abs/1703.02716 (2017).
  * Xu et al. (2017) Huijuan Xu, Abir Das, and Kate Saenko. 2017.  R-C3D: Region Convolutional 3D Network for Temporal Activity Detection. In _ICCV_. 5783–5792.
  * Xu et al. (2019) Yunlu Xu, Chengwei Zhang, Zhanzhan Cheng, Jianwen Xie, Yi Niu, Shiliang Pu, and Fei Wu. 2019.  Segregated Temporal Assembly Recurrent Networks for Weakly Supervised Multiple Action Detection.  _AAAI_.
  * Yang et al. (2018) Ke Yang, Peng Qiao, Dongsheng Li, Shaohe Lv, and Yong Dou. 2018.  Exploring Temporal Preservation Networks for Precise Temporal Action Localization. In _AAAI_. 7477–7484.
  * Zhao et al. (2017) Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. 2017\.  Temporal Action Detection with Structured Segment Networks. In _ICCV_. 2933–2942.
  * Zhong et al. (2018) Jia Xing Zhong, Nannan Li, Weijie Kong, Zhang Tao, and Li Ge. 2018.  Step-by-step Erasion, One-by-one Collection: A Weakly Supervised Temporal Action Detector. In _ACM Multimedia Conference_. 35–44.
  * Zhou et al. (2016) Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016.  Learning Deep Features for Discriminative Localization. In _CVPR_. 2921–2929.