# Superconductivity in kagome metal YRu3Si2 with strong electron correlations

Chunsheng Gong, Shangjie Tian, Zhijun Tu, Qiangwei Yin, Yang Fu, Ruitao Luo,
and Hechang Lei<EMAIL_ADDRESS>Department of Physics and Beijing Key
Laboratory of Opto-electronic Functional Materials $\&$ Micro-nano Devices,
Renmin University of China, Beijing 100872, China

###### Abstract

We report the detailed physical properties of YRu3Si2 with the Ru kagome
lattice at normal and superconducting states. The results of resistivity and
magnetization show that YRu3Si2 is a type-II bulk superconductor with
$T_{c}\sim$ 3.0 K. The specific heat measurement further suggests that this
superconductivity could originate from the weak or moderate electron-phonon
coupling. On the other hand, both large Kadawaki-Woods ratio and Wilson ratio
indicate that there is a strong electron correlation effect in this system,
which may have a connection with the featured flat band of kagome lattice.

##  I Introduction

Due to the unique two-dimensional (2D) structure formed by corner-sharing
triangles, the kagome lattice provides an exciting platform for studying
magnetic frustration, electronic correlation and topological electronic state.
Besides the long-sought spin liquid state in insulating magnetic kagome
materials with strong geometrical frustration Balents ; Broholm ; Shores ; Han
, the topological electronic structure in metallic kagome systems (kagome
metals), such as Dirac point, flat band and von Hove point rooting in the
special symmetry and arrangement of 2D kagome lattice, has also attracted much
attention recently YeL ; KangM ; LiuZ ; KangM2 . More importantly, the
interplay between magnetism and band topology has resulted in the emergence of
various exotic correlated topological phenomena and matters, such as giant
anomalous Hall effect, massive Dirac fermions with large magnetic field
tunability, magnetic Weyl semimetal state, and Chern gapped Dirac fermions
with chiral edge state Nakatsuji ; Nayak ; YeL ; WangQ1 ; LiuE ; WangQ2 ; Yin1
; Yin2 .

In contrast, the experimental studies on other electronic correlation effects
intertwining with band topology in kagome metals are scarce due to the lack of
material systems. The discovery of superconductivity and CDW state in AV3Sb5
(A = K, Rb, Cs) in very recent provide a novel platform to investigate such
relationship Ortiz ; Ortiz2 ; Ortiz3 ; YinQW , which has been intensively
studied in theory previously Ko ; WangWS ; Kiesel ; Mazin . Moreover, the
observed chiral charge order at high temperature and pair density wave state
below superconducting transition temperature $T_{c}$ further imply the
intricate relationship between these correlated ordering states and non-
trivial band topology JiangYX ; ChenH .

In order to deepen the understanding of such correlated topological phenomena,
it is necessary to explore other kagome metals harbouring superconductivity
(kagome superconductors) But only a handful of kagome superconductors have
been reported until now. One of examples is the Laves phase [cubic MgCu2 (C15)
type] superconductors with widely varying $T_{c}$ ranging from 0.07 K to above
10 K Rapp ; SunS ; Gong . However, since the strong interaction between kagome
layer and other layers along the (111) direction of cubic Laves phase, the
featured electronic structure of 2D kagome lattice is strongly disturbed and
the topological features of these materials become hard to discern. Another
family of kagome superconductors is RTm3B2 (R = Y, La, Lu, Th and Tm = Ru, Os,
Rh, Ir) and RRu3Si2 with hexagonal layered CeCo3B2 structure in which
transition metals form kagome layers Ku . Among these materials, LaRu3Si2 with
$T_{c}$ as high as 7.8 K attracted lots of attention because it shows a
typical band structure of kagome lattice and strong electron correlations
which could be closely related to the kagome flat band Barz ; Escorne ; Li ;
BXLi ; III . When compared with LaRu3Si2, except the structure and $T_{c}$
value ($\sim$ 3 K - 3.5 K) Barz ; Vandenberg , the physical properties of
YRu3Si2, especially superconducting properties, are still lacking. In this
work, we carry out a detailed study on the physical properties of YRu3Si2 at
normal and superconducting states. Experimental results indicate that YRu3Si2
is a weakly or moderately coupled BCS superconductor with strong electron
correlations.

##  II Experimental

YRu3Si2 polycrystal was synthesised using an arc-melting method. First, Y
metal filing (99.9 $\%$), Ru powder (99.9 $\%$), and Si powder (99.9 $\%$)
with stoichiometric ratios were mixed, ground thoroughly and then pressed into
a pellet using a hydraulic press in a glover box filled with argon atmosphere.
In order to avoid the formation of YRu2Si2, an amount of extra Ru was added,
similar to the preparation of LaRu3Si2 Li . The pellet was arc-melted under
argon atmosphere and remelted several times from both sides of the pellet to
improve the homogeneity. Crystal structure and phase purity were examined by
powder X-ray diffraction pattern (PXRD) with Cu Kα radiation ($\lambda$ =
$0.15418$ nm) using a Bruker D8 X-ray diffractometer. The lattice parameters
are extracted by fitting the PXRD pattern using the TOPAS4 software.topas
Electrical transport and specific heat measurements were performed in a
Quantum Design PPMS-14T. Magnetization measurements were carried out in a
Quantum Design MPMS3.

##  III Results and discussion

Figure 1: (a) Crystal structure of YRu3Si2. The blue, red and yellow balls
represent Y, Ru and Si atoms, respectively. (b) Top view of the 2D distorted
kagome lattice of Ru atoms. Two different bond distances are labelled with
green and orange lines. (c) Powder XRD pattern and Rietveld refinement of
YRu3Si2 polycrystal.

Fig. 1(a) shows the crystal structure of YRu3Si2, isostructural to LaRu3Si2
Vandenberg . It is formed by stacking Y-Si and Ru layers alternately along the
$c$ axis. A key feature of this compound is the 2D distorted kagome layer
formed by Ru atoms parallel to the $ab$ plane. In this layer, there are two
different Ru-Ru bond distances $d_{\rm Ru-Ru}$ (= 0.2692 nm and 0.2858 nm).
Meanwhile, the rotation of Ru triangles leads to the bond angle for collinear
Ru atoms $\theta$ (= 174.046∘) deviating away from $\theta=$ 180∘ in perfect
kagome lattice (Fig. 1(b)). On the other hand, for Y-Si layer, the Y atoms
form a triangle lattice when the Si atoms construct a honeycomb structure.
Fig. 1(c) shows the PXRD pattern of YRu3Si2 and all peaks can be indexed well
by the $P6_{3}/m$ space group (No. 176). The obtained lattice parameters by
using Rietveld refinement are $a=$ 0.5542(1) nm and $c=$ 0.7150(7) nm,
consistent with the reported values in literature Vandenberg . In addition to
the diffraction peaks of YRu3Si2, there are some extra peaks which originate
from the excess of Ru in the raw material. A fitting result shows that the
weight ratio of YRu3Si2 to Ru is about 84.1 : 15.9.

Figure 2: (a) Temperature dependence of magnetic susceptibility $4\pi\chi(T)$
at 1 mT with ZFC and FC models. (b) Magnetization hysteresis loop for YRu3Si2
at 1.8 K. (c) Low-field dependence of magnetization $4\pi M(\mu_{0}H)$ at
various temperatures below $T_{c}$. (d) $\mu_{0}H_{c1}$ as a function of
$T/T_{c}$. The red line represents the fit using G-L equation.

The temperature dependent magnetic susceptibility $4\pi\chi(T)$ with the zero
field cooling (ZFC) and field cooing (FC) modes is shown in Fig. 2(a). A clear
diamagnetic transition in $4\pi\chi(T)$ curve can be observed and confirms the
occurrence of superconductivity in YRu3Si2. The $T_{c}$ determined from the
ZFC $4\pi\chi(T)$ curve at $\mu_{0}H=$ 1 mT is about 2.98 K, close to the
value reported previously Barz . At $T=$ 1.8 K, the estimated superconducting
volume fraction is about 110 %, indicating a bulk superconductivity in
YRu3Si2. Compared with the ZFC $4\pi\chi(T)$ curve, a relatively weak
diamagnetic signal of FC $4\pi\chi(T)$ curve due to flux pinning effect
suggests that YRu3Si2 is a type-II superconductor. It is further confirmed by
the magnetization hysteresis loop measured at 1.8 K (Fig. 2(b)). Moreover, the
sudden jumps of $M(\mu_{0}H)$ at low-field region can be ascribed to the entry
of Meissner state. The initial magnetization as a function of magnetic field
in the temperature range between 1.8 K and 2.9 K is shown in Fig. 2(c). All
curves clearly fall on the same line and deviate from linearity at different
temperatures. In order to estimate the lower critical field $\mu_{0}H_{c1}$
correctly, the effective field $\mu_{0}H_{\rm eff}$ is calculated by using the
formula $\mu_{0}H_{\rm eff}=\mu_{0}H$ \- $N_{d}M$, where $N_{d}$ is the
demagnetization factor Aharoni . Using the geometry size of rectangular
sample, the calculated value of $N_{d}$ is about 0.522 Aharoni . The
$\mu_{0}H_{c1}$ at each temperature is determined from the field where the
$4\pi M(\mu_{0}H)$ curve deviates from linearity (”Meissner line”). The
obtained $\mu_{0}H_{c1}$ as a function of reduced temperature $T/T_{c}$ is
shown in Fig. 2(d). The temperature dependence of $\mu_{0}H_{c1}$ can be well
fitted by the Ginzburg-Landau (G - L) equation
$\mu_{0}H_{c1}(T)=\mu_{0}H_{c1}(0)[1-({T}/{T_{c}})^{2}]$, where
$\mu_{0}H_{c1}(0)$ is the lower critical field at $T=$ 0 K (red line). The
fitted $\mu_{0}H_{c1}(0)$ for YRu3Si2 is 28.0(3) mT.

Figure 3: (a) Temperature dependence of electrical resistivity $\rho(T)$ for
YRu3Si2 polycrystal at zero field. The red solid line represents the fit using
the formula $\rho(T)=\rho_{0}+AT^{2}+BT^{5}$. Inset: enlarged view of
$\rho(T)$ curve near superconducting transition. (b) Temperature dependence of
$\mu_{0}H_{c2}(T)$. The green and purple lines represent the fits using the G
- L and WHH formulas. Inset: $\rho(T)$ as a function of $T$ at various
magnetic fields.

Fig. 3(a) shows the temperature dependence of electrical resistivity $\rho(T)$
for YRu3Si2 polycrystal from 300 K to 2 K at zero field. The $\rho(T)$
decreases monotonically with decreasing temperature, indicating the metallic
behavior of YRu3Si2. The value of residual resistivity ratio (RRR) $\rho$(300
K)/$\rho$(4 K) is about 8.6. Such relatively small value of RRR can be
explained by the grain boundary effect in YRu3Si2 polycrystal. At high
temperature the $\rho(T)$ curve exhibits a saturation tendency. This behavior
could be described by the Ioffe-Regel limitIoffe , e.g. the electron mean free
path is close to the interatomic distance Zverev . At low temperature region
($T<$ 30 K), the $\rho(T)$ curve can be well fitted using the formula
$\rho(T)=\rho_{0}+AT^{2}+BT^{5}$, where $\rho_{0}$ is the residual
resistivity, $AT^{2}$ and $BT^{5}$ term originates from the electron-electron
and electron-phonon scattering, respectively (red line). The fit gives
$\rho_{0}=4.99(5)$ $\mu\Omega$ cm, $A=$ 2.38(7)$\times 10^{-3}$ $\mu\Omega$ cm
K-2 and $B=$ 5.9(4)$\times 10^{-8}$ $\mu\Omega$ cm K-5. In addition, because
the value of $AT^{2}$ is always larger than $BT^{5}$ at $T<$ 30 K, the
electron-electron scattering process dominates low-temperature resistivity.
With lowering temperature further, a sharp superconducting transition can be
observed at $T_{c,\rm onset}$ = 3.0 K with narrow transition width $\Delta
T_{c}$ = 0.082 K (inset of Fig. 3(a)). It is consistent with $T_{c}$ obtained
from the $4\pi\chi(T)$ curve. The small value of $\Delta T_{c}$ indicates the
high quality of YRu3Si2 polycrystal.

In order to investigate the upper critical field $\mu_{0}H_{c2}$ in YRu3Si2,
the temperature dependence of $\rho(T)$ at various fields up to 0.4 T are
measured (inset of Fig. 3(b)). The $T_{c}$ shifts to lower temperature and the
$\Delta T_{c}$ broadens slightly with increasing field. At $\mu_{0}H=$ 0.4 T,
the superconducting transition can not be observed above 2 K. When defining
the $T_{c}$ as 50% drop from normal state resistivity just above
superconducting transition, the temperature dependence of $\mu_{0}H_{c2}(T)$
for YRu3Si2 is plotted in Fig. 3(b). Using the G - L formula
$\mu_{0}H_{c2}(T)=\mu_{0}H_{c2}(0)(T_{c,0}^{2}-T^{2})/(T_{c,0}^{2}+T^{2})$,
where the $\mu_{0}H_{c2}(0)$ is the zero-temperature upper critical field and
$T_{c,0}$ is superconducting transition temperature at zero field. The fit
(green line) gives $\mu_{0}H_{c2}(0)=$ 0.78(2) T. On the other hand, when
using the one-band Werthamer-Helfand-Hohenberg (WHH) formula (purple line)
Werthamer , the fitted $\mu_{0}H_{c2}(0)$ is 0.655(2) T, close to the value
obtained from the G-L formula. It is lower than that of LaRu3Si2
($\mu_{0}H_{c2}(0)\approx$ 4 T)Li . In addition, the $\mu_{0}H_{c2}(0)$ is
also much smaller than the Pauli paramagnetically limited field
$\mu_{0}H^{p}_{c2}$ (= 1.84 $T_{c}$ = 5.52 T) Maki , indicating that the
orbital pairing-broken mechanism is dominant in YRu3Si2.

Using the fitted values of $\mu_{0}H_{c1}(0)$ (= 28.0(3) mT) and
$\mu_{0}H_{c2}(0)$ (= 0.655(2) T) , the zero-temperature superconducting
characteristic parameters $\xi_{\rm GL}$ (coherence length) and $\lambda_{\rm
GL}$ (penetration depth) can be estimated according to the following two
equations $\xi_{\rm GL}=(\frac{\phi_{0}}{2\pi\mu_{0}H_{c2}(0)})^{\frac{1}{2}}$
and $\mu_{0}H_{c1}=\frac{\phi_{0}}{4\pi\lambda^{2}_{\rm
GL}}\ln\frac{\lambda_{\rm GL}}{\xi_{\rm GL}}$, where $\phi_{0}$ is the
magnetic flux quantum ($h/2e$ = 2.07 $\times 10^{-15}$ T m2). The calculated
value of $\xi_{\rm GL}$ and $\lambda_{\rm GL}$ is 22.43 (2) nm and 24.80(3)
nm, respectively. Correspondingly, the derived GL parameter $\kappa_{\rm
GL}(=\lambda_{\rm GL}/\xi_{\rm GL}$) is 1.11(4), which is larger than
1/$\sqrt{2}$. It confirms that YRu3Si2 is a type-II superconductor. The zero-
temperature thermodynamic critical field $\mu_{0}H_{c}(0)$ is obtained to be
0.42(9) T using the equation
$\mu_{0}H_{c}(0)=[\mu_{0}H_{c1}(0)\mu_{0}H_{c2}(0)/\ln\kappa_{\rm GL}]^{1/2}$.

Fig. 4(a) shows the specific heat $C_{p}(T)$ of YRu3Si2 measured from 300 K to
2 K at zero field. At high temperature, consistent with the Dulong-Petit law,
the $C_{p}(T)$ is close to the value of 3$NR$ ($\sim$ 150 J mol-1K-1), where
$N$ (= 6) is the atomic numbers per formula and $R$ (= 8.314 J mol-1K-1) is
the ideal gas constant, respectively. At low temperature, a specific heat jump
can be observed clearly due to the superconducting transition. It also
confirms the bulk nature of superconductivity in YRu3Si2 (Fig. 4(b)). With
increasing field, this jump shifts to lower temperature accompanying with the
decrease of height. On the other hand, at $\mu_{0}H$ = 0.5 T where the
superconducting transition is suppressed below 2 K, the relationship of
$C_{p}/T$ and $T^{2}$ can not described by a linear behavior (inset of Fig.
4(c)) and this suggests that there may be an anomalous contribution of high-
frequency phonons Li . Thus, the low-temperature $C_{p}(T)$ at $\mu_{0}H$ =
0.5 T is fitted using the formula $C_{p}=\gamma T+\beta T^{3}+\eta T^{5}$,
where the first item is the specific heat of electrons at normal state with
the Sommerfeld coefficient $\gamma$, and the last two items represent the
lattice specific heat when considering the contribution of high-frequency
phonons Yang . The fit gives $\gamma$ = 27.5(8) mJ mol-1K-2, $\beta$ = 0.12(2)
mJ mol-1K-4 and $\eta$ = 0.00141(7) mJ mol-1K-6. By using the formula
$\Theta_{\rm D}=({12\pi^{4}NR}/{5\beta})^{1/3}$, the calculated Debye
temperature $\Theta_{\rm D}$ is 460(26) K. It is larger than the value in
LaRu3Si2 ($\Theta_{\rm D}=$ 412 K) derived using the same method Li , possibly
due to the smaller atomic mass of Y than La.

The specific heat of electrons $C_{\rm ele}$ can be obtained by subtracting
the phonon contribution from the total specific heat (Fig. 4(d)). According to
the method of equal-entropy construction (blue solid lines), the thermodynamic
$T_{c}$ is determined to be 2.84 K, consistenting with the values obtained
from the $\rho(T)$ and $\chi(T)$ curves. The specific heat jump ${\Delta
C_{\rm ele}}/{\gamma T_{c}}$ at $T_{c}$ is 1.21(4) which is smaller than the
value of weak-coupling limit 1.43. It implies that YRu3Si2 might be a weak- or
moderate-coupling superconductor. It has to be noted that the existence of Ru
impurity could also contribute to the reduced specific heat jump. Using the
obtained $\Theta_{\rm D}$ and $T_{\rm c}$, the electron-phonon coupling
constant $\lambda_{\rm e-ph}$ can be calculated according to the McMillan
equationMcMillan ,

$\lambda_{\rm e-ph}=\frac{1.04+\mu^{*}\ln(\frac{\Theta_{\rm
D}}{1.45T_{c}})}{(1-0.62\mu^{*})\ln(\frac{\Theta_{\rm D}}{1.45T_{c}})-1.04}$
(1)

where $\mu^{*}$ is the Coulomb pseudopotential parameter. Taken $\mu^{*}=$
0.13 as for many intermetallic superconductors, the calculated value of
$\lambda_{\rm e-ph}$ is 0.50(2), further confirming that YRu3Si2 is a weak- or
moderate-coupling superconductor. All of physical parameters of YRu3Si2 are
listed in Table I.

Table 1: Physical parameters of YRu3Si2. Parameter | Units | Value
---|---|---
$T_{c}$ | K | 3.0
RRR | - | 8.6
$\rho_{0}$ |  $\mu\Omega$ cm | 4.99(5)
$A$ |  $\mu\Omega$ cm K-2 | 2.38(7)$\times 10^{-3}$
$B$ |  $\mu\Omega$ cm K-5 | 5.9(4)$\times 10^{-8}$
$\mu_{0}H_{c1}(0)$ | mT | 28.0(3)
$\mu_{0}H_{c2}(0)$ | T | 0.655(2)
$\mu_{0}H_{c}(0)$ | T | 0.42(9)
$\xi_{\rm GL}$ | nm | 22.43(2)
$\lambda_{\rm GL}$ | nm | 24.80(3)
$\kappa_{\rm GL}$ | - | 1.11(4)
$\gamma$ | mJ mol-1K-2 | 27.5(8)
$\beta$ | mJ mol-1K-4 | 0.12(2)
$\eta$ | mJ mol-1K-6 | 0.00141(7)
${\Delta C_{\rm ele}}/{\gamma T_{c}}$ | - | 1.21(4)
$\Theta_{\rm D}$ | K | 460(26)
$\lambda_{\rm e-ph}$ | - | 0.50(2)
$\chi(0)$ | emu mol-1 Oe-1 | 5.56(2)$\times$10-4
$T_{\rm E}$ | K | 640(10)
$C$ | emu K mol-1 Oe-1 | 0.01393(7)
$T_{0}$ | K | 1.38(2)
$A/\gamma^{2}$ |  $\mu\Omega$ cm mol2 K2 J-2 | 3.15(2)
$R_{\rm w}$ | - | 1.49(4)

Figure 4: (a) Temperature dependence of zero-field $C_{p}(T)$ from 300 K to 2
K. (b) $C_{p}$ vs $T$ at various magnetic fields. (c) Temperature dependence
of $C_{p}(T)$ at $\mu_{0}H$ = 0.5 T. The red solid line represents the fit
using the formula $C_{p}=\gamma T+\beta T^{3}+\eta T^{5}$. Inset:
${C_{p}}/{T}$ vs $T^{2}$ at $\mu_{0}H$ = 0.5 T. (d) ${C_{\rm ele}}/{T}$ as a
function of $T$ at zero field.

Figure 5: Temperature dependence of $\chi(T)$ measured at $\mu_{0}H$ = 0.5 T
with ZFC mode. The red solid line presents the fit using the formula $\chi(T)$
= $\chi(0)[1-(T/T_{\rm E})^{2}]+C/(T+T_{0})$.

5

In order to get more information about electron correlation effect in YRu3Si2,
temperature dependence of $\chi(T)$ at $\mu_{0}H$ = 0.5 T with ZFC mode is
measured (Fig. 5). YRu3Si2 shows a paramagnetic behavior from 300 K to 2 K and
no long-range magnetic translation is observed. In addition, it is found that
the $\chi(T)$ curve can be well fitted using the following expression as in
LaRu3Si2 Li ,

$\chi(T)=\chi(0)[1-(\frac{T}{T_{\rm E}})^{2}]+\frac{C}{T+T_{0}}$ (2)

Here, the first term represents the Pauli susceptibility related to the
density of state at the Fermi energy level $E_{\rm F}$. The $T_{E}$ represents
a parameter proportional to the $E_{\rm F}$. The second term refers to the
weak magnetic contribution that may be caused by the local moment. The fit
gives $\chi(0)$ = 5.56(2)$\times$10-4 emu mol-1 Oe-1, $T_{\rm E}$ = 640(10) K,
$C$ = 0.01393(7) emu K mol-1 Oe-1 and $T_{0}$ = 1.38(2) K. On the other hand,
the Kadawaki-Woods ratio $A/\gamma^{2}$ and the Wilson ratio $R_{\rm
W}={4\pi^{2}k_{B}^{2}\chi(0)}/{3(g\mu_{B})^{2}\gamma}$ are often used to
characterize the strength of electron correlations, where $g$ is Lande factor
which takes about 2 for an electron and $\mu_{B}$ is the Bohr magneton
Kadowaki ; Wilson . Using above values of $A$ = 2.38(7)$\times 10^{-3}$
$\mu\Omega$ cm, $\gamma$ = 27.5(8) mJ mol-1 K-2 and $\chi(0)$ =
5.56(2)$\times$10-4 emu mol-1 Oe-1, the calculated $A/\gamma^{2}$ and the
$R_{\rm w}$ is 3.15(2) $\mu\Omega$ cm mol2 K2 J-2 and 1.49(4), respectively
(summarized in Table I). The value of $A/\gamma^{2}$ is much larger that those
in transition metals (0.4 $\mu\Omega$ cm mol2 K2 J-2) and relatively smaller
than the universal value 10 $\mu\Omega$ cm mol2 K2 J-2 in heavy-fermion
systems Kadowaki ; Jacko . The value of $R_{\rm W}$ is larger than 1 that is
expected in noninteracting free electron gas. Both large $A/\gamma^{2}$ and
$R_{\rm W}$ suggest strong electron correlations in YRu3Si2.

##  IV Conclusion

In summary, kagome metal YRu3Si2 shows a superconducting transition at
$T_{c}\sim$ 3.0 K. The zero-temperature $\mu_{0}H_{c1}(0)$ and
$\mu_{0}H_{c2}(0)$ is 28.0(3) mT and 0.655(2) T, respectively. The derived
$\kappa_{\rm GL}$ is 1.11(4), confirming that YRu3Si2 is a type-II
superconductor. Moreover, the relatively small ${\Delta C_{\rm ele}}/{\gamma
T_{c}}$ and $\lambda_{\rm e-ph}$ indicates that YRu3Si2 has a weakly or
moderately coupled BCS bulk superconductivity. On the other hand, rather
strong electron correlation effect is identified by the large values of
$A/\gamma^{2}$ and $R_{\rm W}$. This could be related to the existence of
kagome flat band in YRu3Si2, which can result in high density of states near
the $E_{\rm F}$.

##  V Acknowledgements

This work was supported by National Key R&D Program of China (Grant No.
2018YFE0202600), Beijing Natural Science Foundation (Grant No. Z200005),
National Natural Science Foundation of China (Grant No. 11822412 and
11774423), the Fundamental Research Funds for the Central Universities and
Research Funds of Renmin University of China (RUC) (Grant No. 18XNLG14,
19XNLG13, 19XNLG17 and 20XNH062), Outstanding Innovative Talents Cultivation
Funded Programs 2020 of Renmin Univertity of China, and Beijing National
Laboratory for Condensed Matter Physics.

## References

  * (1) L. Balents, Nature 464, 199 (2010).
  * (2) C. Broholm, R. J. Cava, S. A. Kivelson, D. G. Nocera, M. R. Norman, and T. Senthil, Science 367, 6475 (2020).
  * (3) M. P. Shores, E. A. Nytko, B. M. Bartlett, and D. G. Nocera, J. Am. Chem. Soc. 127, 13462 (2005).
  * (4) T. H. Han, J. S. Helton, S. Chu, D. G. Nocera, J. A. Rodriguez-Rivera, C. Broholm, and Y. S. Lee, Nature 492, 406 (2012).
  * (5) L. Ye, M. Kang, J. Liu, F. von Cube, C. R. Wicker, T. Suzuki, C. Jozwiak, A. Bostwick, E. Rotenberg, D. C. Bell, L. Fu, R. Comin, and J. G. Checkelsky, Nature 555, 638 (2018).
  * (6) M. Kang, L. Ye, S. Fang, J.-S. You, A. Levitan, M. Han, J. I. Facio, C. Jozwiak, A. Bostwick, E. Rotenberg, M. K. Chan, R. D. McDonald, D. Graf, K. Kaznatcheev, E. Vescovo, D. C. Bell, E. Kaxiras, J. van den Brink, M. Richter, M. P. Ghimire, J. G. Checkelsky, and R. Comin,, Nat. Mater. 19, 163 (2020).
  * (7) Z. H. Liu, M. Li, Q. Wang, G. W. Wang, C. H. P. Wen, K. Jiang, X. L. Lu, S. C. Yan, Y. B. Huang, D. W. Shen, J.-X. Yin, Z. Q. Wang, Z. P. Yin, H. C. Lei, and S. C. Wang, Nat. Commun. 11, 4002 (2020).
  * (8) M. Kang, S. Fang, L. Ye, H. C. Po, J. Denlinger, C. Jozwiak, A. Bostwick, E. Rotenberg, E. Kaxiras, J. G. Checkelsky, and R. Comin, Nat. Commun. 11, 4004 (2020).
  * (9) S. Nakatsuji, N. Kiyohara, and T. Higo, Nature 527, 212 (2015).
  * (10) A. K. Nayak, J. E. Fischer, Y. Sun, B. Yan, J. Karel, A. C. Komarek, C. Shekhar, N. Kumar, W. Schnelle, J. Kübler, C. Felser, and S. S. P. Parkin, Sci. Adv. 2, e1501870 (2016).
  * (11) Q. Wang, S. S. Sun, X. Zhang, F. Pang, and H. C. Lei, Phys. Rev. B 94, 075135 (2016).
  * (12) E. Liu, Y. Sun, N. Kumar, L. Muechler, A. Sun, L. Jiao, S. Y. Yang, D. Liu, A. Liang, Q. Xu, J. Kroder, V. Süß, H. Borrmann, C. Shekhar, Z. Wang, C. Xi, W. Wang, W. Schnelle, S. Wirth, Y. Chen, S. T. B. Goennenwein, and C. Felser, Nat. Phys. 14, 1125 (2018).
  * (13) Q. Wang, Y. F. Xu, R. Lou, Z. H. Liu, M. Li, Y. B. Huang, D. W. Shen, H. M. Weng, S. C. Wang, and H. C. Lei, Nat. Commun. 9, 3681 (2018).
  * (14) J. X. Yin, S. S. Zhang, H. Li, K. Jiang, G. Chang, B. Zhang, B. Lian, C. Xiang, I. Belopolski, H. Zheng, T. A. Cochran, S. Y. Xu, G. Bian, K. Liu, T. R. Chang, H. Lin, Z. Y. Lu, Z. Wang, S. Jia, W. Wang, and M. Z. Hasan, Nature 562, 91 (2018).
  * (15) J. X. Yin, W. Ma, T. A. Cochran, X. Xu, S. S. Zhang, H. J. Tien, N. Shumiya, G. Cheng, K. Jiang, B. Lian, Z. Song, G. Chang, I. Belopolski, D. Multer, M. Litskevich, Z. J. Cheng, X. P. Yang, B. Swidler, H. Zhou, H. Lin, T. Neupert, Z. Wang, N. Yao, T. R. Chang, S. Jia, and M. Z. Hasan, Nature 583, 533 (2020).
  * (16) B. R. Ortiz, L. C. Gomes, J. R. Morey, M. Winiarski, M. Bordelon, J. S. Mangum, I. W. H. Oswald, J. A. Rodriguez-Rivera, J. R. Neilson, S. D. Wilson, E. Ertekin, T. M. McQueen, and E. S. Toberer, Phys. Rev. Mater. 3, 094407 (2019).
  * (17) B. R. Ortiz, S. M. L. Teicher, Y. Hu, J. L. Zuo, P. M. Sarte, E. C. Schueller, A. M. M. Abeykoon, M. J. Krogstad, S. Rosenkranz, R. Osborn, R. Seshadri, L. Balents, J. F. He, and S. D. Wilson, Phys. Rev. Lett. 125, 247002 (2020).
  * (18) B. R. Ortiz, P. M. Sarte, E. M. Kenney, M. J. Graf, S. M. L. Teicher, R. Seshadri, and S. D. Wilson, Phys. Rev. Mater. 5, 034801 (2021).
  * (19) Q. W. Yin, Z. J. Tu, C. S. Gong, Y. Fu, S. H. Yan, and H. C. Lei, Chin. Phys. Lett. 38, 037403 (2021).
  * (20) W.-H. Ko, P. A. Lee, and X.-G. Wen, Phys. Rev. B 79, 214502 (2009).
  * (21) W.-S. Wang, Z.-Z. Li, Y.-Y. Xiang, and Q.-H. Wang, Phys. Rev. B 87, 115135 (2013).
  * (22) M. L. Kiesel, C. Platt, and R. Thomale, Phys. Rev. Lett. 110, 126405 (2013).
  * (23) I. I. Mazin, H. O. Jeschke, F. Lechermann, H. Lee, M. Fink, R. Thomale, and R. Valentí, Nat. Commun. 5, 4261 (2014).
  * (24) Y.-X. Jiang, J.-X. Yin, M. M. Denner, N. Shumiya, B. R. Ortiz, J. He, X. Liu, S. S. Zhang, G. Chang, I. Belopolski, Q. Zhang, M. S. Hossain, T. A. Cochran, D. Multer, M. Litskevich, Z.-J. Cheng, X. P. Yang, Z. Guguchia, G. Xu, Z. Wang, T. Neupert, S. D. Wilson, and M. Z. Hasan, arXiv: 2012.15709 (2020).
  * (25) H. Chen, H. Yang, B. Hu, Z. Zhao, J. Yuan, Y. Xing, G. Qian, Z. Huang, G. Li, Y. Ye, Q. Yin, C. Gong, Z. Tu, H. Lei, S. Ma, H. Zhang, S. Ni, H. Tan, C. Shen, X. Dong, B. Yan, Z.Wang, and H.-J. Gao, arXiv: 2103.09188 (2021).
  * (26) Ö. Rapp, J. Invarsson, and T. Claeson, Phys. Lett. A 50, 159 (1974).
  * (27) S. S. Sun, K. Liu, and H. C. Lei, J. Phys.: Condens. Matter 28, 085701 (2016).
  * (28) C. S. Gong, Q. Wang, S. H. Wang, and H. C. Lei, J. Phys.: Condens. Matter 32, 295601 (2020).
  * (29) H. C. Ku, G. P. Meisner, F. Acker, and D. C. Johnston, Solid State Commun. 35, 91 (1980).
  * (30) H. Barz, Mater. Res. Bull. 15, 1489 (1980).
  * (31) M. Escorne, A. Mauger, L. C. Gupta, and C. Godart, Phys. Rev. B 49, 12051 (1994).
  * (32) S. Li, B. Zeng, X. G. Wan, J. Tao, F. Han, H. Yang, Z. H. Wang, and H. H. Wen, Phys. Rev. B 84, 214527 (2011).
  * (33) B. X. Li, S. Li, and H. H. Wen, Phys. Rev. B 94, 094523 (2016).
  * (34) C. Mielke, III, Y. Qin, J. X. Yin, H. Nakamura, D. Das, K. Guo, R. Khasanov, J. Chang, Z. Q. Wang, S. Jia, S. Nakatsuji, A. Amato, H. Luetkens, G. Xu, M. Z. Hasan, and Z. Guguchia, Phys. Rev. Mater. 5, 034803 (2021).
  * (35) J. M. Vandenberg and H. Barz, Mater. Res. Bull. 15, 1493 (1980).
  * (36) TOPAS Version 4; Bruker AXS, Karlsruhe, Germany (2007).
  * (37) A. Aharoni, J. Appl. Phys. 83, 3432 (1998).
  * (38) A. F. Ioffe and A. R. Regel, Prog. Semicond. 4, 237 (1961).
  * (39) V. N. Zverev, A. V. Korobenko, G. L. Sun, D. L. Sun, C. T. Lin, and A. V. Boris, JETP Lett. 90, 130 (2009).
  * (40) N. R. Werthamer, E. Helfand, and P. C. Hohenberg, Phys. Rev. 147, 295 (1966).
  * (41) K. Maki, Phys. Rev. 148, 362 (1966).
  * (42) H. D. Yang and J. Y. Lin, J. Phys. Chem. Solids 62, 1861 (2001).
  * (43) W. L. McMillan, Phys. Rev. 167, 331 (1968).
  * (44) K. Kadowaki and S. B. Woods, Solid State Commun. 58, 507 (1986).
  * (45) K. G. Wilson, Rev. Mod. Phys. 47, 773 (1975).
  * (46) A. C. Jacko, J. O. Fjærestad, and B. J. Powell, Nat. Phys. 5, 422 (2009).


# QuickSRNet: Plain Single-Image Super-Resolution Architecture
for Faster Inference on Mobile Platforms

Guillaume Berger Manik Dhingra111 Antoine Mercier Yashesh Savani Sunny Panchal
Fatih Porikli
Qualcomm AI Research
{guilberg, manidhin, amercier, ysavani, sunnpanc<EMAIL_ADDRESS>Contributed equally.Qualcomm AI Research is an initiative of Qualcomm
Technologies, Inc.

###### Abstract

In this work, we present QuickSRNet, an efficient super-resolution
architecture for real-time applications on mobile platforms. Super-resolution
clarifies, sharpens, and upscales an image to higher resolution. Applications
such as gaming and video playback along with the ever-improving display
capabilities of TVs, smartphones, and VR headsets are driving the need for
efficient upscaling solutions. While existing deep learning-based super-
resolution approaches achieve impressive results in terms of visual quality,
enabling real-time DL-based super-resolution on mobile devices with compute,
thermal, and power constraints is challenging. To address these challenges, we
propose QuickSRNet, a simple yet effective architecture that provides better
accuracy-to-latency trade-offs than existing neural architectures for single-
image super-resolution. We present training tricks to speed up existing
residual-based super-resolution architectures while maintaining robustness to
quantization. Our proposed architecture produces 1080p outputs via 2$\times$
upscaling in 2.2 ms on a modern smartphone, making it ideal for high-fps real-
time applications.

##  1 Introduction

Single-image super-resolution (SR) refers to a family of techniques that
recover a high-resolution (HR) image $I_{HR}$ from its low-resolution (LR)
counterpart $I_{LR}$. In recent years, deep learning (DL) based approaches
have become increasingly popular in the field [9, 10, 32, 18, 24, 21, 23, 6,
31], producing impressive results compared to interpolation-based techniques
and hand-engineered heuristics (see Figure 2). However, most existing DL-based
super-resolution solutions are computationally intensive and not suitable for
real-time applications requiring interactive frame rates, such as mobile
gaming. While DL-based super-resolution has been successfully applied to
gaming on high-end GPU desktops [22, 7], neural approaches are still
impractical for mobile gaming due to their high latency and computational
costs. For example, a DL-based architecture such as EDSR [21] takes $75$ ms to
upscale a $540$p image to $1080$p on a state-of-the-art mobile AI accelerator.
This has driven the need for efficient DL-based super-resolution solutions [2,
12, 5, 35, 11] that can be used in real-time applications such as video
gaming, where responsiveness and higher frame rates are essential.

Figure 1: Accuracy-to-latency trade-offs of QuickSRNet (ours) against existing
SISR architectures. We report accuracy after 8-bit quantization and measure
latency on a state-of-the-art mobile AI accelerator.

(a) LR

(b) Interpolation

(c) Algorithmic approach

(d) ML-based approach

Figure 2: Side-by-side comparison of various upscaling approaches: (a) Low-
resolution crop, (b) Bicubic interpolation, (c) Non-ML baseline: FSR1.0 [13],
(d) ML baseline: EDSR [21].

In this work, rather than trying to achieve the state-of-the-art PSNR or SSIM
scores on standard super-resolution benchmarks, we aim to develop efficient
architectures that are suitable for high-fps real-time applications on mobile
devices. To this end, we propose QuickSRNet, a simple single-image super-
resolution neural network that obtains better accuracy-to-latency trade-offs
than existing efficient SR architectures. In particular, we make the following
key contributions:

  * •

We demonstrate that simple architecture designs can be highly effective in
terms of both accuracy and on-device performance.

  * •

We compare a wide variety of architectures in terms of on-device latency,
measured on a device with Snapdragon® 8 Gen 1 Mobile Platform, instead of
FLOPS count, which is not a reliable indicator of on-device performance [16].

  * •

We measure accuracy _after_ 8-bit quantization, a necessary step for better
efficiency on mobile platforms, and describe architectural tricks that improve
robustness to quantization.

  * •

We apply our proposed architecture to a real-world use-case (video gaming) and
compare its visual quality against that of a well-known industrial non-ML
based approach (AMD’s FidelityFX Super Resolution (FSR1.0) algorithm [13]).

  * •

We describe an approach to perform $1.5\times$ upscaling, a setting that is
occasionally used in gaming and XR use-cases but not trivially supported by SR
architectures whose upscaling step is based on a sub-pixel convolution [32].

Figure 3: QuickSRNet architecture. We use the convention QuickSRNet-f$X$-m$Y$
to refer to the version of the architecture that has $Y$ intermediate conv
layers and $X$ feature channels. We use dotted lines to illustrate that the
conv layers are initialized using an identity initialization scheme. In
practice, these skip connections are incorporated into the weights of the
corresponding conv module. $p$ and $ri$ stand for “partial” and “repeat-
interleaving” (see section 3.2 for more details).

##  2 Related work

Several efficient SR architectures have been proposed recently. Overall, these
architectures share many characteristics with the earlier work by [10] and
[32] on FSRCNN and ESPCN respectively: they are usually fully convolutional,
use a relatively small number of layers and channels, all layers run at the
input resolution and the final output is mapped to higher resolution using a
subpixel convolution111In the rest of the paper, we will use the term depth-
to-space operation. In practice, a subpixel convolution amounts to performing
a regular convolution producing $3\times S^{2}$ low-resolution channels, where
$S$ is the scaling factor, followed by a depth-to-space operation to map to
higher resolution.. Compared to these baselines, more recent approaches have
incorporated the following changes:

#### XLSR

[2] uses grouped convolutions to reduce the computational footprint of the
architecture and “clipped” ReLU activations to improve robustness to
quantization.

#### ABPN

[12] employs a VGG-like convnet [34] (i.e. consisting of only $3\times 3$
Conv-ReLU blocks) with an “anchor-based” input-to-output residual connection.
This “anchor-based” connection adds a channel-wise nearest-neighbor upscaled
version of the input to the output before the final depth-to-space operation.
We confirmed that this channel-wise implementation runs faster on our
profiling device than the more common approach of adding the spatially-
upsampled input directly to the output. Thus, we follow the same strategy to
implement input-to-output residual connections in all our experiments.

#### SESR

[5] leverages linear over-parameterized residual modules which are collapsed
into regular convolutions during inference for improved on-device performance.
Other modifications include the use of long residual connections.

#### RepSR

[35] investigates training VGG-like super-resolution architectures. Like ABPN,
their convnet is equipped with a nearest-neighbor upsampling-based input-to-
output connection. Similar to SESR, they find that using over-parameterized
networks during training can boost accuracy. They propose a training scheme
for using Batch Normalization (BN) layers [19] without introducing artifacts
in flat regions of the image, a typical side effect of BN when employed for
super-resolution. At test time, the over-parameterized, BN-equipped network is
collapsed into a simpler, more efficient network.

#### Residual learning for super-resolution

Many SR architectures utilize a long skip connection which adds an upscaled
version of the input $U(I_{LR})$ directly to the output. Efficient
architectures (like [12, 35]) will often implement $U$ as nearest-neighbour
interpolation. During training, SR architectures equipped with this technique
are implicitly optimized to produce a residual $R=I_{HR}-U(I_{LR})$. One
benefit is that the network produces reasonable outputs right after
initialization which stabilizes training. Additionally, the input-to-output
connection makes the architecture significantly more robust to quantization,
as discussed in the next section.

##  3 Methodology

This section contains a detailed description of QuickSRNet as well as
implementation details. The process for developing our proposed SR
architecture began with preliminary experiments, which we present in the next
paragraph.

###  3.1 On the impact of removing the input-to-output residual connection

Architecture | PSNR (dB) | Latency (ms)
---|---|---
FP16 | INT8
ABPN |  31.84 (baseline) |  31.80 (baseline) |  2.17 (baseline)
Res.-free ABPN |  31.75 ($-$0.09) |  31.50 ($-$0.30) |  1.42 ($-$35%)
| QSRNet-Med
---
31.82 ($-$0.02) |  31.77 ($-$0.03) |  1.42 ($-$35%)
Table 1: On the accuracy and latency impacts of removing the input-to-output
residual connection from the ABPN architecture. We report PSNR numbers
obtained on BSD100 via $2\times$ upscaling. Latency numbers were obtained on a
device with Snapdragon 8 Gen 1 using an input resolution of $512\times 512$.

VGG-style architectures such as ABPN [12] or RepSR [35] are already well-
optimized, so it is unclear how much faster they can be made on mobile AI
accelerators. Of course, reducing the number of layers and channels, or
replacing $3\times 3$ kernels with $1\times 1$ kernels, can improve speed at
the cost of accuracy. Instead, our experiments investigate how to effectively
remove the input-to-output residual connection without affecting accuracy.

As observed by [2, 12], long residual connections can have a large impact on
the efficiency of super-resolution architectures, particularly on memory-
limited platforms such as smartphones or VR headsets. To confirm this, we
trained and profiled a residual-free ABPN variant and found that removing the
input-to-output residual connection improves latency by $35\%$. However, this
modification resulted in a marginally lower accuracy and more importantly
reduced robustness to quantization, as can be seen in Table 1. To address
this, we propose QuickSRNet, a residual-free architecture which is robust to
quantization.

###  3.2 QuickSRNet

HR SESR-M7 ABPN EDSR Bicubic
empty Ours-Small
(1.5$\times$ faster) Ours-Medium
(1.3$\times$ faster) Ours-Large
(1.8$\times$ faster)

Figure 4: 4$\times$ Super-resolution results on a crop of an image from
Urban100 dataset (with % latency improvements of our proposed models over
existing networks)

Our architecture, QuickSRNet, follows a VGG-like structure with no input-to-
output residual connection (see Figure 3). This architecture is denoted by
$m$, the number of intermediate convolutional blocks, and $f$, the number of
feature channels in those intermediate layers. To increase robustness to
quantization, we use a residual learning-motivated initialization scheme along
with clipped ReLU activations:

#### Identity initialization

We utilize an intuitive initialization technique where each intermediate
convolutional layer simulates a localized skip connection:

$y=W\circledast x+x$ (1)

where $\circledast$ is the discrete convolution operator and $W$ refers to the
kernel weights. In practice, we collapse the skip connection into the conv
module: $y=(W+I)\circledast x=\hat{W}\circledast x$, where $\hat{W}$ are the
modified weights after collapse and $I$ is the identity of discrete
convolution operators. In this case, collapsing amounts to adding a diagonal
of ones to the randomly initialized kernel sliced at the spatial center:
$\forall i,W[i,i,c_{x},c_{y}]\mathrel{+}=1$ (with $c_{x}=c_{y}=1$ assuming a
$3\times 3$ kernel). This approach is akin to identity initialization [14, 38,
3, 39] and related to the over-parameterized networks used in [36, 8, 5, 35],
except we collapse before training, during initialization.

Eq. 1 only works if $x$ and $y$ have the same dimensions and is therefore not
directly applicable to the first and last layer of the architecture as these
layers respectively change the number of channels from $3$ to $f$ and $f$ to
$3\times S^{2}$, where $S$ is the scaling factor. For these layers, we modify
the initialization scheme as follows:

  * •

Partial identity initialization: the $3$-channel input to the first
convolutional module are added to the first $3$ output channels and the other
$f-3$ output channels are left unchanged.

$y_{i}=\begin{cases}(W\circledast x)_{i}+x_{i},&\text{if }0\leq i<3\\\
(W\circledast x)_{i},&\text{otherwise}\end{cases}$ (2)

  * •

Repeat-interleaving identity initialization: the first $3$ input channels to
the final convolutional module are repeat-interleaved $S^{2}$ times and added
to the output.

$y_{i}=(W\circledast x)_{i}+x_{round\left(\frac{i}{s^{2}}\right)}$ (3)

Similarly to Eq. 1, the skip connections described in Eqs. 2 and 3 are
incorporated into the corresponding convolutional module by adding ones to the
kernel weights at the appropriate location. Intuitively, this initialization
technique makes the input image propagate well throughout the entire network.
The repeat-interleaving scheme used to initialize the final layer mimics the
nearest-neighbour upscaling typically performed in the input-to-output
connection of existing residual architectures.

#### ReLU1

In addition to identity initialization, we found that clipping ReLU
activations between $0$ and $1$ improves robustness to quantization. Compared
to XLSR [2], we use ReLU1s throughout the entire network as opposed to just
the final layer. Note that for this approach to work well with our id-
initialized architecture, it is important to scale input pixels between $0$
and $1$ (centering around $0$ would cause roughly half the pixels propagated
by the first id-initialized conv to be zeroed out).

####

Our experimental results (section 4) show that combining identity
initialization and ReLU1 activations significantly improve robustness to
quantization.

FP16 FP16 FP16 FP16 INT8 INT8 INT8 INT8

Figure 5: Visual comparison of 4$\times$ super-resolved images from DIV2K
produced by QuickSRNet-Medium before and after quantization

###  3.3 Implementation details

#### Baselines

We compare QuickSRNet against the following architectures: FSRCNN [10], ESPCN
[32], XLSR [2], SESR [5], ABPN [12], ERFDN [23] and EDSR [21]. Note that,
rather than reporting PSNR and SSIM scores from the original papers, we re-
implemented, trained and quantized all existing baselines from scratch. As a
result, all models shared most hyper-parameters (batch-size, losses,
optimizer, etc.), including the data loading/augmentation pipeline. We did
however tweak the learning rate for each architecture independently. In some
cases, our re-implementation deviates slightly from the original architecture
when it includes operations that are not supported on the device used for
profiling. For example, we replaced the parametric ReLUs [15] used in SESR and
FSRCNN to regular ReLUs. Despite these minor modifications, we were usually
able to reproduce PSNR and SSIM scores reported in the original papers.

#### Training details

For most experiments, we train the models on the $800$ training images from
the DIV2K dataset [1] and evaluate them on standard SR testsets: Set5 [4],
Set14 [37], BSD100 [25], and Urban100 [17]. We preprocess input and target
images by scaling RGB values between $0$ and $1$. For data augmentation, we
use random cropping, flipping and rotation. The models are trained for 1
million iterations with a batch size of $32$. We use an L1 loss and the Adam
optimizer [20] with hyper-parameters $\epsilon=10^{-8}$ and
$\beta=(0.9,0.999)$. For the learning rate, we found that using an initial
value of $5\times 10^{-4}$ and decaying it by a factor of $0.5$ every $200$K
iterations is a strategy that works well for most architectures.

#### 8-bit quantization

We use the AIMET library [33] to perform model quantization [28] and compute
post-quantization accuracy metrics222Additionally, we confirmed accuracy
numbers on target for a subset of the models and typically found that the
simulated numbers produced by AIMET to be within a $0.02$ range from the
actual numbers obtained on target.. Both weights and activations are quantized
to 8-bit integers (W8A8 setup). We experimented with both Post-Training
Quantization (PTQ) techniques and Quantization Aware Training (QAT). When we
use QAT, we re-initialize the optimizer with a very small learning rate
(usually $4\times 10^{-6}$).

#### On-device profiling

We profile the models on the Hexagon Processor of a device with Snapdragon 8
Gen 1 and report the average latency obtained on $100$ inputs of spatial
resolution $512\times 512$. Before profiling, the model is converted from
PyTorch [29] to ONNX. Please see the appendix for more details about the model
conversion steps.

QuickSRNet specs | $\bm{2\times}$ | $\bm{3\times}$ | $\bm{4\times}$ | Latency
---|---|---|---|---
FP16 | INT8 | FP16 | INT8 | FP16 | INT8 | Measurements (ms)
f32 - m1 | 31.43 | 31.38 | 28.41 | 28.38 | 26.94 | 26.91 |  0.99 ($-$22%)
f32 - m2 (small) | 31.61 | 31.58 | 28.57 | 28.55 | 27.07 | 27.06 |  1.14 ($-$35%)
f32 - m3 | 31.72 | 31.63 | 28.67 | 28.63 | 27.16 | 27.12 |  1.21 ($-$34%)
f32 - m5 (medium) | 31.82 | 31.77 | 28.75 | 28.72 | 27.24 | 27.21 |  1.42 ($-$35%)
f32 - m7 | 31.88 | 31.81 | 28.81 | 28.76 | 27.30 | 27.27 |  1.74 ($-$30%)
f32 - m11 | 31.95 | 31.80 | 28.86 | 28.80 | 27.35 | 27.29 |  2.38 ($-$22%)
f64 - m11 (large) | 32.08 | 31.97 | 28.98 | 28.93 | 27.47 | 27.43 |  7.63 ($-$7%)
Table 2: PSNRs (dB) and latencies (ms) of various QuickSRNet configurations ($f$ : number of feature channels, $m$ : number of convolutional blocks in the network). We report PSNR numbers obtained before and after quantization. We also report latency measurements on a $512\times 512$ input, obtained on a device with Snapdragon 8 Gen 1, and gains introduced by not using an input-to-output residual connection. Existing Models | $\bm{2\times}$ | $\bm{3\times}$ | $\bm{4\times}$ | Latency
---|---|---|---|---
FP16 | INT8 | FP16 | INT8 | FP16 | INT8 | Measurements (ms)
XLSR | 31.62 | 31.32 | 28.59 | 28.31 | 27.09 | 26.82 | 1.59
ESPCN | 31.37 | 30.19 | 28.35 | 27.86 | 26.87 | 26.44 | 1.83
SESR-M3 | 31.57 | 31.40 | 28.52 | 28.47 | 27.02 | 26.97 | 2.02
ABPN | 31.80 | 31.74 | 28.73 | 28.70 | 27.22 | 27.20 | 2.09
SESR-M5 | 31.68 | 31.53 | 28.63 | 28.56 | 27.11 | 27.05 | 2.23
SESR-M7 | 31.76 | 31.67 | 28.68 | 28.61 | 27.16 | 27.07 | 2.33
SESR-M11 | 31.84 | 31.74 | 28.77 | 28.60 | 27.25 | 27.18 | 3.06
FSRCNN | 31.50 | 31.22 | 28.49 | 28.32 | 26.99 | 26.89 | 3.45
SESR-XL | 32.02 | 31.94 | 28.91 | 28.85 | 27.39 | 27.35 | 3.75
ERFDN | 32.20 | 32.06 | 29.08 | 28.98 | 27.57 | 27.48 | 19.50
EDSR | 32.21 | 32.08 | 29.04 | 28.89 | 27.61 | 27.53 | 37.95
Table 3: PSNRs (dB) and latencies (ms) of existing SISR solutions on BSD100.
Please note that we re-implemented, trained, and quantized all architectures
from scratch. Latency numbers were measured on a device with Snapdragon 8 Gen
1, using a $512\times 512$ input.

ABPN

QSRNet-Medium

Figure 6: Visual artifacts by ABPN vs QuickSRNet-Medium on $4\times$-upscaled images from Urban100. More examples can be found in the supplementary material. Figure 7: Ablation study comparing the post-quantization PSNR drop from FP16 when removing identity initialization and/or ReLU1 activations from the architecture design Specification | Post-training | No optimizations | QAT | Per-channel QAT | Per-channel Adaround
---|---|---|---|---|---
FP16 | INT8 | INT8 | INT8 | INT8
QuickSRNet-Small | 31.61 |  30.81 ($-$0.80) |  31.34 ($-$0.27) | 31.57 ($-$0.04) |  31.56 ($-$0.05)
QuickSRNet-Medium | 31.82 |  30.74 ($-$1.08) |  31.61 ($-$0.21) |  31.75 ($-$0.07) | 31.77 ($-$0.05)
QuickSRNet-Large | 32.07 |  31.37 ($-$0.70) |  31.90 ($-$0.10) |  31.97 ($-$0.10) | 31.99 ($-$0.08)
Table 4: Impact of various quantization techniques on accuracy. Both
activations and weights are quantized to 8-bit integers using per-tensor
quantization for activations and either per-tensor or per-channel quantization
for weights.

(a) LR

(b) Bicubic

(c) FSR1.0

(d) QSRNet-Small

Figure 8: SISR ($2\times$) for Gaming: (a) Low-res crop, (b) Bicubic upsampling, (c) FSR1.0, and (d) QuickSRNet-Small (ours) Method | PSNR | SSIM
---|---|---
Bicubic | 28.88 | 0.8683
FSR1.0 | 29.01 | 0.8707
QuickSRNet-Small | 29.71 | 0.8806
Table 5: PSNR/SSIM scores for different $2\times$ single-image super-
resolution solutions for gaming

##  4 Experimental results

In this section, we compare QuickSRNet against existing SR architectures in
terms of accuracy-to-latency trade-offs and demonstrate the effectiveness of
our training tricks to improve robustness to quantization through ablation
studies.

#### Scaling laws of QuickSRNet

We experimented with several architecture specifications, varying the number
of conv modules $m$ and the number of feature channels $f$. PSNR and SSIM
scores on the BSD100 dataset obtained with each specification and a scaling
factor of $2$ can be found in Table 2. As expected, larger/wider networks
obtain higher PSNR/SSIM scores. The measured latency for each specification is
reported on the last column and we indicate for each architecture the latency
improvement introduced by removing the input-to-output connection. In the rest
of the paper, we only use a subset of these model configurations: QuickSRNet-
small (a.k.a. QuickSRNet-f32-m2), QuickSRNet-medium (a.k.a.
QuickSRNet-f32-m5), and QuickSRNet-large (a.k.a. QuickSRNet-f64-m11). As can
be seen in Figure 1 and Tables 2 and 3, these $3$ variants obtain similar
accuracy scores in contrast to SESR-M7, ABPN and EDSR respectively while being
significantly faster. Please refer to Figure 4 for a comparison of
QuickSRNet-{small, medium, large} versus SESR-M7, ABPN and EDSR in terms of
image quality and latency improvement. More quantitative results and side-by-
side comparisons for $2\times$, $3\times$ and $4\times$ upscaling are
available in the supplementary materials (see Figure 12, 12 and 12).

#### W8A8 quantization

For all our experiments, we quantize both, model weights and activations, to
8-bit integers. Without any optimizations, we observe a significant drop after
quantization (see Table 4). Although finetuning the quantized weights via QAT
can recover some of this drop, we found per-channel quantization essential to
yield optimal results. Furthermore, we experimented with several post-training
quantization methods, including: cross-layer equalization (CLE) [27], bias
correction (BC) [27], and adaptive rounding (AdaRound) [26], and found
AdaRound to obtain comparable performance to per-channel QAT, outperforming
the other PTQ approaches. CLE did not work well in our experiments, most
likely because it skews the activation values outside the ReLU1 range. In an
attempt to further improve post-quantization accuracy, we tried finetuning the
per-channel Adarounded weights using QAT but this did not improve post-
quantization accuracy.

#### Robustness to quantization

Overall, QuickSRNet quantizes well to W8A8. As can be seen in Figure 5, images
produced by the quantized model are indistinguishable from their full-
precision counterparts. In Figure 7, we visualize the drop in PSNR post
quantization and show the benefits of combining identity initialization and
ReLU1 activations. Regardless of the model size, removing one or both of these
ingredients from the model design results in a significantly worse accuracy
after quantization.

#### Less prone to block artifacts

Our experiments show that architectures with a nearest-neighbour upsampling
skip connection tend to produce outputs with block-like artifacts of size
$S\times S$. Interestingly, our residual-free architecture seems less prone to
this issue and produces more perceptually pleasing results. A visual
comparison of such artifacts can be seen in Figure 6.

Target resolution | 540p | 720p | 1080p | 1440p | 2160p
---|---|---|---|---|---
Latency (ms) | 0.69 | 0.95 | 2.24 | 4.25 | 8.15
Table 6: QuickSRNet-Small latency (ms) running at different target resolutions
on a device with Snapdragon 8 Gen 1.

(a) Naïve baseline

(b) Proposed approach

Figure 9: Two different architecture modifications to implement $1.5\times$
upscaling: (a) Naïve approach, where we repurpose a $3\times$ architecture by
adding an average pooling layer on top, (b) Our approach, where we halve the
resolution inside the network and map to target resolution using a $3\times$
subpixel conv.

##  5 DL-based SISR for mobile gaming

A real-world application of efficient super-resolution is video gaming. While
DL-based super-resolution (or supersampling) has already been commericalized
on high-end gaming desktops [22, 7], these solutions are not supported on
mobile platforms yet. One specificity of gaming content is that rasterized
images are significantly more aliased than natural images. Nevertheless, we
find QuickSRNet-Small to work well on this domain, with no changes needed
apart from re-training it on gaming data333We plan to release this dataset
soon as part of an upcoming publication focused on super-resolution for
gaming.. Figure 8 shows some results obtained by QuickSRNet-Small when applied
to gaming content. We compare our results against non-ML based single-frame
upscaling approaches, including an FSR1.0 baseline [13] which was specifically
designed for this use case. Overall, we find that QuickSRNet-Small produces
better-looking images compared to the other baselines. The visual benefits
also translate into PSNR and SSIM gains, as can be seen in Figure 5.

QuickSRNet | Bicubic | Naïve | Proposed
---|---|---|---
Specification | Baseline | Approach
Small | 32.47 | 34.71 | 34.89
Medium | 34.87 | 35.13
Large | 35.18 | 35.47
Table 7: PSNRs (dB) evaluated after quantization on BSD100 dataset via
$1.5\times$ upscaling

In terms of latency, Table 6 shows QuickSRNet-Small latency measurements at
various target resolutions, from $540$p to $4$k. In the future, we would like
to extend our architecture to the multi-frame scenario444FSR2.0, DLSS2, XeSS
are all multi-frame solutions., which would most likely lead to better
quality.

###  5.1 QuickSRNet $\bm{1.5\times}$

Standard super-resolution datasets are usually limited to $2\times$, $3\times$
or $4\times$ upscaling and non-integer scaling factors are rarely explored. On
the other hand, $1.5\times$ upscaling is often proposed in VR and gaming
applications555Both DLSS and FSR support $1.5\times$ via their “Quality”
mode.. In this section, we describe an approach to perform $1.5\times$
upscaling, a setting that is not trivially supported by most efficient SR
architectures as non-integer scaling factors are not compatible with the final
sub-pixel convolution.

####  $\bm{3\times}$ upscaling followed by $\bm{2\times}$ downscaling baseline

A naïve approach to $1.5\times$ upscaling consists in downscaling by a factor
$2$ the output of a $3\times$ SR model. This can be achieved by adding a
$2\times 2$ average pooling layer at the end of the architecture.

#### Proposed $\bm{1.5\times}$ upscaling approach

Instead, we propose to halve the resolution inside the network using a space-
to-depth operation with a block-size of $2$ which we then map to target
resolution using a $3\times$ subpixel convolution. To compensate for the
$4\times$ increase of channels due to the space-to-depth operation, we
implement the subpixel convolution using a $1\times 1$ kernel.

####

Figure 9 shows the two considered $1.5\times$ architecture heads. As can be
seen in Table 7, the proposed approach significantly outperforms the naïve
$3\times$ upscaling followed by $2\times$ downscaling baseline. Visual
comparisons can be found in the appendix.

##  6 Conclusion

####

In this study, we propose QuickSRNet, an efficient super-resolution
architecture for mobile platforms. We have thoroughly analyzed the performance
of our models and existing ones, systematically checking accuracy after
quantization and profiling latency on a mobile device. Our experiments have
shown that QuickSRNet is well suited for real-time applications on mobile
devices due to its high speed and good accuracy. We have also demonstrated the
effectiveness of our solution on a real world use case (mobile gaming) and
believe that our training tricks to improve robustness to quantization are
applicable to other works. We have released the implementation and pretrained
weights (including quantized weights) of QuickSRNet models as part of the
AIMET model zoo666For QuickSRNet-large, the released version of the model
includes the input-to-output residual connection as this leads to slightly
higher accuracy and the latency improvement (-7%) is minimal for larger
architectures.. To further improve the on-device efficiency of QuickSRNet, we
plan to investigate lower precision models (e.g. W4A8), which is a promising
direction [30]. We believe that QuickSRNet provides a practical solution for
applications that require real-time super-resolution capabilities.

## References

  * [1] Eirikur Agustsson and Radu Timofte.  Ntire 2017 challenge on single image super-resolution: Dataset and study.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 126–135, 2017.
  * [2] Mustafa Ayazoglu.  Extremely lightweight quantization robust real-time single-image super resolution for mobile devices.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2472–2479, 2021.
  * [3] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley.  Rezero is all you need: Fast convergence at large depth.  In Uncertainty in Artificial Intelligence, pages 1352–1361. PMLR, 2021.
  * [4] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel.  Low-complexity single-image super-resolution based on nonnegative neighbor embedding.  2012\.
  * [5] Kartikeya Bhardwaj, Milos Milosavljevic, Liam O’Neil, Dibakar Gope, Ramon Matas, Alex Chalfin, Naveen Suda, Lingchuan Meng, and Danny Loh.  Collapsible linear blocks for super-efficient super resolution.  Proceedings of Machine Learning and Systems, 4:529–547, 2022.
  * [6] X Chen, X Wang, J Zhou, and C Dong.  Activating more pixels in image super-resolution transformer. arxiv 2022\.  arXiv preprint arXiv:2205.04437.
  * [7] Hisham Chowdhury, Rense Robert Kawiak, Gabriel Ferreira de Boer, and Lucas Xavier.  Intel xess-an ai based super sampling solution for real-time rendering.(2022).  In Game Developers Conference, 2022.
  * [8] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun.  Repvgg: Making vgg-style convnets great again.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13733–13742, 2021.
  * [9] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  IEEE transactions on pattern analysis and machine intelligence, 38(2):295–307, 2015.
  * [10] Chao Dong, Chen Change Loy, and Xiaoou Tang.  Accelerating the super-resolution convolutional neural network.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 391–407. Springer, 2016.
  * [11] Tingxing Tim Dong, Hao Yan, Mayank Parasar, and Raun Krisch.  Rendersr: A lightweight super-resolution model for mobile gaming upscaling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3087–3095, 2022.
  * [12] Zongcai Du, Jie Liu, Jie Tang, and Gangshan Wu.  Anchor-based plain net for mobile image super-resolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2494–2502, 2021.
  * [13] GPUOpen-Effects.  Fidelityfx-fsr2/readme.md at master · gpuopen-effects/fidelityfx-fsr2, Oct 2022.
  * [14] Moritz Hardt and Tengyu Ma.  Identity matters in deep learning.  arXiv preprint arXiv:1611.04231, 2016.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  * [16] Lennart Heim, Andreas Biri, Zhongnan Qu, and Lothar Thiele.  Measuring what really matters: Optimizing neural networks for tinyml.  arXiv preprint arXiv:2104.10645, 2021.
  * [17] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja.  Single image super-resolution from transformed self-exemplars.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5197–5206, 2015.
  * [18] Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang.  Lightweight image super-resolution with information multi-distillation network.  In Proceedings of the 27th acm international conference on multimedia, pages 2024–2032, 2019.
  * [19] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. pmlr, 2015.
  * [20] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [21] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136–144, 2017.
  * [22] Edward (Shiqiu) Liu, Robert Pottorff, Guilin Liu, Karan Sapra, Jon Barker, David Tarjan, Lei Yang, Kevin Shih, Marco Salvi, Andrew Tao, and Bryan Catanzaro.  Dlss 2.0 – image reconstruction for real-time rendering with deep learning.  In Game Developers Conference 2020, GPU Technology Conference 2020, 2020.
  * [23] Jie Liu, Jie Tang, and Gangshan Wu.  Residual feature distillation network for lightweight image super-resolution.  In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 41–55. Springer, 2020.
  * [24] Zhi-Song Liu, Li-Wen Wang, Chu-Tak Li, and Wan-Chi Siu.  Hierarchical back projection network for image super-resolution.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0–0, 2019.
  * [25] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416–423. IEEE, 2001.
  * [26] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort.  Up or down? adaptive rounding for post-training quantization.  In International Conference on Machine Learning, pages 7197–7206. PMLR, 2020.
  * [27] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling.  Data-free quantization through weight equalization and bias correction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325–1334, 2019.
  * [28] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort.  A white paper on neural network quantization.  arXiv preprint arXiv:2106.08295, 2021.
  * [29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [30] Qualcomm.  Efficient real-time int4 4k super-resolution on mobile.  In NeurIPS Demo Track, 2022.
  * [31] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  arXiv:2104.07636, 2021.
  * [32] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874–1883, 2016.
  * [33] Sangeetha Siddegowda, Marios Fournarakis, Markus Nagel, Tijmen Blankevoort, Chirag Patel, and Abhijit Khobare.  Neural network quantization with ai model efficiency toolkit (aimet).  arXiv preprint arXiv:2201.08442, 2022.
  * [34] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [35] Xintao Wang, Chao Dong, and Ying Shan.  Repsr: Training efficient vgg-style super-resolution networks with structural re-parameterization and batch normalization.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 2556–2564, 2022.
  * [36] Sergey Zagoruyko and Nikos Komodakis.  Diracnets: Training very deep neural networks without skip-connections.  arXiv preprint arXiv:1706.00388, 2017.
  * [37] Roman Zeyde, Michael Elad, and Matan Protter.  On single image scale-up using sparse-representations.  In Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7, pages 711–730. Springer, 2012.
  * [38] Hongyi Zhang, Yann N Dauphin, and Tengyu Ma.  Fixup initialization: Residual learning without normalization.  arXiv preprint arXiv:1901.09321, 2019.
  * [39] Jiawei Zhao, Florian Schäfer, and Anima Anandkumar.  Zero initialization: Initializing neural networks with only zeros and ones.  arXiv preprint arXiv:2110.12661, 2021.

## Supplementary Material

HR SESR-M7 ABPN EDSR Bicubic Ours-Small Ours-Medium Ours-Large

Figure 10: Visual comparison of $2\times$ super-resolution results by
QuickSRNet and existing solutions on Urban100 images.

HR SESR-M7 ABPN EDSR Bicubic Ours-Small Ours-Medium Ours-Large

Figure 11: Visual comparison of $3\times$ super-resolution results by
QuickSRNet and existing solutions on Urban100 images.

HR SESR-M7 ABPN EDSR Bicubic Ours-Small Ours-Medium Ours-Large

Figure 12: Visual comparison of $4\times$ super-resolution results by
QuickSRNet and existing solutions on DIV2K images.

ABPN

QSRNet-Medium

ABPN

QSRNet-Medium

Figure 13: More examples of visual artifacts by ABPN vs QuickSRNet-Medium
($4\times$) on Urban 100 images.

(a) LR

(b) Bicubic

(c) FSR1.0

(d) QSRNet-Small

Figure 14: SISR ($2\times$) for Gaming: (a) Low-res crop, (b) Bicubic
upsampling, (c) FSR1.0, and (d) QuickSRNet-Small (ours).

Table 8: QuickSRNet PSNRs (dB) evaluated for different scaling factors ($2\times$, $3\times$, and $4\times$) on benchmark SISR datasets before and after quantization Scaling | QuickSRNet Specification | Set5 | Set14 | BSD100 | Urban100
---|---|---|---|---|---
Factor | FP16 | INT8 | FP16 | INT8 | FP16 | INT8 | FP16 | INT8
2$\times$ | f32 - m1 | 36.83 | 36.67 | 32.35 | 32.28 | 31.43 | 31.38 | 29.66 | 29.61
f32 - m2 (small) | 37.12 | 36.97 | 32.57 | 32.53 | 31.61 | 31.58 | 30.15 | 30.10
f32 - m3 | 37.30 | 37.06 | 32.72 | 32.57 | 31.72 | 31.63 | 30.43 | 30.30
f32 - m5 (medium) | 37.39 | 37.22 | 32.82 | 32.75 | 31.82 | 31.77 | 30.75 | 30.66
f32 - m7 | 37.51 | 37.27 | 32.95 | 32.84 | 31.88 | 31.81 | 30.93 | 30.84
f32 - m11 | 37.59 | 37.19 | 33.00 | 32.86 | 31.95 | 31.80 | 31.14 | 30.91
f64 - m11 (large) | 37.87 | 37.61 | 33.29 | 33.18 | 32.12 | 32.04 | 31.74 | 31.64
3$\times$ | f32 - m1 | 32.75 | 32.69 | 29.08 | 29.05 | 28.41 | 28.38 | 26.19 | 26.16
f32 - m2 (small) | 33.10 | 33.03 | 29.29 | 29.25 | 28.57 | 28.55 | 26.53 | 26.51
f32 - m3 | 33.33 | 33.25 | 29.39 | 29.35 | 28.67 | 28.63 | 26.77 | 26.72
f32 - m5 (medium) | 33.58 | 33.49 | 29.49 | 29.46 | 28.75 | 28.72 | 27.02 | 26.99
f32 - m7 | 33.69 | 33.53 | 29.60 | 29.52 | 28.81 | 28.76 | 27.16 | 27.10
f32 - m11 | 33.81 | 33.63 | 29.68 | 29.60 | 28.86 | 28.80 | 27.35 | 27.27
f64 - m11 (large) | 34.14 | 34.01 | 29.88 | 29.82 | 29.02 | 28.98 | 27.81 | 27.76
4$\times$ | f32 - m1 | 30.48 | 30.42 | 27.31 | 27.27 | 26.94 | 26.91 | 24.47 | 24.45
f32 - m2 (small) | 30.84 | 30.82 | 27.55 | 27.54 | 27.07 | 27.06 | 24.74 | 24.74
f32 - m3 | 31.04 | 30.95 | 27.65 | 27.58 | 27.16 | 27.12 | 24.90 | 24.86
f32 - m5 (medium) | 31.27 | 31.21 | 27.79 | 27.76 | 27.24 | 27.21 | 25.08 | 25.06
f32 - m7 | 31.39 | 31.29 | 27.83 | 27.80 | 27.30 | 27.27 | 25.22 | 25.18
f32 - m11 | 31.50 | 31.36 | 27.93 | 27.85 | 27.35 | 27.29 | 25.32 | 25.27
f64 - m11 (large) | 31.77 | 31.73 | 28.15 | 28.12 | 27.50 | 27.48 | 25.74 | 25.72
Table 9: QuickSRNet PSNRs (dB) and SSIM numbers evaluated for different scaling factors ($2\times$, $3\times$, and $4\times$) on benchmark SISR datasets before quantization Scaling |  QuickSRNet Specification |  Set5 |  Set14 |  BSD100 |  Urban100
---|---|---|---|---|---
Factor |  PSNR / SSIM |  PSNR / SSIM |  PSNR / SSIM |  PSNR / SSIM
2$\times$ |  f32 - m1 |  36.83 / 0.9563 |  32.35 / 0.9085 |  31.43 / 0.8900 |  29.66 / 0.8999
f32 - m2 (small) |  37.12 / 0.9575 |  32.57 / 0.9107 |  31.61 / 0.8925 |  30.15 / 0.9067
f32 - m3 |  37.30 / 0.9583 |  32.72 / 0.9117 |  31.72 / 0.8942 |  30.43 / 0.9104
f32 - m5 (medium) |  37.39 / 0.9586 |  32.82 / 0.9130 |  31.82 / 0.8955 |  30.75 / 0.9142
f32 - m7 |  37.51 / 0.9593 |  32.95 / 0.9136 |  31.88 / 0.8964 |  30.93 / 0.9164
f32 - m11 |  37.59 / 0.9594 |  33.00 / 0.9142 |  31.95 / 0.8973 |  31.14 / 0.9186
f64 - m11 (large) |  37.87 / 0.9603 |  33.29 / 0.9166 |  32.12 / 0.8992 |  31.74 / 0.9248
3$\times$ |  f32 - m1 |  32.75 / 0.9112 |  29.08 / 0.8234 |  28.41 / 0.7880 |  26.19 / 0.8026
f32 - m2 (small) |  33.10 / 0.9157 |  29.29 / 0.8282 |  28.57 / 0.7919 |  26.53 / 0.8128
f32 - m3 |  33.33 / 0.9180 |  29.39 / 0.8298 |  28.67 / 0.7947 |  26.77 / 0.8195
f32 - m5 (medium) |  33.58 / 0.9206 |  29.49 / 0.8327 |  28.75 / 0.7971 |  27.02 / 0.8266
f32 - m7 |  33.69 / 0.9216 |  29.60 / 0.8335 |  28.81 / 0.7986 |  27.16 / 0.8303
f32 - m11 |  33.81 / 0.9226 |  29.68 / 0.8346 |  28.86 / 0.8000 |  27.35 / 0.8347
f64 - m11 (large) |  34.14 / 0.9258 |  29.88 / 0.8397 |  29.02 / 0.8038 |  27.81 / 0.8459
4$\times$ |  f32 - m1 |  30.48 / 0.8659 |  27.31 / 0.7559 |  26.94 / 0.7147 |  24.47 / 0.7262
f32 - m2 (small) |  30.84 / 0.8741 |  27.55 / 0.7635 |  27.07 / 0.7196 |  24.74 / 0.7382
f32 - m3 |  31.04 / 0.8773 |  27.65 / 0.7656 |  27.16 / 0.7226 |  24.90 / 0.7447
f32 - m5 (medium) |  31.27 / 0.8821 |  27.79 / 0.7699 |  27.24 / 0.7253 |  25.08 / 0.7517
f32 - m7 |  31.39 / 0.8838 |  27.83 / 0.7709 |  27.30 / 0.7275 |  25.22 / 0.7573
f32 - m11 |  31.50 / 0.8856 |  27.93 / 0.7729 |  27.35 / 0.7289 |  25.32 / 0.7619
f64 - m11 (large) |  31.77 / 0.8908 |  28.15 / 0.7797 |  27.50 / 0.7344 |  25.74 / 0.7761

### Exporting QuickSRNet to ONNX for on-device profiling

To run the model on a mobile device, we shuffle the weights of some of the
convolutional layers, before depth-to-space and after space-to-depth (for
$1.5\times$ model) operations. This is necessary because the data layout of
PyTorch’s depth-to-space and space-to-depth operations (CRD) is not optimized
on the device used for profiling in this paper. For better on-device
performance, we change the data layout of these operations from CRD to DCR.
The appropriate method of creating a QuickSRNet model instance with the
shuffled weights (in DCR format) can be done with the following steps. Below
are a bunch of prerequisites to accomplish this task:

  * •

The PyTorch implementation of QuickSRNet can be found here

  * •

Pre-trained weights (including AIMET-quantized weights and encodings) are
available here

  * •

A Jupyter Notebook that shows how to load and use QuickSRNet is also available
here

#### Step 1

Load the quantized QuickSRNet model from the checkpointed weights and
encodings. With the PyTorch implementation of QuickSRNet, the model can be
instantiated with the appropriately shuffled weights as follows:

⬇

import torch

\par# Use one of QuickSRNetSmall, QuickSRNetMedium or QuickSRNetLarge with the
desired scaling factor.

scaling_factor = 2

model = QuickSRNetSmall(scaling_factor=scaling_factor)

\parstate_dict = torch.load(model_checkpoint_path,
map_location=’cpu’)[’state_dict’]

model.load_state_dict(state_dict)

model.to(device) # ‘device‘ is one of ’cuda’ or ’cpu’

\par# Re-arrange the weights of the appropriate conv layer(s)

model.to_dcr()

\par

#### Step 2 (optional)

To use QuickSRNet quantized using AIMET, use the following steps:

⬇

dummy_input_shape = (1, 3, 256, 256) # Expected input shape for the model (1 x
C x H x W)

dummy_input = torch.randn(dummy_input_shape)

\parsim = QuantizationSimModel(model=model,

dummy_input=dummy_input,

quant_scheme=QuantScheme.post_training_tf_enhanced,

default_output_bw=8,

default_param_bw=8)

sim.set_and_freeze_param_encodings(encoding_path=encoding_path)

sim.compute_encodings(forward_pass_callback=pass_calibration_data,

forward_pass_callback_args=(calibration_data,

scaling_factor,

use_cuda))

\par

#### Step 3

Export the model to ONNX:

⬇

import os

import torch

from aimet_torch.onnx_utils import OnnxExportApiArgs

\parfilename = ”<onnx_filename>”

output_dir = ”<output_dir>”

model_save_path = ”<output_dir>/<filename>.onnx”

\par# PixelUnshuffle does not map to space-to-depth without the code below

import torch.onnx.symbolic_helper as sym_help

import torch.onnx.symbolic_opset11 as opset11

from torch.onnx.symbolic_helper import parse_args, _unimplemented

\par@parse_args(’v’, ’i’)

def pixel_unshuffle(g, self, downscale_factor):

rank = sym_help._get_tensor_rank(self)

if rank is not None and rank != 4:

return _unimplemented(”pixel_unshuffle”, ”only support 4d input”)

return g.op(”SpaceToDepth”, self, blocksize_i=downscale_factor)

opset11.pixel_unshuffle = pixel_unshuffle

\par# Set ‘use_quantized‘ to ‘True‘ if exporting the quantized model, else
‘False‘

if use_quantized:

sim.export(output_dir,

filename,

dummy_input,

onnx_export_args=OnnxExportApiArgs(opset_version=11))

else:

torch.onnx.export(model, dummy_input, model_save_path, export_params=True,
opset_version=11)

\par

#### Step 4

Convert the ONNX space-to-depth and/or depth-to-space operations to DCR:

⬇

import onnx

from onnx.helper import make_attribute

\pardef overwrite_onnx_d2s_mode_to_dcr(onnx_path):

”””Manual override of the depth-to-space mode to DCR.”””

\paronnx_model = onnx.load(onnx_path)

graph = onnx_model.graph

for node in graph.node:

if node.op_type == ’DepthToSpace’:

depth_to_space_attribute = node.attribute

\parfound = False

for idx, attr in enumerate(node.attribute):

if attr.name == ’mode’:

found = True

break

if found:

node.attribute.pop(idx)

\parnew_attr = make_attribute(’s’, ’DCR’)

new_attr.name = ’mode’

\pardepth_to_space_attribute.extend([new_attr])

onnx.save(onnx_model, onnx_path)

\paronnx_path = ”<output_dir>/<filename>.onnx” # Path to the exported ONNX
file

overwrite_onnx_d2s_mode_to_dcr(onnx_path)

\par

#### Step 5

Re-order per-channel encodings for the quantized model to DCR:

⬇

import json

\pardef reorder_per_channel_encodings_to_dcr(encodings_path, layer_names):

”””

Used to re-arrange the per-channel encodings of the conv layer(s) preceding
the final depth-to-space operation.

\parThis is necessary because the data layout of PyTorch’s depth-to-space
operation (CRD) is

not optimized on device. For better on-device performance, the data layout
needs to be changed

to DCR.

\parNote: in the case of per-layer quantization, this function does not do
anything.

”””

\parwith open(encodings_path) as f:

encodings = json.load(f)

\parnew_encodings = encodings.copy()

to_shuffle = [key for layer_name in layer_names for key in
encodings[’param_encodings’] if layer_name in key]

for key in to_shuffle:

per_channel_enc = encodings[’param_encodings’][key]

if len(per_channel_enc) > 1:

scaling_factor = int((len(per_channel_enc) / 3) ** 0.5)

new_encodings[’param_encodings’][key] = [per_channel_enc[i + k *
(scaling_factor ** 2)]

for i in range(scaling_factor ** 2) for k in range(3)]

else:

# per-layer quantization: do nothing

pass

\parwith open(encodings_path, ’w’) as f:

json.dump(new_encodings, f, sort_keys=True, indent=4)

\parreorder_per_channel_encodings_to_dcr(encodings_path, [’anchor’,
’conv_last’])

\par