# Experimental demonstration of indefinite causal order induced quantum heat
extraction

Huan Cao1,2 These two authors contributed equally to this work. Ning-ning
Wang1,2 These two authors contributed equally to this work. Zhih-Ahn Jia1,2
Chao Zhang1,2<EMAIL_ADDRESS>Yu Guo1,2 Bi-Heng Liu1,2 Yun-Feng
Huang1,2<EMAIL_ADDRESS>Chuan-Feng Li1,2<EMAIL_ADDRESS>Guang-Can Guo1,2
1CAS Key Laboratory of Quantum Information, University of Science and
Technology of China, Hefei, 230026, China
2CAS Center For Excellence in Quantum Information and Quantum Physics, Hefei,
230026, China

###### Abstract

In the classical world, physical events always happen in a fixed causal order.
However, it was recently revealed that quantum mechanics allows events to
occur with indefinite causal order (ICO). In this study, we use an optical
quantum switch to experimentally investigate the application of ICO in
thermodynamic tasks. Specifically, we demonstrate that when a working system
interacts with two thermal reservoirs in an ICO, non-classical heat transfer
can be observed, even through they share the same temperature. Using such a
process, we simulate an ICO refrigeration cycle and investigate its
properties. We also show that by passing through the ICO channel multiple
times, one can extract more heat per cycle and thus obtain a higher
refrigeration efficiency. Our results provide inspirations for further
improving the efficiency of quantum thermodynamic tasks and shed new light on
the development of a new class of thermodynamic resource theories without
presumed causal order.

It is a deeply rooted concept that in a physical theory, there is a well-
defined pre-existing classical causal structure for which physical events
happen in a definite causal order. However, from the Bell-Kochen-Specker
theorem Bell (1964); Kochen and Specker (1975), we known that quantum
mechanics is incompatible with the viewpoint that physical observables have
pre-existing values independent of the measurement context. Inspired by this,
many recent studies on causal structure of quantum mechanics have shown that
if we assume the causal relation to obey the laws of quantum mechanics, it is
possible for two events to occur with a superposed causal order. Thus, there
is no pre-existing causal relation Oreshkov et al. (2012); Brukner (2014). The
quantum causal structure becomes especially crucial when quantum physics and
general relativity become relevant Oriti (2009); Hossenfelder (2017); Marletto
and Vedral (2017); Bose et al. (2017); Peres and Terno (2004). A typical
example is the quantum spacetime causal structure in the study of quantum
gravity Christodoulou and Rovelli (2019); DeWitt (1967); Rovelli (1990);
Gambini et al. (2004); Hardy (2009). For instance, when a massive object is in
a superposition of two or more distinct spatial positions, the states of the
gravitational field produced by the object will also be in a superposition,
implying that the spacetime causal order that originates from the spacetime
geometry will also be in a superposition.

In addition to the fundamental properties of indefinite causal order (ICO) and
causal structure, the applications of ICO as an operational resource in
quantum information and computation also attract considerable research
interests Taddei et al. (2019); Jia et al. (2019). It provides remarkable
enhancements ranging from channel discrimination Chiribella (2012),
communication and computation complexity Guérin et al. (2016); Feix et al.
(2015); Araújo et al. (2014) to quantum metrology Zhao et al. (2020);
Mukhopadhyay et al. (2018) , quantum information transmission Ebler et al.
(2018); Salek et al. (2018); Chiribella et al. (2018) etc. Recently, some of
these predictions have also been studied in the experiment, in which optical
quantum switch is utilized to simulate the ICO process Procopio et al. (2015);
Rubino et al. (2017); Goswami et al. (2018); Guo et al. (2020); Goswami et al.
(2020); Wei et al. (2019).

In thermodynamics, entropy in closed systems always tends to increase
definitely. An interesting question is what will happen when apply ICO in
thermodynamic tasks. One of such example is the recent discovery of ICO-based
quantum refrigeration Felce and Vedral (2020). There are several different
ways for quantum refrigeration: the standard one is powered by energy injected
by a time dependent driving force Campisi et al. (2015); Campisi and Fazio
(2016); the Maxwell demon can also be used to steer the heat by means of a
feedback control loop Maruyama et al. (2009); Elouard et al. (2017); while
another method is to use invasive quantum measurements as a resource to power
the refrigeration Buffoni et al. (2019). All the above refrigeration protocols
works in a world with pre-existing causal structures. The ICO-based protocol
provides a good supplement for which no pre-existing causal relation is
assumed Felce and Vedral (2020). This sheds new light on both of the quantum
causal structure and quantum thermodynamics.

In this paper, we experimentally demonstrate the advantages of ICO in
performing quantum thermodynamic tasks Mukhopadhyay et al. (2018); Guha et al.
(2020); Felce and Vedral (2020); Markes and Hardy (2011). In particular, by
using an optical quantum simulator, we demonstrate the ICO induced heat
extraction and investigate its feasibility to construct a quantum
refrigerator. We also show that by interacting with reservoirs in an ICO
multiple times, one can extract more heat from the reservoirs. The high
accuracy achieved in our experiment is expected to lead to more operational
protocols and thus contribute to a broader research into the use of ICO as a
resource.

Protocol outline.— Before detailing the experimental realization of the ICO-
driven quantum heat extraction, let us first recall how the protocol works.
Consider a system with Hamiltonian $\mathcal{H}$ and energy eigenstates
$\left|n\right\rangle$ for energy level $E_{n}$. After thermocontact with a
thermal reservoir with inverse temperature $\beta$, the resulting equilibrium
state of the system is always $T=e^{-\beta\mathcal{H}/Z}=\sum_{n}e^{-\beta
E_{n}}/Z\left|n\right\rangle\left\langle n\right|$ regardless of the initial
system state $\rho$, where
$Z=\mathrm{Tr}\left(e^{-\beta\mathcal{H}}\right)=\sum_{n}e^{-\beta E_{n}}$ is
the partition function. This thermodynamics operation can be characterized by
a quantum channel, namely, a completely positive trace preserving (CPTP) map
$\mathcal{T}\colon\mathcal{L}\left(\mathcal{H}\right)\rightarrow\mathcal{L}\left(\mathcal{H}\right)$
for which $\mathcal{T}\left(\rho\right)=T$ for all density operators $\rho$.
We denote the Kraus operators for $\mathcal{T}$ as $K_{i}$. The Kraus
decomposition is thus $\mathcal{T}\left(\rho\right)=\sum_{i}K_{i}\rho
K_{i}^{\dagger}$, where the Kraus operators $\left\\{K_{i}\right\\}$ satisfy
$\sum_{i}K_{i}^{\dagger}K_{i}=I$.

Now consider the situation where the system in state $\rho$ undergoes
thermocontact sequentially with two identical thermal reservoires with the
same temperature. If we assume the definite causal order, then the process is
given either by $\mathcal{T}^{1}\circ\mathcal{T}^{2}\left(\rho\right)$ or
$\mathcal{T}^{2}\circ\mathcal{T}^{1}\left(\rho\right)$, or potentially a
classical probabilistic mixture of these two processes, and the same thermal
state $T$ is obtained. However, when in a world with ICO, the two events
‘thermocontact with thermal reservoir 1 firstly’ and ‘thermocontact with
thermal reservoir 2 firstly’ can occur in a superposed causal order. This will
yield an interesting result—the resulting state is different from $T$. Such an
operation can be optically simulated using the quantum switch Chiribella
(2012); Chiribella et al. (2013); Procopio et al. (2015). The action of ICO is
achieved by routing photons through two channels with the visiting order being
tailored by the control qubit Procopio et al. (2015); Rubino et al. (2017);
Goswami et al. (2018); Guo et al. (2020); Goswami et al. (2020); Wei et al.
(2019). when control qubit $\left|\phi_{c}\right\rangle=\left|1\right\rangle$
or $\left|\phi_{c}\right\rangle=\left|0\right\rangle$ , the operations
$\mathcal{T}^{2}\circ\mathcal{T}^{1}$ or $\mathcal{T}^{1}\circ\mathcal{T}^{2}$
is carried out respectively. We denote the corresponding channel as
$\mathcal{S}^{T}$. In terms of Kraus operators, we have
$\mathcal{S}^{T}\left(\rho_{c}\otimes\rho\right)=\sum_{ij}M_{ij}\left(\rho_{c}\otimes\rho\right)M_{ij}^{\dagger}$
and

$M_{ij}=\left|0\right\rangle\left\langle
0\right|_{c}K_{i}^{1}K_{j}^{2}+\left|1\right\rangle\left\langle
1\right|_{c}K_{j}^{2}K_{i}^{1}$ (1)

where $K_{i}^{1}$ ($K_{j}^{2}$) represents the $i-$th ($j-$th) Kraus operator
of the thermalizing channels $\mathcal{T}^{1}$ ($\mathcal{T}^{2}$).

Considering the most simple non-trivial case, a two-level system, the ground
state is $\left|0\right\rangle$ (the excited state is $\left|1\right\rangle$)
with energy $E_{0}=0$ ($E_{1}=\Omega$), thus, the Hamiltonian for the system
is $\mathcal{H}=\Omega\left|1\right\rangle\left\langle 1\right|$. The thermal
state at a given temperature is $\rho=diag\left(1,e^{-\beta\Omega}\right)/Z$,
where $Z=1+e^{-\beta\Omega}$. In the following we set $\Omega=1$ for
simplicity. If the ancillary control qubit is initialized as
$\left|\phi_{c}\right\rangle=\left(\left|0\right\rangle+\left|1\right\rangle\right)_{c}/\sqrt{2}$.
the resultant output state undergoing ICO with two identical thermalizing
channels will be

$\displaystyle\mathcal{S}\left(\rho_{c}\otimes\rho\right)$ $\displaystyle=$
$\displaystyle\sum_{ij}M_{ij}\left(\rho_{c}\otimes\rho\right)M_{ij}^{\dagger}$
$\displaystyle=$
$\displaystyle\frac{1}{2}\left[\left(\left|0\right\rangle\left\langle
0\right|_{c}+\left|1\right\rangle\left\langle 1\right|_{c}\right)\otimes
T\right.$ $\displaystyle\left.+\left(\left|0\right\rangle\left\langle
1\right|_{c}+\left|1\right\rangle\left\langle 0\right|_{c}\right)\otimes T\rho
T\right]$

Where the control qubit is
$\rho_{c}=\left|\phi_{c}\right\rangle\left\langle\phi_{c}\right|$. Note that
the ICO evolution inside the quantum switch entangles the control qubit with
the resultant system state; if the control qubit is projected into
$\left|\pm\right\rangle=\left(\left|0\right\rangle\pm\left|1\right\rangle\right)_{c}/\sqrt{2}$,
the system state will collapse into

$\mathrm{Tr}_{c}\left[\left|\pm\right\rangle\left\langle\pm\right|_{c}\mathcal{S}\left(\rho_{c}\otimes\rho\right)\right]=\frac{1}{2}\left(T\pm
T\rho T\right)$ (3)

with the corresponding probability $p_{\pm}=\frac{1}{2}\mathrm{Tr}\left[T\pm
T\rho T\right]$. We note that the temperature of output system state could be
different from the thermal state $T$, with the temperature depending on the
measurement outcome in $\left|\pm\right\rangle_{c}$ basis and initial system
state $\rho$. This intriguing phenomenon suggests that the ICO can either
extract heat from or dump heat into the reservior. Hence ICO is potentially
useful for realizing thermodynamic tasks, such as refrigerating or cooling the
input system.

Figure 1: Schematic of the experimental apparatus. The quantum switch contains
two identical thermalizing channels (with pink planes underneath) in a
indefinite causal order. One of the causal order is presented by a bluish
optical path, the other is presented by a red optical path. The BS2 concludes
the superposition. BS: beam splitter, PBS: polarization beam spliter, HWP:
half wave plate, IF: interferometer filter.

Experimental implementation.— We build a high-performance tabletop photonic
quantum switch to realize the ICO process. A spontaneous parametric down
conversion (SPDC) produces heralded single photons at 780 nm. The heralded
photons, which act as working substances, are then fed into a Mach-Zenhder
interferometer, as shown in Fig 1. Here we use photonic polarization to
simulate the two energy levels of the working system, where horizontal
(vertical) polarization state H (V) represents the ground (excited) state. The
population of the excited state is related to the temperature of the working
system. A system state at an arbitrary temperature is prepared by randomly
rotating the polarization of the photons into H or V with a probability
proportional to its temperature. Energy detection can be realized by
measurements in Pauli $\sigma_{z}$ basis; thus, it is possible to detect the
temperature. To realize the ICO quantum channel (see eq.(Experimental
demonstration of indefinite causal order induced quantum heat extraction)),
the beam splitter 1 (BS1) introduces two spatial modes as the control qubit.
The polarization qubit undergoes the causal order
$\mathcal{T}^{1}\circ\mathcal{T}^{2}$ in one spatial mode, while
$\mathcal{T}^{2}\circ\mathcal{T}^{1}$ in the other one. BS2 then coherently
combines these spatial modes and projects the control qubit onto
$\left|\pm\right\rangle$. The ICO is valid when the spatial modes inside
quantum switch admit the interferometer. A phase-locking system is adopted to
ensure the stability of the path interferometer, with an average
interferometric visibility of more than 99.7% for four hours, which
sufficiently guarantees the ICO performance (See Section II in the
Supplementary Material for details). The action of the thermalizing channel
can be simulated using a probabilistic mixture of two fully amplitude damping
(AD) channels Fisher et al. (2012); Lu et al. (2017); Mancino et al. (2017),
$\mathcal{T}\left(\rho\right)=\sum_{i=0}^{3}p_{i}K_{i}\rho K_{i}^{\dagger}$,
where the $\left\\{K_{0},K_{1}\right\\}$ ($\left\\{K_{2},K_{3}\right\\}$)
constitute a channel damping to ground (excited) state, and
$\overrightarrow{p}=\left\\{p_{0},p_{1},p_{2},p_{3}\right\\}$ denote the
probability vectors, which depend on the temperature of the thermalizing
channel. The AD channel is realized using polarization M-Z interferometer-like
setup with half-wave plates (HWPs) and polarization beam splitters (PBSs), as
shown in Fig.1. We have performed the process tomography of the thermalizing
channel at several different temperatures. The average process fidelity
exceeded 99.9%, verifying the credible simulation of the channel (see section
II in Supplementary Material for details).

Figure 2: (a) Energy change $\Delta
E_{\pm}=\mathrm{Tr}\left(\rho_{\pm}\mathcal{H}\right)-\mathrm{Tr}\left(\rho\mathcal{H}\right)$
where $\rho_{\pm}=\left[T\pm T\rho T\right]/\mathrm{Tr}\left[T\pm T\rho
T\right]$ is the normalized output system state with the control qubit
measured in $\left|\pm\right\rangle$. The weighted energy change is presented
as $\Delta\widetilde{E}_{\pm}$. We account for the deviation of experimental
data to theoretical prediction when the $E_{C}\rightarrow 0$ that at ultra
cool temperature, the success probability $P_{-}$ approaches to zero, thus the
collecting signal counting is low but the noise due to experimental
imperfection never decreases, which causes the signal-to-noise level to
gradually deteriorate. (b) The energy of the working system
$E_{\rho}=\mathrm{Tr}\left(\rho\mathcal{H}\right)$ when the control qubit is
measured at
$\left|+\right\rangle,\left|-\right\rangle,\left|0\right\rangle,\left|1\right\rangle$.
The histograms represent the theoretical prediction, while the plots represent
the experimental result. The dotted line shows the corresponding quantity of
the input state. The quoted error in the experimental data points in plots
reflects the impact of Poissonian statistics on the collecting count numbers.

Results of the ICO process.— We first verify the non-classical heat transfer
driven by the ICO. We implement numbers of experiments by traversing the
temperatures of thermalizing channel which belongs to the reservoir in the
following discussion. The system state is initialized into the thermal state
with the same temperature ($1/\beta_{C}$) as the reservoir, $\rho=T$. Fig. 2
(a) shows the measured energy change $\Delta E$ of the system qubit (as the
working substance) after passing through the ICO channel. The horizontal
coordinate $E_{C}=e^{-\beta_{C}}/\left(1+e^{-\beta_{C}}\right)$ ranges from 0
to 0.5, corresponding to the temperatures from zero to infinity (here we use
the energy of the thermal state of the working system to represent the
reservoir’s temperature for simplicity). By extrapolating our experimental
data ( red and blue dots in Fig. 2 (a)) to theoretical prediction (red and
blue lines in Fig. 2 (a)), we find that the working substance can extract the
heat flow from the reservoir when the control qubit is measured in
$\left|-\right\rangle$, even though they initially share the same temperature.
This intriguing phenomenon is applicable in arbitrary temperature case with
the exceptions of zero and infinity. The heat transfer decreases when the
temperature increases, whereas the successful probability
$p_{-}=\mathrm{Tr}\left[\left|-\right\rangle\left\langle-\right|S\left(\rho_{c}\otimes\rho\right)\right]$
increases (inset in Fig. 2 (a)). The weighted energy change
$\Delta\widetilde{E}_{\pm}=p_{\pm}\Delta E_{\pm}$ is also presented (orange
and green dots and lines 2 (a)). For comparison, we also measure the energy
transfer when the control qubit is measured in computational basis
$\left\\{\left|0\right\rangle,\left|1\right\rangle\right\\}$, by exemplifying
the reservoir’s temperature to be $E_{C}=0.25$ (Fig.2(b)). Such a case yields
the classical outcome in which the output working substance has the same
temperature as the reservoir, and no heat extraction could achieve.

Obviously, the non-classical heat transfer driven by ICO can be used for
thermodynamic tasks. For example, when the working substance appears at the
heating branch, we can interact it with an external hot reservoir to release
the extracted heat, thus refrigerating the reservoir; when the working
substance appears at the cooling branch, we can send it back to the reservoir
to erase the unwanted heat exchange. For the opposite purpose (i.e., to
construct a heat engine or cool the working substance), one can instead
discard the heating branch and reserve the cooling branch. An interesting
question is whether the working substance can become colder or hotter after
passing through the ICO channel multiple times.

Figure 3: (a) Energy of the working system after passing through the ICO
channel multiple times. Each step will generate a two-branch outcome. The
latter step is performed only when the former step yields a failed outcome
$\left|+\right\rangle$ (multi-pass condition, as shown by the arrows in each
step, with intermediate ones are abbreviated with ellipsis). The saturated
energy in both outcomes are indicated by red and blue lines. In this
experiment, the initial working substance had the same temperature as the
reservoir with $E_{C}=0.25$. (b) Coefficients of the quantum refrigerator
based on the classical strategy (blue) and multi-pass strategy (red). Here the
coefficient is calculated in steady-state solution. Tht lines show the
theoretical predictions, while the dots are the experimental data.

Results of multi-pass ICO.— Our second result is to investigate this multi-
pass strategy. We start with the working substance at the same temperature as
the reservoirs. As an example, we still adopt the initial temperature to be
$E_{C}=0.25$, as shown by the first point in Fig.3(a). At each step, a single
run of quantum switch is carried out. Hence one initial state will generate a
two-branch outcome (indicated by the arrows in Fig.3(a)). Here we only
consider the case in which the working substance becomes colder, and then send
it into the next step as initial state.In the experiment, since photons will
be annihilated after being measured in each step, we use the measurement
results (classical information) to determine the state preparation in the next
step to simulate this iterative process (as the loop depicted in Fig.1). The
experimental results for the 10-steps ICO are summarized in Fig.3(a), in which
the iteration process is indicated by the arrows. We observe that when the
multi-pass ($N\geq 2$) ICO is implemented , a colder working substance in the
unwanted branch could jump into a higher temperature compared to in the
single-step process ($N=1$). This means that the multi-pass ICO may release
the restriction for external reservoir for heat dumping. Interestingly,
Fig.3(a) shows that the working substance will quickly saturates to a specific
temperature, which means the output working substance remains unchanged in its
input state when the control qubit is measured to be $\left|+\right\rangle$.
We theoretically calculate this steady-state solution for all temperatures of
the reservoir and also experimentally sample five points
$E_{C}=\left\\{0.05,0.15,0.25,0.35,0.45\right\\}$, finding that the working
substance always tends toward this steady-state solution after several
iterations (see section I of Supplementary Material for details).

Figure 4: Schematic diagram of a single cycle of quantum refrigerator. The
cold and hot reservoir is denoted by the cyan and moccasin squares
respectively, and the working substance is denoted by the ball with the color
corresponding to its temperature. The working substance is initially in
thermal equilibrium with the cold reservoir and then undergoes the ICO. The
cycle continues (going up into the hot reservoir) by control qubit on
$\left|-\right\rangle$, or recycle (going down). Two recycling strategies are
presented by the dotted arrow (classical strategy) and solid arrow (multi-pass
strategy). Before entering the ICO quantum switch, erasing the register of
control qubit consumes the quantum work $\Delta W$. The total heat extraction
per cycle $\Delta E$ is related to the temperature of particle entering
($\beta_{-}$) and leaving the external hot reservoir ($\beta_{H}$) (depicted
by the particles on the centreline).

Construction of a quantum refrigerator—In the following we consider the
refrigeration task driven by ICO. We construct an operational cycle to realize
a refrigerator. The schematic diagram of a single cycle of quantum
refrigerator is shown in Fig.4, of which the complete process is described as
follows. In the first stroke (i), the working substance is initialized by
classically interacting with the cold reservoir (Preparing a colder working
substance required additional work cost, thus is excluded in our discuss).
Then interacts with the two cold reservoirs superposed in ICO. In the second
stroke (ii), the control qubit is measured. If the control qubit is collapsed
into $\left|-\right\rangle$, the working system successfully extracts heat
from the cold reservoir, followed by proceeding to next stroke. If the control
qubit collapses into $\left|+\right\rangle$, two alternative strategies are
available as listed above: (a) the working system classically contacts the
cold reservoir to recover its initialized state, thereby undoing the unwanted
heat change, and a new cycle is implemented. (termed classical strategy); and
(b) the ICO quantum switch is repeatedly passed through until the desired
outcome is obtained. (termed multi-pass strategy). In the third stroke (iii),
The working substance makes classical thermal contact with external hot
reservoir for heat release, followed by a classical thermal contact with cold
reservoir once again for initialization; subsequently, a new cycle is started.

To evaluate the performance of the quantum refrigerator, we introduce the
coefficient of performance, which is calculated by dividing the heat change
from the cold reservoir by the work cost of measurement Abdelkhalek et al.
(2016). To calculate the heat flow, we adopt the assumption that all the
thermalizations are isochoric, which assures that the internal energy change
of working system refers to the heat exchange with reserviors. The heat flow
out of cold reservoirs per cycle lies in two aspects: one results from
themalization inside quantum switch in stroke (i) when a desired outcome
$\left|-\right\rangle$ of the control qubit is measured. We calculate the heat
flow extractied from cold reservior $\Delta E_{1}=E_{-}-E_{C}$, where
$E_{-}=\mathrm{Tr}\left(\rho_{-}\mathcal{H}\right)$. The other aspect results
from the classical thermal contact of working system with cold reservoir in
stroke (iii), which is given by $\Delta E_{2}=-\left(E_{H}-E_{C}\right)$,
where $E_{H}$ is the energy of the thermal state of the external hot
reservoir. The total heat change in cold reservoir in a single cycle is
$\Delta E=\Delta E_{1}+\Delta E_{2}$, related to the temperature of hot
reservoir and cold reservoir. Here, we fix the hot reservoir
$\beta_{H}=\beta_{C}$ to maximize the heat change. In this case, the total
heat change $\Delta E=E_{-}-E_{C}$.

The magic of the ICO refrigerator can also be explained by the Maxwell’s
demon-like cooling mechanism. In each cycle of the quantum refrigerator, the
control qubit need to be measured and the result is stored in a register. A
following operation is determined by the information in the register, as shown
by the loop in Fig.1. Since we assume the control qubit is degenerate in
energy, the measurement itself does not cost energy. Rather, the energy cost
comes from resetting the register to its initial state in order to proceed to
the next cycle, which refers to Landauer’s erasure Landauer (1961). The work
cost is given by $\Delta W=-\frac{1}{\beta_{R}}S$, where $S=\left(p_{+}\ln
p_{+}+p_{-}\ln p_{-}\right)$ is the Shannon entropy of the register and
$\beta_{R}$ is the inverse temperature of the resetting reservoir. Therefore
the coefficient of quantum refrigerator is given by

$\eta=\frac{\Delta E}{\bar{n}\Delta W}$ (4)

where $\bar{n}=\frac{1}{p_{-}}$ is the average number of measurements consumed
to realize one cycle of the quantum refrigerator.

For the classical strategy, the coefficient of quantum refrigerator can be
directly calculated by Eq. (4). When it comes to multi-pass strategy, it
potentially contains many steps per cycle. The heat flow extraction or work
cost in each step is not identical. However, we can rationally approximate the
coefficient for multi-pass strategy by assuming that the quantum refrigerator
working at the steady-state point (see section I of Supplementary Material for
details). That is because single cycle is much likely to undergo multiple
steps and evolve into equilibrium, especially for a low temperature of cold
reservoir in practice, where the probability of mesurement outcome
$\left|-\right\rangle$ approaches to zero with the temperature decreasing. we
comparatively present the coefficients of quantum refrigerator under both the
classical and multi-pass strategy in Fig. 3(d), in which the solid lines show
the theoretical prediction and dots show the experimental results for several
sample temperatures $E_{C}=\left\\{0.05,0.15,0.25,0.35,0.45\right\\}$. The
results show that the multi-pass strategy is always superior to classical
strategy. The results also provide a deep insights into how to improve the
efficiency of quantum thermodynamic tasks by replacing irreversible isochoric
thermalizations with a reversible process to the extent possible.

We provide the strategies of quantum refrigerator that involve both quantum
and classical operation. Dispite a measurement inside strokes, both the
outcomes are all taken into account, hence the refrigerator cycle does not
depend on postselection to gain its advantages. In addition, instead of
feedback control loop in which the control qubit is measured, the quantum
refrigerator can also be realized without projection but rather consumption of
purity of control qubit Felce and Vedral (2020). In the present work, we have
demonstrated ICO-driven nonclassical heat transfer between system and
reservoirs even though they share the same initial temperature, for which heat
change via direct thermalization would be inaccessible. Our findings are
beneficial for advancing a new class of resource theories in quantum
thermodynamics based ICO. Especially in the case where the control qubit is
degenerate in energy, which means it can not be used for direct
thermalization, the ICO provide an effective way to ultilize its free energy.
Our results confirm that the ICO can be a useful resource in quantum
thermodynamics, which provide a new paradigm of work extraction alternative to
other non-classical features Perarnau-Llobet et al. (2015); Francica et al.
(2017). Therefore we expect that our work will advance further investigations
on the exotic properties of indefinite causal order, as well as its
superiority in quantum tasks.

We thank Yong-Xiang Zheng, Xue Li, Xiao Liu for beneficial discussions. This
work was supported by National Natural Science Foundation of China (11734015,
62075208, 11774335, 11874345, 11821404, 11904357), Anhui Initiative in Quantum
Information Technologies (AHY070000, AHY020100, AHY060300), National Key
Research and Development Program of China (2017YFA0304100, 2016YFA0301300,
2016YFA0301700), Key Research Program of Frontier Sciences, CAS (QYZDY-SSW-
SLH003), Science Foundation of the CAS (ZDRW-XH-2019-1), the Fundamental
Research Funds for the Central Universities, Science and Technological Fund of
Anhui Province for Outstanding Youth (2008085J02).

Noted—— Recently, we become aware of a related work by Nie et al. which
experimentally study quantum thermodynamics driven by ICO on nuclear spins
using the nuclear magnetic resonance system Nie et al. (2020).

## References

  * Bell (1964) J. S. Bell, Physics Physique Fizika 1, 195 (1964).
  * Kochen and Specker (1975) S. Kochen and E. P. Specker, in _The logico-algebraic approach to quantum mechanics_ (Springer, 1975), pp. 293–328.
  * Oreshkov et al. (2012) O. Oreshkov, F. Costa, and Č. Brukner, Nature communications 3, 1 (2012).
  * Brukner (2014) Č. Brukner, Nature Physics 10, 259 (2014).
  * Oriti (2009) D. Oriti, _Approaches to quantum gravity: Toward a new understanding of space, time and matter_ (Cambridge University Press, 2009).
  * Hossenfelder (2017) S. Hossenfelder, _Experimental search for quantum gravity_ (Springer, 2017).
  * Marletto and Vedral (2017) C. Marletto and V. Vedral, Physical review letters 119, 240402 (2017).
  * Bose et al. (2017) S. Bose, A. Mazumdar, G. W. Morley, H. Ulbricht, M. Toroš, M. Paternostro, A. A. Geraci, P. F. Barker, M. Kim, and G. Milburn, Physical review letters 119, 240401 (2017).
  * Peres and Terno (2004) A. Peres and D. R. Terno, Reviews of Modern Physics 76, 93 (2004).
  * Christodoulou and Rovelli (2019) M. Christodoulou and C. Rovelli, Physics Letters B 792, 64 (2019).
  * DeWitt (1967) B. S. DeWitt, Physical Review 160, 1113 (1967).
  * Rovelli (1990) C. Rovelli, Physical Review D 42, 2638 (1990).
  * Gambini et al. (2004) R. Gambini, R. A. Porto, and J. Pullin, New Journal of Physics 6, 45 (2004).
  * Hardy (2009) L. Hardy, in _Quantum reality, relativistic causality, and closing the epistemic circle_ (Springer, 2009), pp. 379–401.
  * Taddei et al. (2019) M. M. Taddei, R. V. Nery, and L. Aolita, Physical Review Research 1, 033174 (2019).
  * Jia et al. (2019) D. Jia, F. Costa, et al., Physical Review A 100, 052319 (2019).
  * Chiribella (2012) G. Chiribella, Physical Review A 86, 040301 (2012).
  * Guérin et al. (2016) P. A. Guérin, A. Feix, M. Araújo, and Č. Brukner, Physical review letters 117, 100502 (2016).
  * Feix et al. (2015) A. Feix, M. Araújo, and Č. Brukner, Physical Review A 92, 052326 (2015).
  * Araújo et al. (2014) M. Araújo, F. Costa, and Č. Brukner, Physical review letters 113, 250402 (2014).
  * Zhao et al. (2020) X. Zhao, Y. Yang, and G. Chiribella, Physical Review Letters 124, 190503 (2020).
  * Mukhopadhyay et al. (2018) C. Mukhopadhyay, M. K. Gupta, and A. K. Pati, arXiv preprint arXiv:1812.07508 (2018).
  * Ebler et al. (2018) D. Ebler, S. Salek, and G. Chiribella, Physical review letters 120, 120502 (2018).
  * Salek et al. (2018) S. Salek, D. Ebler, and G. Chiribella, arXiv preprint arXiv:1809.06655 (2018).
  * Chiribella et al. (2018) G. Chiribella, M. Banik, S. S. Bhattacharya, T. Guha, M. Alimuddin, A. Roy, S. Saha, S. Agrawal, and G. Kar, arXiv preprint arXiv:1810.10457 (2018).
  * Procopio et al. (2015) L. M. Procopio, A. Moqanaki, M. Araújo, F. Costa, I. A. Calafell, E. G. Dowd, D. R. Hamel, L. A. Rozema, Č. Brukner, and P. Walther, Nature communications 6, 1 (2015).
  * Rubino et al. (2017) G. Rubino, L. A. Rozema, A. Feix, M. Araújo, J. M. Zeuner, L. M. Procopio, Č. Brukner, and P. Walther, Science advances 3, e1602589 (2017).
  * Goswami et al. (2018) K. Goswami, C. Giarmatzi, M. Kewming, F. Costa, C. Branciard, J. Romero, and A. White, Physical review letters 121, 090503 (2018).
  * Guo et al. (2020) Y. Guo, X.-M. Hu, Z.-B. Hou, H. Cao, J.-M. Cui, B.-H. Liu, Y.-F. Huang, C.-F. Li, G.-C. Guo, and G. Chiribella, Physical Review Letters 124, 030502 (2020).
  * Goswami et al. (2020) K. Goswami, Y. Cao, G. Paz-Silva, J. Romero, and A. White, Physical Review Research 2, 033292 (2020).
  * Wei et al. (2019) K. Wei, N. Tischler, S.-R. Zhao, Y.-H. Li, J. M. Arrazola, Y. Liu, W. Zhang, H. Li, L. You, Z. Wang, et al., Physical review letters 122, 120504 (2019).
  * Felce and Vedral (2020) D. Felce and V. Vedral, Physical Review Letters 125, 070603 (2020).
  * Campisi et al. (2015) M. Campisi, J. Pekola, and R. Fazio, New Journal of Physics 17, 035012 (2015).
  * Campisi and Fazio (2016) M. Campisi and R. Fazio, Journal of Physics A: Mathematical and Theoretical 49, 345002 (2016).
  * Maruyama et al. (2009) K. Maruyama, F. Nori, and V. Vedral, Reviews of Modern Physics 81, 1 (2009).
  * Elouard et al. (2017) C. Elouard, D. Herrera-Martí, B. Huard, and A. Auffeves, Physical Review Letters 118, 260603 (2017).
  * Buffoni et al. (2019) L. Buffoni, A. Solfanelli, P. Verrucchi, A. Cuccoli, and M. Campisi, Physical review letters 122, 070603 (2019).
  * Guha et al. (2020) T. Guha, M. Alimuddin, and P. Parashar, arXiv preprint arXiv:2003.01464 (2020).
  * Markes and Hardy (2011) S. Markes and L. Hardy, in _Journal of Physics-Conference Series_ (2011), vol. 306, p. 012043\.
  * Chiribella et al. (2013) G. Chiribella, G. M. D’Ariano, P. Perinotti, and B. Valiron, Physical Review A 88, 022318 (2013).
  * Fisher et al. (2012) K. A. Fisher, R. Prevedel, R. Kaltenbaek, and K. J. Resch, New Journal of Physics 14, 033016 (2012).
  * Lu et al. (2017) H. Lu, C. Liu, D.-S. Wang, L.-K. Chen, Z.-D. Li, X.-C. Yao, L. Li, N.-L. Liu, C.-Z. Peng, B. C. Sanders, et al., Physical Review A 95, 042310 (2017).
  * Mancino et al. (2017) L. Mancino, M. Sbroscia, I. Gianani, E. Roccia, and M. Barbieri, Physical Review Letters 118, 130502 (2017).
  * Abdelkhalek et al. (2016) K. Abdelkhalek, Y. Nakata, and D. Reeb, arXiv preprint arXiv:1609.06981 (2016).
  * Landauer (1961) R. Landauer, IBM journal of research and development 5, 183 (1961).
  * Perarnau-Llobet et al. (2015) M. Perarnau-Llobet, K. V. Hovhannisyan, M. Huber, P. Skrzypczyk, N. Brunner, and A. Acın, Phys. Rev (2015).
  * Francica et al. (2017) G. Francica, J. Goold, F. Plastina, and M. Paternostro, npj Quantum Information 3, 1 (2017).
  * Nie et al. (2020) X. Nie, X. Zhu, C. Xi, X. Long, Z. Lin, Y. Tian, C. Qiu, X. Yang, Y. Dong, J. Li, et al., arXiv preprint arXiv:2011.12580 (2020).


EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH (CERN)

 LHCb-PUB-2017-017

Signal coupling to embedded pitch adapters in silicon sensors

We have examined the effects of embedded pitch adapters on signal formation in
n-substrate silicon microstrip sensors with data from beam tests and
simulation. According to simulation, the presence of the pitch adapter metal
layer changes the electric field inside the sensor, resulting in slowed signal
formation on the nearby strips and a pick-up effect on the pitch adapter. This
can result in an inefficiency to detect particles passing through the pitch
adapter region. All these effects have been observed in the beam test data.

Keywords: Silicon sensors; Detector Technology; Testbeam; Sensor irradiation;
Pitch adapters

© CERN on behalf of the LHCb collaboration, licence CC-BY-4.0.

##  1 Introduction

The pitch between strips of silicon microstrip sensors and the pitch between
input channels of the application-specific integrated circuits (ASICs) used
for readout can often differ by large amounts. One solution is to use external
adapters such as metal traces on a glass substrate, but this presents its own
difficulties for the construction of large detectors. While the required wire
bond geometry is consistent with industry standards, the number of bonds
needed doubles. Moreover, additional space and material is introduced, which
may be difficult to accommodate in densely packed silicon detectors.

Another solution to this problem is to use embedded pitch adapters built into
the silicon sensors themselves. During sensor fabrication a second metal layer
of traces with bond pads designed to match the readout ASIC pitch is placed on
top of the silicon dioxide passivation. The AC-coupled metal strips of the
first layer are directly connected with metal vias to this second layer above.

This technology has been studied for the silicon tracker upgrade for the ATLAS
experiment [1] and the Upstream Tracker (UT) upgrade for the LHCb experiment
[2]. In the studies of the p-substrate sensor prototypes designed for the
ATLAS experiment it was shown that the pitch adapters have an effect on the
measured capacitance between strips [3]. However, no evidence of “pick-up”
(creating spurious signals on far away strips) or of signal loss on nearby
strips was found. Sensors with increased strip segmentation utilizing routing
lines in the first metal layer have been studied for upgrades to the CMS
experiment [4]. In these sensors, charge pick-up on the routing lines and
corresponding loss of signal on the nearby sensor strips was observed.

During research and development for the UT, we observed a loss of efficiency
localized in the pitch adapter (PA) region in both n- and p-substrate sensors.
Maps of the locations of passing charged particles that did not leave a
reconstructed signal in the prototype sensor are shown in Fig. 1. The
inefficiency maps out the location of the embedded pitch adapter on these
sensors.

(a) n-substrate sensor

(b) p-substrate sensor

Figure 1:  Positions on two sensors of charged particles for which a
corresponding signal is not found in the sensor. The inefficient band at the
top corresponds to the edge of the sensor active area. The pattern of
inefficiency is confined to the pitch adapter region of the sensor and only in
the region between two strips. The n-type sensor (1(a)) shows a small
inefficiency near certain bond pads, while the p-type sensor (1(b)) shows a
much stronger effect. The vertical bands with no missed hits are regions
removed from analysis due to strips flagged as bad.

In this paper, we focus on the effect in n-substrate sensors, using prototypes
designed for the LHCb upgrade. We have simulated the effects of the PA on
signal formation in the sensor to understand the cause of the observed
effects. We then confirm further predictions of the simulation regarding the
effect of design geometry and time evolution on the pick-up effect.

###  1.1 Sensor description

The sensors used are miniature
($$1.8\text{\,}\mathrm{cm}\text{/}$\times$1.4\text{\,}\mathrm{cm}\text{/}$$)
silicon microstrip sensors produced by Hamamatsu Photonics [5]. The silicon is
$320\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ thick and of p+-in-n type.
Each sensor has 64 strips with a pitch of
$190\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$. The sensor’s AC-coupled
strips are connected to the readout ASICs with a pitch of approximately
$80\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ through the use of an
embedded pitch adapter.

The design of the pitch adapter is pictured in Fig. 2. Metal traces with a
width of approximately $10\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$
connect each channel to a bonding pad $65\text{\,}\mathrm{\SIUnitSymbolMicro
m}\text{/}$ wide (perpendicular to the strips) and
$210\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ long (parallel to the
strips). The distance between the metal traces in the densest region varies
from $10\text{\,}\mathrm{\SIUnitSymbolMicro
m}\text{/}20\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$. These traces and
bonding pads are located on top of the active sensor area. The metal layer of
the pitch adapter is separated from the first metal layer containing the strip
AC-metal by a layer of silicon dioxide that is
$2\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ thick. The sensor’s metal
strips are $123\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ wide, with a
gap between of $67\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$.

Figure 2: Zoomed in view of the sensor metalization layers, showing one-half
of the pitch adapter. The hatched purple vertical bands show the metal strips
above each silicon strip. Thirty-two channels are connected via the second-
metal layer shown in solid red, to the closely spaced bond pads in the center.
The other thirty-two channels are similarly connected with a mirror-image
design not pictured.

The analysis in this paper focuses on the bond pad region, where the larger
metal area is expected to cause stronger effects. The bond pads almost
completely bridge the inter-strip region when they are centered relative to
the strips below. In other cases, the bond pad is shifted to one side or the
other, partially or fully overlapping the metal strip.

##  2 Simulation

We simulate the operation of the sensor in the pitch adapter region using the
TCAD Sentaurus package, part of Synopsys software [6]. We use a two
dimensional simulation of the full sensor in a slice perpendicular to the
strips, including two implant strips and a metal strip to model the pitch
adapter bond pad. The geometry of the model is shown in Fig. 3.

###  2.1 Model description

The exact parameters of the sensors manufactured by Hamamatsu are not known,
therefore the choice of many parameters for the simulation was inspired by
studies of similar silicon sensors produced by Hamamatsu for CMS [7]. We
assume the bulk to be doped with phosphorus (n-type) with a concentration of
$3\text{\times}{10}^{12}\text{\,}\text{/}{\mathrm{cm}}^{3}$. The strips
implants are doped with boron (p+) with a much higher concentration of
$1\text{\times}{10}^{19}\text{\,}\text{/}{\mathrm{cm}}^{3}$. A Gaussian error
function is used for the implant doping profile with a depth of
$1\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ and a width of
$1\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$. Variations of the implant
profile depth and width in ranges 1-$5\text{\,}\mathrm{\SIUnitSymbolMicro
m}\text{/}$ and 0.5-$1.5\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$,
respectively, have shown no significant effects on the results of the
simulation. The bottom side of the bulk is doped with phosphorus (n+) with a
concentration of $0.5\text{\times}{10}^{19}\text{\,}\text{/}{\mathrm{cm}}^{3}$
to achieve ohmic contact with the metal backplane. A density of positive
charges trapped at the boundary of the silicon and silicon dioxide of
$1\text{\times}{10}^{11}\text{\,}\text{/}{\mathrm{cm}}^{2}$ is assumed for
unirradiated sensors. To emulate the effect of irradiation, the charge density
is increased to its saturation value of
$3\text{\times}{10}^{12}\text{\,}\text{/}{\mathrm{cm}}^{2}$. These numbers are
based on typical values for silicon–silicon dioxide boundaries [8, 9], but
have not been measured on the tested sensors. It is possible that the tested
sensors start with a lower charge density than
$1\text{\times}{10}^{11}\text{\,}\text{/}{\mathrm{cm}}^{2}$ when unirradiated,
but changes to the qualitative behavior of the sensors are not observed below
this value.

We chose for the width of the simulation region to be twice the pitch of
$190\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$, and thus it includes two
p+ strips each $120\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ wide. The
p+ implants are connected to electrical ground via
$1.5\text{\,}\mathrm{M\SIUnitSymbolOhm}\text{/}$ resistors and AC coupled to
the metal strips placed at $300\text{\,}\mathrm{nm}\text{/}$ above them in the
silicon dioxide layer. When simulating the pitch adapter pads, a second metal
layer on top of the silicon dioxide is held at electrical ground to represent
the $65\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ wide pitch adapter
pads. Three pads are included with a separation of
$10\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$; the center pad is placed
over the inter-strip region; its edge is at a distance of
$2.5\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ from the edge of the
strip. A positive high voltage bias of $300\text{\,}\mathrm{V}\text{/}$ (the
depletion voltage is approximately $250\text{\,}\mathrm{V}\text{/}$) is
applied to the backside contact. An adaptive meshing with cell dimensions from
$$20\text{\,}\mathrm{\SIUnitSymbolMicro
m}\text{/}$\times$50\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$$ to
$$1\text{\,}\mathrm{\SIUnitSymbolMicro
m}\text{/}$\times$1\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$$ is used.

Figure 3: Two dimensional model of the sensor used for the simulation. The
successive insets zoom in on the details of the implant and metalization
geometry near the sensor surface. The metal layers of the strip AC-metal and
pitch adapter have had their thickness exaggerated to improve the
visualization.

It was shown in Ref. [7] that the TCAD Sentaurus default values for the most
important parameters (permittivity, electron affinity, band gap energy,
permittivity of silicon dioxide) are well justified and therefore were used in
this study. Though the default generation and recombination times of electron
and holes are lower than those of very clean silicon, we do not expect it to
have a significant influence on the results of this investigation. The
simulation of charge carrier mobility accounts for velocity saturation at high
fields and doping dependency. The generation and recombination of charge
carriers is implemented according to the Shockley-Read-Hall recombination
model. At the boundary an absence of electric field and carrier current
perpendicular to the boundary are required [10].

###  2.2 Results of the simulation

The electrostatic potential maps from simulation are shown in Fig. 4 for
configurations with and without the pitch adapter metal layer. The presence of
the pitch adapter layer changes the electric field in the inter-strip region
close to the silicon surface. The charge carriers (holes) are attracted to the
silicon-oxide surface and only then travel towards the strip implants. A shift
in the PA pad position with respect to the middle of the inter-strip region
results in a remarkable effect. An electrostatic potential at the silicon-
oxide boundary rises near the strip opposite to the shift direction, producing
a barrier as shown in Fig. 5. The gradient of the potential in the $x$
direction increases the coupling to the second metal and decreases the charge
collection.

This potential creates a region of zero electric field as shown in Fig. 6. The
field prevents the charge carriers from reaching the nearby strip for a
timescale long compared to the usual readout. This results in a signal loss on
the strip and signal pick-up on the corresponding PA pad, where the induced
charge is longer lived. In case of saturated charge density in the silicon-
oxide interface (corresponding to an irradiated sensor) the effect of the PA
pads is suppressed and the resulting potential is the same as without PA pads.

(a) No PA

(b) With PA

Figure 4: Simulated map of the electrostatic potential in the inter-strip
region at $300\text{\,}\mathrm{V}\text{/}$ bias. The left figure corresponds
to a configuration without pitch adapter pads, while the right one includes
them. The white line represents the boundary of the depletion region. The
metal layers of the strip AC-metal and pitch adapter have had their thickness
exaggerated to improve the visualization.

Figure 5: Simulated electrostatic potential profiles at the silicon–oxide
boundary in the inter-strip region for various PA pad positions and without
PA. Zero PA shift centers the PA metal pad between two strips,
$2.5\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ from either strip; each
shift increases the distance on one side. Right figure shows zoomed profiles.
The dotted line with black circles represents the case without any PA pads.
The dashed line with crosses represents the case of an irradiated sensor
modeled by using saturation value of
$3\text{\times}{10}^{12}\text{\,}\text{/}{\mathrm{cm}}^{2}$ for oxide charge
density (irrespective of presence and position of PA pads).

(a) Centered PA

(b) PA shifted $10\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$

Figure 6:  Simulated electric field maps in the region near the silicon
surface close to a strip implant. The color scale represents the field
strength and the printed arrows the direction. In (6(a)), the PA is centered.
In (6(b)), the PA is shifted $10\text{\,}\mathrm{\SIUnitSymbolMicro
m}\text{/}$, creating a zero field region near the surface that blocks charge
moving towards the strip on the left side. The metal layers of the strip AC-
metal and pitch adapter have had their thickness exaggerated to improve the
visualization.

To investigate signal pulse shapes, we simulate hits of minimum ionizing
particles (MIP) penetrating the sensor perpendicularly to its plane, near to
the halfway point between two strips. The MIP ionization density is tuned to
produce approximately $2\text{\times}{10}^{4}\text{\,}\mathrm{e}$lectron-hole
pairs in the bulk. The total charge induced on the strips’ coupling capacitors
and on the pitch adapter metal (before any shaping) as a function of the
integration time are shown in Fig. 7 for two variants of PA position. The
reference shape corresponds to the pulse from a particle passing directly
under a strip. When the PA pads are centered with respect to the inter-strip
region, slower collection on the nearby strips is observed with respect to the
reference while a pick-up signal on the PA emerges with a peaking time of
approximately $5\text{\,}\mathrm{ns}\text{/}$ (Fig. 7(a)). In the case where
the PA pads are shifted $10\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$,
resulting in a distance to the farther strip of
$12.5\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$, the pick-up signal on
the PA becomes more similar to the reference while the signal on the strips is
reduced significantly. All the charge eventually reaches the strips on the
order of microseconds. This case is likely to result in a localized region of
inefficiency.

(a) Centered PA

(b) PA shifted $10\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$

Figure 7: Simulated charge appearing on the strip AC and PA contacts versus
time since MIP hit in the inter-strip region
$5\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ closer to one of the strips.
The reference curve corresponds to charge on the strip AC contact in response
to a MIP passing under the strip. To account for charge sharing, the sum of
the charge collected on both strips is used.

In order to qualitatively compare with data from beam tests, a simple shaping
function of the form:

$W(t)=\frac{t}{\tau}e^{-t/\tau},$ (1)

has been applied to the charges, where $\tau$ is a shaping time of
$25\text{\,}\mathrm{ns}\text{/}$. The results are shown in Fig. 8. When the PA
is centered, the slowed development of the signal on the nearby strips should
be visible as a shift in the time of the pulse peak. The pick-up signal on the
PA resembles a very large cross-talk signal. When the PA shifts, the signal on
the nearby strips is greatly reduced. The PA then sees a pulse similar in
strength to a real signal.

(a) Centered PA

(b) PA shifted $10\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$

Figure 8: Simple pulse shapes produced from the simulated charge appearing on
the strip AC and PA contacts versus time since MIP hit in the inter-strip
region $5\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ closer to one of the
strips. The reference curve corresponds to the one in Fig. 7. To account for
charge sharing, the sum of the charge collected on both strips is used. The
vertical lines denote a $25\text{\,}\mathrm{ns}\text{/}$ interval, motivated
by the time window imposed by the data acquisition system on data from the
beam test.

##  3 Beam test

The sensors have been tested using beams delivered from the CERN SPS to the H8
beam line in the CERN North Area. This beam delivers positively charged
hadrons with a momentum of $180\text{\,}\mathrm{GeV}\text{/}$. The width of
the beam is approximately $1\text{\,}\mathrm{cm}\text{/}$, allowing the beam
to simultaneously illuminate the pitch adapter area and the edges of the
sensor active area.

###  3.1 Experimental setup

The sensor, referred to as the device under test (DUT), is placed at the
center of the Timepix3 telescope; an earlier iteration of this telescope is
described in Ref. [11]. This telescope consists of 8 Timepix3 pixel modules
used to reconstruct the trajectory of passing charged particles. These tracks
are extrapolated to where the DUT is placed with a pointing resolution of
approximately $2\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$. This allows
for a precise determination of the impact points of the beam particles with
respect to the sensor structures like the strip implants and pitch adapter.
The beam impacts the DUT perpendicularly.

The sensors are read out using the Beetle ASIC [12]. This chip outputs the
signals on every channel as an analog pulse with a shaping time on the order
of tens of $\mathrm{ns}\text{/}$. These signals are processed using the MAMBA
data-acquisition (DAQ) board produced by Nuclear Instruments [13]. This DAQ
system samples and digitizes the analog pulse at a fixed time that can be
moved in $25\text{\,}\mathrm{ns}\text{/}$ increments. The beam particles
arrive asynchronously with the $40\text{\,}\mathrm{MHz}\text{/}$ clock used
for readout. This allows for the study of particles arriving at all times
within the window.

A copy of the scintillator signal used for triggering the DUT is fed into the
data stream of the Timepix3 telescope. A shared time-stamp is used to match
tracks reconstructed from the Timepix3 hits with the DUT data for the
corresponding trigger.

To test the effects of radiation, sensors were irradiated at the IRRAD [14]
facility using protons with a momentum of $24\text{\,}\mathrm{GeV}\text{/}$.
Dosimetry measurements were used to determine the particle fluence in the
$3\text{\,}\mathrm{mm}\text{/}$ band surrounding the pitch adapter. Sensors
with six different fluences ranging from zero to
$2\text{\times}{10}^{13}\text{\,}\mathrm{1~{}\mega~{}\text{neutron
equivalent}\per\centi\squared}\text{/}$ were used in our tests. The latter
fluence represents the maximum expected for UT sensors with these pitch
adapters after 50$\mbox{\,fb}^{-1}$ of data collected in LHCb.

Miniature sensors from this study were found to have signal-to-noise ratios of
at least 20. Further information on similar beam tests conducted with
prototype sensors from the same project may be found in Ref. [15].

###  3.2 Data sample and analysis

In order to isolate the effects of the pitch adapter and compare to
simulation, specific beam events are chosen in which the passing particle goes
through the bond pads of the pitch adapter. Only events in which a single
track is found in the Timepix3 telescope are selected. Using the coordinates
from tracks extrapolated to the DUT position, the side and top edges of the
sensor are determined. The position of the track in the local $x$
(perpendicular to each sensor strip) and local $y$ (parallel to each strip)
directions is then calculated. Results from multiple runs with the same
sensor, taken in a short period of time, are combined together, but the
position calibration is repeated for each run independently.

Fourteen strip pairs have been identified from the sensor design for which a
single PA bond pad is found over the inter-strip region. Some of these pads
are centered between the two strips, and some are shifted towards one side or
the other. Tracks passing within $19\text{\,}\mathrm{\SIUnitSymbolMicro
m}\text{/}$ of the center of the inter-strip region, at the correct range of
local $y$ values to be under the bond pad are identified. These 14 rectangular
regions together make up the “PA region”. For each strip pair, the channel
that is connected to the PA bond pad in that region is also identified. The
output for these three channels are then studied together to analyze the
effect of the pitch adapter.

For comparison, a control region is defined using the inter-strip regions of
the same 14 strip pairs. The control region requires the track to pass away
from the PA region by requiring values of local $y$ farther down the sensor.
The signal response in the control region has been validated to be free of
pitch adapter effects by comparing it with tracks that pass directly
underneath a single strip.

For each event in the sample, the ADC counts for the two channels
corresponding to the nearby strips as well as the channel connected to the PA
bond pad are studied. The ADC values used are taken after pedestal and common-
mode noise subtraction are performed. Events in which one of the three strips
(nearby or PA connected) is flagged as noisy or dead are removed from the
analysis.

The data output also includes a timing value for the event. The
$25\text{\,}\mathrm{ns}\text{/}$ readout window is divided into ten
$2.5\text{\,}\mathrm{ns}\text{/}$ time bins, corresponding to different
trigger arrival times relative to the $40\text{\,}\mathrm{MHz}\text{/}$
readout clock. The data in each time bin is therefore sampled from the analog
Beetle output pulse at different times relative to the start time of the
pulse. This allows us to reconstruct the shape of the output pulse inside the
$25\text{\,}\mathrm{ns}\text{/}$ window.

##  4 Results from beam test

Two dimensional views of the sampled ADC versus time are shown in Fig. 9 for
the PA and control regions. The control region shows a single clear peak in
the signal pulse as a function of time. Two qualitatively different signal
patterns are clear in the data from the PA region. In some events, there is a
clear signal peak on the two closest strips similar to the control region,
while in others the nearby strips show a value close to zero. In the latter
case, the resulting data would often fail clustering requirements to separate
signal from noise and the hit would be missed. These two classes of events are
separated by requiring the sum ADC on the two nearby strips to be less than or
greater than 150. Since the signal is split almost evenly between the two
strips, this is approximately equivalent to a cut of three times the noise
applied to each strip separately.

(a) Control region

(b) PA region

Figure 9: Sum of the ADC values for the two strips near the track location
versus time for an unirradiated sensor. The tracks in the control region are
shown in (9(a)) and in the PA region in (9(b)).

To reconstruct the pulse shape for each class of events, the peak ADC value is
first determined separately for each time bin. The pulse is reconstructed
separately for nearby strips and the PA connected strip. For the nearby
strips, the ADC values of the two channels are added together before finding
the peak. Three different methods for peak finding are used:

  * •

For good signals ($\text{ADC}>150$) on the nearby strips, the ADC distribution
is fitted with a Landau convolved with a Gaussian. The peak position is taken
from the most probable value.

  * •

False “signals” on the PA connected strip show a different ADC distribution
without the characteristic high-side tail of a Landau distribution, but this
may be due to the small number of recorded events. For these outputs, the mean
value from a Gaussian fit is used to find the peak position.

  * •

For channels without clear signals (cross-talk and inefficient hits), the mean
of the ADC values is taken as the peak position.

Once the peak value in each bin has been determined, the resulting points are
fit with a quadratic function to qualitatively describe the shape of the pulse
within the $25\text{\,}\mathrm{ns}\text{/}$ window. The results are shown in
Fig. 10. The two cases of a “good” signal ($\text{ADC}>150$) found or not
found on the nearby strips are compared. In the pitch adapter region, good
signals are slowed in time and reduced in magnitude relative to the control
region pulse; the peak occurs approximately $5\text{\,}\mathrm{ns}\text{/}$
later. Simultaneously, a large cross-talk like signal appears on the PA
connected strip, already decreasing in the sampled time window. These results
match qualitatively the expectation from simulation.

(a) With “good” signal

(b) No “good” signal

Figure 10: Reconstructed pulse shapes within the
$25\text{\,}\mathrm{ns}\text{/}$ readout window for particles passing halfway
between two strips. The pulses in the control and PA regions represent the sum
of the output on the two nearby channels. In (10(a)), a good signal is found
on the nearby strips when the particle lands in the PA region; in (10(b)) a
good signal is not found.

When the signal on the nearby strips is lost, a pulse similar in shape to a
good signal is instead found on the PA connected strip. The nearby strips show
only a small signal above zero. The source of these events can be determined
by further subdividing the data sample. For two of the 14 strip pairs, the PA
bond pad is centered above the inter-strip region. For six pairs it is shifted
by approximately $10\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ towards
one side, and the other six towards the other side. In Fig. 11, the fraction
of missing hits for each of these three groupings is shown as a function of
the inter-strip position normalized by the strip pitch ($\Delta x/P$). For the
centered bond pads, no inefficiency is found. The shifted bond pads show an
inefficiency in the opposite side of the inter-strip region which is
consistent with the expectation from simulation. The local inefficiency
reaches approximately 50% at the highest. The overall effect averaged over the
sensor is small since the region of inefficiency is limited to this small
area.

Figure 11: Fraction of particles with missing signals on the nearby channels
as a function of the inter-strip position for particles passing through the PA
region. $\Delta x/P=0.5$ corresponds to halfway between the two strips. Three
groups of channels are shown, corresponding to cases in which the PA bond pad
is either centered between the two strips, or shifted towards one side or the
other.

We find that the effects of the PA are reduced with irradiation, as shown in
Fig. 12. By a dose of $6.4\text{\,}\mathrm{kGy}\text{/}$ in the silicon
dioxide layer, no PA coupling effects are found. At lower levels of
irradiation, the effects on the nearby strip signals and on the PA connected
channel are reduced but not totally eliminated. With the measurements
available on these sensors, we are unable to determine precisely the level of
charge build-up in the oxide layer as a function of dose.

(a) Near channels

(b) PA connected channel

Figure 12: Pulse shapes for three sensors with different radiation doses in
the silicon dioxide layer. The radiation doses are calculated based on
ionization in the silicon dioxide by the protons used for irradiation. As
irradiation increases, the PA coupling effect is reduced, both in the delay in
the signal observed in the nearest two channels (12(a)) and in the PA
connected channel signal (12(b)).

##  5 Conclusions

We have observed effects on signal collection due to the presence of embedded
pitch adapters over the active sensor area in n-substrate silicon microstrip
sensors. Some pitch adapter geometries result in a loss of signal on the
nearest strips when a charged particle passes underneath the second metal
layer. This results in an inefficiency for particle detection after thresholds
are applied to reject noise. Simultaneously, a signal-like pulse is found on
the channel connected to the pitch adapter element.

Using simulation, we are able to understand these effects in more detail. The
presence of the second metal layer over top of the region between two implant
strips alters the electric fields inside the bulk of the silicon. When the
metal layer is centered over the inter-strip region, this results in a slower
time evolution of the signal pulse. We observe this change in timing in our
data. The channel connected to the pitch adapter experiences a large cross-
talk signal, which we also observe.

When the pad in the second metal layer is shifted relative to the inter-strip
region, a potential barrier can form that impedes the collection of signal
over a time scale long enough to cause an inefficiency. When this occurs, the
pitch adapter connected channel outputs a larger pulse that is easily mistaken
for a true hit. The inefficient hits in our data are limited to cases in which
the pitch adapter bond pad is shifted by approximately
$10\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$.

We have observed that these problems disappear with irradiated sensors. This
could be explained by the build up of positive charge at the silicon
dioxide–silicon interface, which is known to occur with irradiation [8].
Simulation shows that a surface charge density around
$1\text{\times}{10}^{12}\text{\,}\mathrm{\text{ $e$
}}\text{/}{\mathrm{cm}}^{2}$ should suffice. It is plausible that the
irradiation of the sensors used for the beam test has produced this charge
density, although we were not able to directly measure this charge.

We have investigated in simulation scenarios that may reduce the effects of
the PA. We predict that doubling the thickness of the silicon dioxide layer
from $2\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ to
$4\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ should remove the
inefficiency and pick-up signals we have observed.

All of these effects are localized to the small regions directly underneath
pitch adapter metal and halfway between two strips. For these sensors, we only
observe problems near the larger bond pads; no significant effect is seen
under the thin traces. For all particles passing through other parts of the
detector, we observe no effects on signal formation due to the pitch adapters.

Similar effects have also been observed in p-substrate sensors. Detailed
understanding of the effects in these sensors requires future beam test and
simulation efforts.

##  6 Acknowledgements

We would like to express our gratitude to Hamamatsu for producing the sensors
used in this study. We thank the CERN accelerator departments for the
excellent performance of the SPS which provided the beam used in our tests. We
also thank the VELO Timepix telescope group for the excellent instrument that
made the precision studies possible. The IRRAD facility at CERN also provided
valuable help irradiating the tested sensors. We acknowledge the support of
the National Science Foundation for this work.

## References

  * [1] M. Ullan et al., _Embedded pitch adapters for the ATLAS tracker upgrade_ , Nucl. Instrum. Meth. A732 (2013) 178 , Vienna Conference on Instrumentation 2013
  * [2] LHCb collaboration, _LHCb Tracker Upgrade Technical Design Report_ , CERN-LHCC-2014-001.  LHCb-TDR-015
  * [3] M. Ullan et al., _Embedded pitch adapters: A high-yield interconnection solution for strip sensors_ , Nucl. Instrum. Meth. A831 (2016) 221 , Proceedings of the 10th International “Hiroshima” Symposium on the Development and Application of Semiconductor Tracking Detectors
  * [4] W. de Boer et al., _A fourfold segmented silicon strip sensor with read-out at the edges_ , Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 788 (2015) 154
  * [5] Hamamatsu Photonics. http://www.hamamatsu.com
  * [6] Synopsys. http://www.synopsys.com
  * [7] R. Eber, Investigations of new Sensor Designs and Development of an effective Radiation Damage Model for the Simulation of highly irradiated Silicon Particle Detectors, PhD thesis, Karlsruhe Institute of Technology, 2013, IEKP-KA/2013-27
  * [8] E. H. Nicollian and J. R. Brews, MOS (Metal Oxide Semiconductor) Physics and Technology, John Wiley and Sons, New York, 1982
  * [9] J. Schwandt et al., _Surface effects in segmented silicon sensors_ , Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 845 (2017) 159 , Proceedings of the Vienna Conference on Instrumentation 2016
  * [10] D. Pennicard, _Introduction to Sentaurus TCAD_ , http://twiki.ppe.gla.ac.uk/pub/DetDev/SynopsysTCAD/Introduction_to_Sentaurus_TCAD.ppt, slide 14
  * [11] K. Akiba et al., _The Timepix telescope for high performance particle tracking_ , Nucl. Instrum. Meth. A723 (2013) 47, arXiv:1304.5175
  * [12] Löchner, S. and Schmelling, M. , _The Beetle Reference Manual - chip version 1.3, 1.4 and 1.5_ , Tech. Rep. LHCb-2005-105, 2006
  * [13] Nuclear Instruments. http://www.nuclearinstruments.eu
  * [14] B. Gkotse et al., _A New High-intensity Proton Irradiation Facility at the CERN PS East Area_ , in Technology and Instrumentation in Particle Physics 2014, p. 354, Proceedings of Science, June, 2014
  * [15] A. Abba et al., _Study of prototype sensors for the Upstream Tracker Upgrade_ , LHCb-PUB-2016-007


# UIT-HWDB: Using Transferring Method to Construct A Novel Benchmark for
Evaluating Unconstrained Handwriting Image Recognition in Vietnamese

Nghia Hieu Nguyen1,2, Duong T.D. Vo1,2, Kiet Van Nguyen1,2
1University of Information Technology, Ho Chi Minh City, Vietnam
2Vietnam National University, Ho Chi Minh City, Vietnam
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recognizing handwriting images is challenging due to the vast variation in
writing style across many people and distinct linguistic aspects of writing
languages. In Vietnamese, besides the modern Latin characters, there are
accent and letter marks together with characters that draw confusion to state-
of-the-art handwriting recognition methods. Moreover, as a low-resource
language, there are not many datasets for researching handwriting recognition
in Vietnamese, which makes handwriting recognition in this language have a
barrier for researchers to approach. Recent works evaluated offline
handwriting recognition methods in Vietnamese using images from an online
handwriting dataset constructed by connecting pen stroke coordinates without
further processing. This approach obviously can not measure the ability of
recognition methods effectively, as it is trivial and may be lack of features
that are essential in offline handwriting images. Therefore, in this paper, we
propose the Transferring method to construct a handwriting image dataset that
associates crucial natural attributes required for offline handwriting images.
Using our method, we provide a first high-quality synthetic dataset which is
complex and natural for efficiently evaluating handwriting recognition
methods. In addition, we conduct experiments with various state-of-the-art
methods to figure out the challenge to reach the solution for handwriting
recognition in Vietnamese.

###### Index Terms:

Vietnamese handwriting recognition, unconstrained handwriting recognition,
Vietnamese handwriting image dataset, synthetic dataset

##  I Introduction

Handwriting Recognition has been a challenging task in image processing and
computational linguistics. This task is traditionally divided into two main
categories: online handwriting recognition and offline handwriting
recognition. Because of the linguistic aspect and low-resource status of
Vietnamese, recognizing Vietnamese handwriting images is still an arduous
task. As collecting handwriting images is problematic, there are not many
large-scale and high-standard dataset for researching handwriting recognition
in Vietnamese.

Previous works [1, 2] attempted to synthetically conduct offline handwriting
datasets by using handwriting fonts to render images. However, datasets
constructed by these approaches is not guaranteed to be natural as (1) for the
approach proposed in [2], the stable form of computer fonts opposes to the
various handwriting styles of human, which obviously makes font-rendered
handwriting images have different distribution to realistic handwriting
images; and (2) for the approach proposed in [1], the colors of strokes and
background are not considered carefully, and re-arranging segmented characters
to a defined line does not clearly express the unconstrained properties of
realistic handwriting images. To this end, we analysed and conducted
experiments and then propose a simple but effective method, called
Transferring method, to construct an offline handwriting dataset from an
online handwriting dataset. Using this method, we introduce the first offline
handwriting dataset in Vietnamese in order to motivate research community to
explore and conduct more experiments to ultimately create an effective
framework to tackle offline handwriting recognition task in Vietnamese.

##  II Related works

Marti et al. [3] constructed the IAM dataset by creating forms containing
computer-printed texts for volunteers to write down these texts in their own
writing style. After collecting these filled forms, line segmentation
algorithms and word segmentation algorithms were used to extract lines from
handwritten texts and words from extracted handwritten lines. On the other
hand, the RIMES dataset [4] was constructed based on the content of mails. In
both cases, a large number of annotators were called in to annotate the
datasets, and the annotation processes were re-corrected many times for the
quality assurance.

Motivated by previous works and the shortage of Vietnamese offline handwriting
dataset, we propose a novel method, called Transferring method, to construct
an offline handwriting dataset in Vietnamese by synthesizing fundamental
attributes from different handwriting datasets. Our conducted experiments
ensure the resulting dataset totally keep mainly essential properties of an
offline handwriting image dataset, which bridge the gap between handwriting
images captured in the real world and images constructed synthetically.
Consequently, we introduce a novel offline handwriting image dataset in
Vietnamese constructed using the Transferring method, named UIT-HWDB.

##  III Challenge from writing characteristics of Vietnamese language

Like other Western languages, Vietnamese uses the modern Latin script for its
writing system. However, there is a clear difference that Vietnamese relies
heavily on diacritics, which makes Vietnamese handwriting more complicated in
a distinct way. In the Vietnamese alphabet, there are seven letters that
always have their attached diacritics: _ă_ (breve), _â_ , _ê_ , _ô_
(circumflex), _ơ_ , _ư_ (horn) and _đ_ (bar); and five additional diacritics
used to designate tone: grave (as in _à_), acute (as in _á_), hook above (as
in _ả_), tilde (as in _ã_), and dot below (as in _ạ_) [5]. When being
scrawled, some diacritics may be mistakenly read or seen as others, especially
for computer vision systems. Specifically, a hook above glyph may look like a
horn glyph (_ủ_ versus _ư_), an acute glyph may be carelessly written to be
like a grave glyph (_á_ versus _à_), and a dot below under a letter _i_ may
make the character _ị_ look like an exclamation mark (_ị_ versus _!_). The
examples above, and many more mistakes caused by carelessly written
diacritics, frequently happen in real life and be an obstacle for optical
character recognition for Vietnamese handwriting. Those flaws are also what we
have encountered in our research, and we will analyze how the models stumble
through them in Section 5.

##  IV UIT-HWDB Dataset

###  IV-A Transferring method

We present in this section the Transferring method to synthetically construct
a Vietnamese handwriting image dataset. The novel constructed dataset must
have the fully essential properties of a realistic handwriting image dataset.
To achieve this requirement, the Transferring method first constructs the
handwritten characters into an image, then applies the color variation to make
the image more natural. The way of considering these real-world factors is
detailed as follows:

Algorithm 1 Pseudocode for the Transferring method ($U(a,b)$ is the discrete
uniform distribution in range $[a,b]$, $Beta(\alpha,\beta)$ is the beta
distribution).

  * •

$coords=({x_{n},y_{n}})_{n\in\mathbb{N}}$: the sequence of coordinates of a
pen stroke.

  * •

$all\\_strokes$: the sequence of coordinates sequences of all strokes in the
image.

  * •

$bg\\_dis=({\alpha_{m},\beta_{m}})_{m\in\mathbb{N}}$: the sequence of $m$
pairs of parameters for approximated beta distributions of background color.

  * •

$stroke\\_dis=({\alpha_{m},\beta_{m}})_{m\in\mathbb{N}}$: the sequence of $m$
pairs of parameters for approximated beta distributions of stroke color.

a handwriting image containing characters drawed by the input coordinates.

procedure render_image($all\\_strokes$)

$image\leftarrow 255$ for all pixels

for $coords$ in $all\\_strokes$ do

for $ith$ in $[1:length(coords)]$ do

$image[coords_{ith-1}:coords_{ith}]\leftarrow 0$

end for

end for

return image

end procedure

procedure render_color(image)

$stroke\\_ps\leftarrow[]$ $\triangleright$ Sequence of stroke color variables

$bg\\_ps\leftarrow[]$ $\triangleright$ Sequence of background color variables

for $ith$ in [1:m] do

$\alpha,\beta\leftarrow stroke\\_dis[ith]$

$stroke\\_ps.append(X\sim Beta(\alpha,\beta))$

end for

for $ith$ in [1:m] do

$\alpha,\beta\leftarrow bg\\_dis[ith]$

$bg\\_ps.append(X\sim Beta(\alpha,\beta)$)

end for

$ith\sim U(1,m)$ $\triangleright$ Randomly use $ith$ random variable,
regarding both stroke and background color

for each pixel $p$ in image do

if $p$ is 0 then $\triangleright$ Is stroke pixel

$p\leftarrow stroke\\_ps[ith]\times 255.0$

else$\triangleright$ Is background pixel

$p\leftarrow bg\\_ps[ith]\times 255.0$

end if

end for

end procedure

$image\leftarrow RENDER\\_IMAGE(all\\_strokes)$

$image\leftarrow RENDER\\_COLOR(image)$

####  IV-A1 Human handwriting style

The handwriting style is one of the specific properties that distinguish
handwritten text from computer-printed text. Moreover, the variation of
handwriting style is the most important characteristic required for a
handwriting dataset, both online and offline.

Previous works [2] automatically used handwriting fonts to construct offline
handwriting images, which causes the gap between natural handwriting images
and synthetic handwriting images. To bridge this gap, we inherit handwritten
characters from an online handwriting dataset, the VNOnDB [6]. This approach
guarantees the vast variation in natural writing style of human.

####  IV-A2 Color

Recent works [7, 8, 9] constructed handwriting images in Vietnamese by
connecting coordinates of pen stroke in the VNOnDB dataset [6]. They kept the
ink color as black (0 for pixel value) and the background color as white (255
for pixel value). This approach is intuitively not natural. Ignoring the color
factors also means that we unintentionally reduce the complexity of the image
compared with an image captured in reality, which causes the gap between
research and realistic applications. To address these downsides, we take into
account the complexity of the color.

For more details, we analyze the colors from the IAM dataset [3]. The IAM
dataset was collected manually, containing the real-world variation of both
stroke colors and background colors. Therefore, the distributions of color in
this dataset are natural enough to bridge the gap between synthetic images and
real-world captured images. In the IAM dataset, for each image subset we
intend to sample the stroke color values and background color values to get
the corresponding color sampling distributions. We find that the color
distributions have a finite range from 0 to 255 and have variously flexible
shapes and skewnesses. After visualizing the color sampling distributions and
taking into account the aforementioned statement, we finally come up with
approximating the beta distribution for stroke colors and background colors of
each subset. We then calculate and have a list of $\alpha$ and $\beta$ shape
parameters for the beta distribution of the stroke colors and background
colors. We apply to the strokes and background of handwriting images with
random colors that follow the stroke and background color beta distribution
which were generated from randomly picked pairs of $\alpha$ and $\beta$
parameters from the list. In this way, we ensure the nature of colors for
handwriting images created synthetically.

####  IV-A3 Stroke-width

Although most synthetic handwriting datasets are constructed in various ways
that resemble images captured in the real world, they still miss the stroke-
width variation. But in our work, experiments show that this factor is not
essential and can be slightly ignored to keep the Transferring method simple
without reducing the complexity of the dataset. We will carefully analyze this
statement in Section 5.

####  IV-A4 Annotation

Ground truth texts for the dataset are inherited from the ground truth texts
of the VNOnDB dataset [6]. However, from VNOnDB, there are some incorrect
labels for handwriting images. Therefore, we also re-correct their mislabeled
ground truth texts during the image constructing process.

###  IV-B UIT-HWDB dataset

Using the Transferring method with the VNOnDB as the online handwriting
dataset and the IAM as the source for extracting colors, we construct the
first novel offline handwriting dataset, the UIT-HWDB dataset. In more detail,
our dataset has two parts: UIT-HWDB-word (110,745 unconstrained handwritten-
word images) and UIT-HWDB-line (7,273 unconstrained handwritten-line images).

TABLE I: Statistical comparison between handwriting image datasets (Para. stands for Paragraph, * indicates dataset in English, ** indicates dataset in French, + indicates dataset in German, - indicates dataset in Latin, ++ indicates dataset in Vietnamese). Dataset | Words | Lines | Para.
---|---|---|---
RIMES** | 83,493 | 11,365 | 1,500
IAM* | 115,320 | 13,353 | -
Washinton Database* | 4,894 | 656 | -
Parzival Database+ | 23,487 | 4,477 | -
Saint Gall Database- | 11,597 | 1,410 | -
UIT-HWDB (ours)++ | 110,745 | 7,273 | -

We additionally reconstruct the test sets for the UIT-HWDB dataset as the
original test sets of the VNOnDB generally contain easy-to-read images, which
can lead to over-confidence in the ability of handwriting recognition methods.
Specifically, we manually select both easy-to-read and hard-to-read
handwriting images for our test set to evaluate the ability of recognition
methods on many levels. The dataset is available at
https://github.com/hieunghia-pat/UIT-HWDB-dataset for research purposes.

####  IV-B1 Human evaluation

Our Transferring method is a semi-automatic method, which is undoubtedly prone
to potential errors while constructing images. To carefully evaluate the
errors of our Transferring method, we randomly collect 1,200 images from the
UIT-HWDB-word train set and 600 images from the UIT-HWDB-line train set, then
invite four people to manually evaluate the images (300 word-level images and
150 line-level images for each person). For the evaluation metric, we use the
Cohen’s kappa. As our method keeps the handwriting style, background color and
stroke color consistent, the main factor that can cause the rendered
characters to be abnormal is the choice of stroke width (detailed in section
5). Therefore the guideline for the four people to evaluate the images is to
observe whether the rendered characters are deformed because of the improperly
applied stroke width (stroke-width error) or the handwriting process of the
annotators (non-stroke-width error). Results show the annotators’ agreement
for non-stroke-width error with Cohen’s kappa coefficient $\mathbb{\kappa}=1$
and $0$ for stroke-width error on the three collected sets. Our Transferring
method totally keeps the original shape of characters.

##  V Experiments

###  V-A Baseline methods

The early approach for offline handwriting recognition is the combination of
Convolutional Neural Network (CNN) [10], Recurrent Neural Network (RNN) [11],
and Connectionist Temporal Classification (CTC) loss [12], which we call CTC-
based methods. [13] proposed another combination using attention module [14]
in their RNN layers. To enhance the ability of attention mechanisms in
handwriting recognition methods, [15] proposed transformer-based methods,
which have inherited the recent success of the transformer architecture in
Natural Language Processing (NLP). In this experiment, we use TransformerOCR
[16] and NRTR [17] as the transformer-based methods. For attention-based
methods, we implement Attention-based Encoder Decoder (AED) [8] and finally we
use CRNN [18] and a method proposed in [19] (BiCRNN for short) as the CTC-
based methods.

###  V-B Evaluation metric

For the Optical Character Recognition task, Damerau–Levenshtein Distance is
the widely-used metric for measuring the performance of recognition methods.
In this work, we use both two instances of this distance: Character Error Rate
(CER) and Word Error Rate (WER).

###  V-C Experiments on the stroke-width factor

In this experiment, we implemented the baseline methods on the word-level and
line-level images.

To carefully analyze the effect of stroke-width factor on the ability of
current handwriting image recognition methods, we created two versions for the
UIT-HWDB-line dataset and UIT-HWDB-word dataset, named line-v1 and line-v2
(word-v1 and word-v2, respectively). The first version of these two datasets
contains handwriting images without any variation in stroke width, while those
in the second version contain the variation in the width of handwritten
strokes. line-v1 and line-v2 are used to investigate the ability of
TransformerOCR (transformer-based method) and BiCRNN (CTC-based method), while
word-v1 and word-v2 are used to analyze the ability of NRTR (transformer-based
method) and CRNN (CTC-based method). Note that we padded blank pixels to
images in all training schemes in order to let all images have the same size
and keep their original scale.

To mimic the variation of stroke-width factor, we approximate the width of pen
stroke following the assumption: stroke-width is thinner when the writers draw
up than when they draw down. Therefore, based on the ratio between the
difference of two horizontal coordinates $|\Delta x|$ and the difference of
two vertical coordinates $|\Delta y|$ of two adjacent points, we construct a
function $w$ which approximates the stroke width of handwritten characters:

$w(\theta)=m\cdot d(\theta)$ (1)
$d(\theta)=\displaystyle\frac{1}{1+e^{\alpha\theta+\beta}}$ (2)

In this experiment, we set $\alpha=-0.1$, $\beta=1.13$,
$\theta=\arctan{\displaystyle\frac{\Delta y}{\Delta x}}$ and $m$ is the
maximum thickness value a stroke can have.

To construct images with constant stroke width, we set for $\theta$ a constant
value which makes the function $d(\theta)$ close to 1.

In practice, we observed the proper range for getting $m$ is $[2,5]$. For the
UIT-HWDB dataset, we randomly selected a value for $m\sim U(2,5)$ where
$U(a,b)$ is the discrete uniform distribution in range $[a,b]$ for every
image.

TABLE II: Results of TransformerOCR and BiCRNN on two versions of UIT-HWDB-line dataset (the first two rows were trained on line-v1 and last two rows were trained on line-v2). | line-v1 | line-v2
---|---|---
| CER | WER | CER | WER
TransformerOCR | 11.59 | 21.38 | 11.42 | 21.67
BiCRNN | 12.06 | 31.84 | 12.28 | 31.49
TransformerOCR | 16.88 | 30.61 | 12.18 | 22.33
BiCRNN | 11.50 | 29.78 | 10.77 | 27.54

The experimental results are shown in Table II. According to Table II, when we
trained TransformerOCR and BiCRNN on line-v1 and then evaluated these two
models on both line-v1 and line-v2 test sets, we recognized they had
approximately the same results. However, when we trained these models on
line-v2 and then evaluated them on the line-v1 test set and line-v2 test set,
they performed well on the line-v2 but yielded a few drawbacks on the line-v1.
In conclusion, these results showed that with their strong CNN structures,
TransformerOCR and BiCRNN, when being trained on version 1, can generalize
well the pattern of handwritten text in images, hence they concurrently
yielded the same results when predicting on the test set of version 2.

TABLE III: Results of NRTR and CRNN on the two versions of the UIT-HWDB-word dataset (the first two rows were trained on word-v1 and last two rows were trained on word-v2). | word-v1 | word-v2
---|---|---
| CER | WER | CER | WER
NRTR | 8.28 | 22.00 | 10.10 | 25.48
CRNN | 9.64 | 20.27 | 10.74 | 23.26
NRTR | 18.01 | 37.73 | 8.32 | 21.21
CRNN | 16.66 | 31.93 | 9.77 | 20.62

Nevertheless, coming from NRTR and CRNN, we obtained another behavior. As
Table III indicates, these two models only performed well on the version of
the dataset that they had been trained. These results show that CRNN and NRTR
can catch the specific attributes of each version of the dataset; thus, they
are quite sensitive when making a prediction on another version of the images.

Finishing these experiments, we conclude that with current deep learning
methods, stroke-width does not participate in forming the complexity of
offline handwriting recognition task. Therefore we can ignore this factor to
keep our Transferring method simple without lacking the nature of a
handwriting image dataset.

###  V-D Experiments on baseline methods

TABLE IV: Results of baseline methods on UIT-HWDB-word and UIT-HWDB-line test set. | UIT-HWDB-word | UIT-HWDB-line
---|---|---
Model | CER | WER | CER | WER
CRNN | 9.93 | 20.76 | 31.23 | 100.00
BiCRNN | 8.07 | 18.08 | 11.76 | 30.94
NRTR | 8.25 | 21.31 | 49.72 | 91.70
TransformerOCR | 5.29 | 10.37 | 11.42 | 21.67
AED | 6.70 | 14.68 | 12.53 | 35.31

We evaluated the baseline methods on UIT-HWDB-word and UIT-HWDB-line
containing word-level and line-level handwriting images, respectively. Our
results are shown in Table IV.

As shown in Table IV, both NRTR and CRNN obtained bad results on the UIT-HWDB-
line dataset. This is because CRNN and NRTR were originally proposed to
predict cropped-word scene text images, hence their backbone (CNN structure)
were designed to be simple in order to enhance their performance [18, 17]. In
contrast, Puigcerver et al. [19] and Feng et al. [16] designed the BiCRNN and
the TransformerOCR with every deep CNN structures. Therefore, these two models
outperform NRTR and CRNN on recognizing both line-level and word-level
handwriting images.

###  V-E Results analysis of the baseline methods

As the results in our experiments, TransformerOCR achieved the best results
among other methods on both the word-level and line-level dataset. Therefore,
we focus on analyzing the common mistakes that TransformerOCR suffered.

Figure 1: Wrong-number prediction because of the low frequency of numbers in
training dataset. Figure 2: Examples for failed cases that the Transformer can
not read scribbled characters.

After observing the images in the test set and the respective prediction of
the TransformerOCR, we conducted the two types of error of the TransformerOCR
which are (1) the wrong prediction of number characters and (2) the
misunderstanding of scrawled handwritten characters.

For the first mistake (Figure 1), as the domain of the UIT-HWDB dataset is
daily newspaper, the occurrence of numerical characters is lower than of
alphabet characters, which causes the model to not learn enough to make the
correct decision when facing these numerical characters. Statistically, the
frequency of numerical characters (including Roman numerical characters) in
the UIT-HWDB-word train set is approximately 0.065 and in the UIT-HWDB-word
test set is approximately 0.0066 (we calculated these frequencies on the word-
level dataset because the line-level is the line-segmented version of
paragraph-level, and the word-level is the word-segmented version of the line-
level, which indicates the frequency of number characters in the word-level
dataset is the frequency of them in the line-level and paragraph-level
dataset). But as the low occurrence of number characters on the train set as
well as the test set, this type of mistake is typically not the main factor to
reduce the performance of the TransformerOCR method.

The last type of error is the misunderstanding of scribbled characters which
is the main challenge of recognizing handwriting images as well as the main
challenge of our dataset. As depicted in Figure 2, TransformerOCR totally
failed to achieve the acceptable prediction for images containing scrawled
handwritten characters. For humans, we find these images are hard-to-read, but
this does not mean reading these images is infeasible. The only way to read
these words is to infer them together with meaning of the lines. Therefore we
suggest the insight to improve the TransformerOCR or deep learning methods in
general for this challenge is finding a way to exploit the meaning of the
sentence to make models have better inference.

##  VI Conclusion and Future Works

We have introduced the Transferring method that synthetically forms a novel
handwriting image dataset, which is useful for low-resource languages
including Vietnamese. In addition, we presented a new benchmark for evaluating
handwriting image recognition methods in Vietnamese constructed using this
Transferring method. Finally, we have analyzed the confusion of state-of-the-
art methods when performing on our dataset. We have identified that the
appearance of number characters and badly scrawled text in the already
linguistically complicated dataset are the causes behind the shortcomings in
the predictions of the models. In the future, we continue to conduct
experiments with various augmentation methods to tackle the shortage of number
characters in our dataset. Moreover, we will analyze and research the way to
effectively combine the handwriting recognition methods with pre-trained
language models (like BERTology [20]) to make them infer better when making
predictions for scribbled handwriting images.

## References

  * [1] X. Shen and R. Messina, “A method of synthesizing handwritten chinese images for data augmentation.” in _International Conference on Frontiers in Handwriting Recognition (ICFHR)_ , 2016.
  * [2] P. Krishnan and C. Jawahar, “Generating synthetic data for text recognition,” _arXiv preprint arXiv:1608.04224_ , 2016.
  * [3] U.-V. Marti and H. Bunke, “The iam-database: an english sentence database for offline handwriting recognition,” _International Journal on Document Analysis and Recognition_ , vol. 5, pp. 39–46, 2002.
  * [4] E. Grosicki and H. El-Abed, “Icdar 2011 - french handwriting recognition competition,” in _2011 International Conference on Document Analysis and Recognition_ , 2011, pp. 1459–1463.
  * [5] D. Truong, _Vietnamese Typography_. Blurb, Incorporated, 2015. [Online]. Available: https://books.google.com.vn/books?id=8I4tjwEACAAJ
  * [6] H. T. Nguyen, C. T. Nguyen, P. T. Bao, and M. Nakagawa, “A database of unconstrained vietnamese online handwriting and recognition experiments by recurrent neural networks,” _Pattern Recognition_ , vol. 78, pp. 291–306, 2018.
  * [7] A. D. Le, H. T. Nguyen, and M. Nakagawa, “End to end recognition system for recognizing offline unconstrained vietnamese handwriting,” _arXiv preprint arXiv:1905.05381_ , 2019.
  * [8] A. D. Le, H. T. Nguyen, and M. Nakagawa, “Recognizing unconstrained vietnamese handwriting by attention based encoder decoder model,” in _2018 international conference on advanced computing and applications (ACOMP)_. IEEE, 2018, pp. 83–87.
  * [9] V.-L. Ly, T. Doan, and N. Q. Ly, “Transformer-based model for vietnamese handwritten word image recognition,” in _2020 7th NAFOSTED Conference on Information and Computer Science (NICS)_ , 2020, pp. 163–168.
  * [10] K. O’Shea and R. Nash, “An introduction to convolutional neural networks,” _arXiv preprint arXiv:1511.08458_ , 2015.
  * [11] A. Sherstinsky, “Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network,” _Physica D: Nonlinear Phenomena_ , vol. 404, p. 132306, 2020.
  * [12] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 369–376.
  * [13] J. Michael, R. Labahn, T. Grüning, and J. Zöllner, “Evaluating sequence-to-sequence models for handwritten text recognition,” in _2019 International Conference on Document Analysis and Recognition (ICDAR)_. IEEE, 2019, pp. 1286–1293.
  * [14] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _arXiv preprint arXiv:1406.1078_ , 2014.
  * [15] L. Kang, P. Riba, M. Rusiñol, A. Fornés, and M. Villegas, “Pay attention to what you read: Non-recurrent handwritten text-line recognition,” _arXiv preprint arXiv:2005.13044_ , 2020.
  * [16] X. Feng, H. Yao, Y. Yi, J. Zhang, and S. Zhang, “Scene text recognition via transformer,” _arXiv preprint arXiv:2003.08077_ , 2020.
  * [17] F. Sheng, Z. Chen, and B. Xu, “Nrtr: A no-recurrence sequence-to-sequence model for scene text recognition,” in _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , 2019, pp. 781–786.
  * [18] B. Shi, X. Bai, and C. Yao, “An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 11, pp. 2298–2304, 2016.
  * [19] J. Puigcerver, “Are multidimensional recurrent layers really necessary for handwritten text recognition?” in _2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)_ , vol. 01, 2017, pp. 67–72.
  * [20] A. Rogers, O. Kovaleva, and A. Rumshisky, “A primer in bertology: What we know about how bert works,” _TACL_ , vol. 8, 2020.