# Semi-Markov Processes in Open Quantum Systems. III. Large Deviations of
First Passage Time Statistics

Fei Liu<EMAIL_ADDRESS>School of Physics, Beihang University, Beijing
100083, China  Shihao Xia, Shanhe Su  Department of Physics, Xiamen
University, Xiamen 361005, China

###### Abstract

A semi-Markov process method is used to calculate large deviations of first
passage time statistics of counting variables in open quantum systems. The
core formula is an equation of poles. Although it also calculates large
deviations of counting statistics of the same variables, the degrees of the
equation are distinct with respect to the two statistics. Because the former
is usually lower than the latter in the quantum case, analytical solutions for
the first passage time statistics are possible. We illustrate these results
via a driven two-level quantum system and apply them to explore quantum
violations of the classical kinetic and thermodynamic uncertainty relations.

##  I Introduction

In our preceding three papers [1, 2, 3], a semi-Markov process (sMP) method
was developed to investigate the large deviations (LDs) of the counting
statistics in open quantum systems [4, 5, 6, 7]. Our results show that the
method is not only intuitive in terms of the physical picture but also
effective in analyzing and calculating the LDs [8]. In addition, the method
shows a certain flexibility when it is extended to a complex situation in
which random resetting with or without memory is involved in the quantum
processes [3]. When we combine these results with an earlier application of an
sMP in deriving a quantum uncertainty relation [9] and the latest application
in analyzing the dynamics of metastable Markov open quantum systems [10], we
believe that the sMP method still has the potential to explore other new
issues related to nonequilibrium quantum processes. In this paper, we report a
new case: calculating the LDs of the first passage time (FPT) statistics.

In contrast to the LDs of the counting statistics, which describe
nonequilibrium large fluctuations in the counting variables at fixed large
times, the LDs of the FPT statistics describe large fluctuations in time as
the same counting variables reach large thresholds [11, 12, 13]. Although the
two statistics are distinct, Gingrich and Horowitz demonstrated that they are
indeed conjugated; that is, their respective scaled cumulant generating
functions (SCGFs) are inverse functions of each other [14]. Importantly, the
latter can be calculated by finding the largest real eigenvalues of the tilted
generators of the classical or quantum master equation [15, 5, 6, 16]. We want
to emphasize that this conjugated relation does not indicate that to obtain
the LDs of the FPT statistics, solving for the LDs of the counting statistics
is inevitable. Moreover, previous studies have shown that calculating the
SCGFs of the counting statistics is not easy even for simple open quantum
systems because we have to find the largest real roots of higher-order
polynomials [17, 2], which is usually analytically intractable. Therefore,
developing a method for calculating the LDs of FPT statistics that is
independent of solving for the LDs of counting statistics is essential. In
this work, we will show that the sMP method is available for achieving this
aim.

The remainder of this paper is organized as follows. In Sec. (II), we sketch
the sMP method for open quantum systems. In Sec. (III), after defining a
generating function for the LDs of the FPT statistics, we derive an equation
of poles for solving the SCGFs. In Sec. (IV), a driven quantum two-level
system (TLS) is used to illustrate our theoretical results, which include
analytically calculating the SCGFs of the FPT statistics, developing an
inverse function method for calculating the SCGFs of the counting statistics,
and exploring quantum violations of the classical uncertainty relations.
Section (V) concludes this paper.

##  II Semi-Markov process method for open quantum systems

The sMP method has been reviewed several times in previous papers [1, 2, 3].
Here, its introduction is brief. The dynamics of open quantum systems whose
ensemble dynamics are characterized by the Markov quantum master equation [18]
can be unraveled into quantum jump trajectories in single systems [19, 20, 21,
22, 23, 24, 18]. A quantum jump trajectory is composed of deterministic pieces
and random collapses of wave functions. Figure (1) shows this scenario in a
driven TLS: the gray dots in the right column represent quantum states
$|1\rangle$ and $|2\rangle$, and the curves and arrows denote that the wave
functions of a single system depart from or randomly collapse into these
states. If the whole evolution of the wave functions is traced, the trajectory
is essentially Markovian. Nevertheless, if only the collapse events of the
wave functions are considered, which include the collapsed quantum states and
collapse times, the trajectory can be thought of as a realization of a certain
sMP. Under this consideration, the role of the continuous pieces is replaced
by the waiting time distributions (WTDs) of the sMP [25].

Figure 1: Schematic diagrams of a driven quantum two-level system surrounded
by a finite temperature environment and quantum jump trajectories.

##  III Calculating large deviations of first passage time statistics

We consider an open quantum system with $M$ collapsed quantum states. We
assume that a quantum jump trajectory of the system with $N$ collapses starts
with a fixed quantum state $\phi_{\eta}$ and that the last collapsed state is
$\phi_{\gamma}$. We denote it as

$\displaystyle{\bf
X}(t,N)=(\gamma,\alpha_{N-1},\cdots,{\alpha_{2}},{\alpha_{1}},\eta),$ (1)

where $t$ is the duration of the trajectory, ${\alpha_{i}}$ is the type of
collapsed state that occurs at time $t_{i}$, $i=1,\cdots,N$, and
$\alpha_{N}=\gamma$. The time-extensive counting variable in the quantum jump
trajectory is

$\displaystyle C[{\bf X}(t,N)]=\sum_{i=1}^{N}w_{\alpha_{i}\alpha_{i-1}}.$ (2)

where the coefficient $w_{\alpha_{i-1}\alpha_{i}}$ is a weight and is
specified by the collapsed states at the adjacent times $t_{i-1}$ and $t_{i}$.
Note that here, $\alpha_{0}$ is $\eta$. Although the weights are arbitrary, in
practice, they simply equal $\pm 1$ or $0$. In this paper, we focus on these
random variables. In this case, Eq. (2) is always an integer.

###  III.1 Generating function

We are interested in $F(T|n)$, which is the probability distribution for the
counting variable (2) first passing a fixed value $n$ at time $T$, especially
in the LDs of the random variable $\tau=T/|n|$. We assume that the
distribution satisfies the LD principle [8]:

$\displaystyle F(T|n)\sim e^{-nI_{+}(\tau)}u(n)+e^{nI_{-}(\tau)}u(-n),$ (3)

where $I_{\pm}$ are the rate functions as $n$ tends to positive infinity and
negative infinity, respectively, and $u(n)$ is the unit step function. We
assume that the values of $n$ are positive and negative. It is more convenient
to investigate the moment generating function (MGF) than the distribution
itself:

$\displaystyle M_{F}(\nu|n)=\int_{0}^{+\infty}dTF(T|n)e^{-\nu T}.$ (4)

From the LD principle (3), we have the following approximation:

$\displaystyle M_{F}(\nu|n)$ $\displaystyle\sim$ $\displaystyle
e^{n\psi_{+}(\nu)}u(n)+e^{n\psi_{-}(\nu)}u(-n),$ (5)

where $\psi_{\pm}$ are the SCGFs and are equal to the Legendre transforms of
the rate functions $I_{\pm}$, that is,

$\displaystyle\psi_{\pm}(\nu)=\mp\min_{\tau}\\{\nu\tau+I_{\pm}(\tau)\\}.$ (6)

The presence of a fixed integer makes the calculation of the MGF complex. This
difficulty can be circumvented by calculating the $z$-transform of $n$ [26]:

$\displaystyle Z_{F}(\nu|z)=\sum_{n=-\infty}^{+\infty}\ M_{F}(\nu|n)z^{-n}.$
(7)

The reason is clear from Eq. (13). We simply call Eq. (7) the generating
function. According to Eq. (5), this function is expected to have the
following approximation:

$\displaystyle
Z_{F}(\nu|z)\sim\frac{1}{1-z^{-1}e^{\psi_{+}(\nu)}}+\frac{1}{1-ze^{-\psi_{-}(\nu)}}.$
(8)

Two comments are in order. First, the exponentials of the SCGFs of the FPT
distribution are just the two poles of the generating function. Hence, if we
obtained these poles, e.g., $z^{*}(\nu)$, we would have

$\displaystyle\psi_{\pm}(\nu)=\ln z^{*}(\nu).$ (9)

Second, the validity of Eq. (7) indicates that the region of convergence (ROC)
of $Z_{F}$ in the $z$-plane is a circle whose radius is in the range of
$\exp[\psi_{+}(\nu)]$ to $\exp[\psi_{-}(\nu)]$ [26]. We emphasize that Eq. (9)
does not imply that the SCGFs are simply solved from the algebraic equation of
the generating function because the number of poles could be more than two.
Further analyses are needed 111In this work, fortunately, the number of poles
of the quantum TLS is indeed less than or equal to 2..

###  III.2 Equation of poles

Here, we show how to calculate the generating function $Z_{F}$ of the open
quantum system. We let $T_{\alpha\beta}(n|t)$ be the joint transition
probability that the quantum system starts from quantum state $\phi_{\beta}$,
the last collapsed state is $\phi_{\alpha}$, and the counting variable (2) is
$n$ at time $t$, simultaneously. We let $F_{\alpha}(T|n)$ be the FPT
distribution of the counting variable passing through $n$ at time $T$ and the
last collapsed state being $\phi_{\alpha}$. Obviously,

$\displaystyle F(T|n)=\sum_{\alpha=1}^{M}F_{\alpha}(T|n).$ (10)

Note the difference between $t$ and $T$: the former is a fixed duration of the
quantum process, whereas the latter is a random variable. Saito and Dhar [11]
and Ptaszynski [28] argued that the probabilities satisfy the following
equation:

$\displaystyle\sum_{\beta=1}^{M}{\hat{T}}_{\alpha\beta}(n|\nu)P_{\beta}(0)=\sum_{\gamma=1}^{M}\hat{T}_{\alpha\gamma}(0|\nu){\hat{F}}_{\gamma}(\nu|n),$
(11)

where $P_{\beta}(0)$ is the probability of the quantum system being in state
$\phi_{\beta}$ initially. In the remainder of this paper, we use a “hat”
placed over a symbol to denote the Laplace transform with respect to time $t$.
Further applying the $z$-transforms of $n$ to both sides of Eq. (11) yields

$\displaystyle\sum_{\beta=1}^{M}{\bar{\hat{T}}}_{\alpha\beta}(z|\nu)P_{\beta}(0)=\sum_{\gamma=1}^{M}\hat{T}_{\alpha\gamma}(0|\nu){\bar{\hat{F}}}_{\gamma}(\nu|z).$
(12)

Here, we place a “bar” over a symbol to denote the z-transform of the discrete
$n$. Equation (12) is more concise when it is written in matrix form:

$\displaystyle\bar{\hat{{\mathbb{T}}}}(z|\nu){\bf
P}(0)={\hat{{\mathbb{T}}}}(0|\nu){\bar{\hat{\bf F}}}(\nu|z),$ (13)

where we use a symbol in bold font to indicate a column matrix, e.g., $[{\bf
P}(0)]_{\alpha}=P_{\alpha}(0)$, and use a symbol in blackboard bold to
indicate a matrix, e.g.,
$[{\bar{\hat{{\mathbb{T}}}}}]_{\alpha\beta}=\bar{\hat{T}}_{\alpha\beta}$. By
substituting Eq. (37) of the counting statistics into Eq. (13), we arrive at
the final expression for the generating function of the FPT distribution:

$\displaystyle Z_{F}(\nu|z)$ $\displaystyle=$ $\displaystyle{\bf
1}^{T}{\bar{\hat{\bf F}}}(\nu|z)$ (14) $\displaystyle=$ $\displaystyle{\bf
1}^{T}{\hat{{\mathbb{T}}}^{-1}(0|\nu)}\hat{{\mathbb{S}}}(\nu){\left[{\mathbb{I}}-{\mathbb{W}}(\nu,z)\right]^{-1}}{\bf
P}(0),$

where the row matrix is ${\bf 1}^{T}=(1\cdots,1)$, superscript $T$ denotes the
transpose, and ${\mathbb{I}}$ is the unit matrix. In Eq. (14), the elements of
the matrices $\mathbb{W}$ and $\mathbb{S}$ are

$\displaystyle[{\mathbb{W}}]_{\alpha\beta}=\hat{p}_{\alpha|\beta}(\nu)z^{-w_{\alpha\beta}}$
(15)

and

$\displaystyle[\hat{{\mathbb{S}}}]_{\alpha\beta}={\hat{S}}_{\alpha}(\nu)\delta_{\alpha\beta},$
(16)

respectively, where $\delta_{\alpha\beta}$ is the Kronecker delta symbol. In
the former equation, $\hat{p}_{\alpha|\beta}(\nu)$ is the Laplace transform of
the WTD $p_{\alpha|\beta}(t)$ for the system starting from the quantum state
$\phi_{\beta}$, continuously evolving, and then collapsing to the quantum
state $\phi_{\alpha}$ at age $t$. In the latter equation,
$\hat{S}_{\alpha}(\nu)$ is the Laplace transform of the survival distribution
(SD) $S_{\alpha}(t)$ for the system starting from the quantum state
$\phi_{\beta}$ and continuously evolving until time $t$ without collapse. The
matrix element $\bar{\hat{T}}_{\alpha\beta}$ is explained in Appendix A. The
preceding discussion clearly shows that the generating function $Z_{F}$ is
rational, i.e., a ratio of polynomials in $z$. Therefore, the inverse
$z$-transform or the MGF $M_{F}(\nu|n)$ must be a linear combination of the
right-sided ($\propto u(n)$) and left-sided ($\propto u(-n)$) exponentials
[26], the bases of which are determined by the poles and the latter satisfy
the following algebraic equation:

$\displaystyle\det[{\mathbb{I}}-{\mathbb{W}}(\nu,z)]=0.$ (17)

This is the main result of this work. We call this the equation of poles.
Because the ROC of a rational $z$-transform is always bounded by its two poles
and does not contain any poles [26], the inverse $z$-transforms for the other
poles with smaller magnitude than the left-bounded pole are the right-sided
exponentials, whereas the inverse $z$-transforms for the other poles with
larger magnitude than the right-bounded pole are the left-sided exponentials.
According to the LD principle [8], the terms on the right-hand side of Eq. (5)
are the dominant functions of the MGF when $|n|$ tends to infinity. Therefore,
we arrive at a key conclusion that $z^{*}$ in Eq. (9) are just the right-
bounded and left-bounded poles of the ROC. Crucially, these bounded poles are
also given by the valid condition of Eq. (14) that the convergence radius of
the matrix $\mathbb{W}$ must be less than 1. Thus far, the value $n$ of the
counting variable (2) has been assumed to be positive and negative. If the
value of a counting variable is always nonnegative, the previous discussion is
slightly modified to adapt to the new situation. Here, we do not present this
procedure because of its simplicity. We emphasize that only the terms with
subscript $+$ in Eqs. (3)-(9) are currently involved.

Before closing this section, we comment on the known conjugation relationship
between the LDs of the FTP and the counting statistics [14] in the quantum
case. We have proven that the SCGF of the counting statistics is obtained by
solving for the largest real root of the same equation of poles (17) with
respect to $\nu$ instead of $\lambda$; see Appendix A. Because both statistics
are based on the same ROC, the corresponding SCGFs are indeed inverse
functions of each other. Notably, the degrees of the algebraic equation could
be distinct with respect to $z$ and $\nu$. This distinction is exploited when
we attempt to analytically solve for the LDs of the FPT statistics in the
quantum TLS.

##  IV Quantum Two-Level System

We use the typical quantum TLS to illustrate the previous theoretical results.
This system is driven by a resonant field and is surrounded by an environment
with inverse temperature $\beta$; see the schematic diagram in the left column
of Fig. (1). In the interaction picture, the quantum master equation of the
system is

$\displaystyle\partial_{t}\rho(t)$ $\displaystyle=$ $\displaystyle-{\rm
i}\left[H,\rho(t)\right]+r_{-}[\sigma_{-}\rho(t)\sigma_{+}-\frac{1}{2}\\{\sigma_{+}\sigma_{-},\rho(t)\\}]$
(18)
$\displaystyle+r_{+}[\sigma_{+}\rho(t)\sigma_{-}-\frac{1}{2}\\{\sigma_{-}\sigma_{+},\rho(t)\\}].$

Here, the Planck constant $\hbar$ is set to 1,
$H=-\Omega(\sigma_{-}+\sigma_{+})/2$ represents the interaction Hamiltonian
between the TLS and the resonant field, $\sigma_{\pm}$ represents the raising
and lowering Pauli operators, and $\Omega$ represents the Rabi frequency. The
coefficients $r_{-}=r_{0}(\bar{n}+1)$ and $r_{+}=r_{0}{\bar{n}}$ are the
pumping and damping rates, respectively, where $r_{0}$ is the spontaneous
decay rate and $\bar{n}$ is the Bose–Einstein distribution of the environment.
The two rates satisfy the detailed balance condition
$r_{-}=r_{+}\exp{(\beta\omega_{0})}$, and $\omega_{0}$ is the energy level
difference. Obviously, the excited state $|1\rangle$ and the ground state
$|2\rangle$ are the two collapsed quantum states. Then, the equation of pole
(17) is simply

$\displaystyle[1-\hat{p}_{1|1}(\nu)z^{-w_{11}}][1-\hat{p}_{2|2}(\nu)z^{-w_{22}}]-\hat{p}_{2|1}(\nu)\hat{p}_{1|2}(\nu)z^{-w_{12}-w_{21}}=0.$
(19)

The Laplace transforms of the WTDs of the system are exactly known [1].
Considering that they are useful in the calculations, we list them at the end
of Appendix B.

###  IV.1 Two exact large deviations of first passage time distributions

We consider two counting variables and solve for their respective SCGFs for
the FPT statistics. The first is the number of collapses to the quantum state
$|2\rangle$, that is, $w_{21}=w_{22}=1$ and $w_{11}=w_{12}=0$. In quantum
optics, collapses to the ground state indicate emissions of photons. Hence,
this variable is also interpreted as a count of single photons [24]. Note that
its value is nonnegative. By substituting the Laplace transforms of the WTDs
$p_{\alpha|\beta}$ for $\alpha,\beta=1,2$ into Eq. (19), simple algebraic
operations lead us to

$\displaystyle[2r_{+}r_{-}\xi(\nu)+r_{-}\Omega^{2}]z^{-1}-2\xi(\nu)[\xi^{2}(\nu)+4\mu^{2}]+r_{+}\Omega^{2}=0,$
(20)

where $\xi(\nu)=\nu+r/2$, $r=r_{-}+r_{+}$, $16\mu^{2}=4\Omega^{2}-\delta^{2}$,
and $\delta=r_{-}-r_{+}$. According to Eq. (9), the SCGF is then simply

$\displaystyle\psi_{+}(\nu)=\ln\frac{2r_{+}r_{-}\xi(\nu)+r_{-}\Omega^{2}}{2\xi(\nu)[\xi^{2}(\nu)+4\mu^{2}]-r_{+}\Omega^{2}}.$
(21)

The weights of the second counting variable are $w_{21}=w_{22}=1$ and
$w_{11}=w_{12}=-1$. Now, its value can be positive or negative. If we multiply
it $\omega_{0}$, the resulting quantity will be the heat of a quantum jump
trajectory [29, 30]. Simple algebraic operators of Eq. (19) yields

$\displaystyle z^{-2}-2b(\nu)z^{-1}+r_{+}/r_{-}=0,$ (22)

where the coefficient
$b(\nu)=\left[\xi^{3}(\nu)+\xi(\nu)(4\mu^{2}-r_{-}r_{+})\right]/\Omega^{2}r_{-}$.
This is a quadratic equation in $z$. Then, the two SCGFs of the FPT statistics
are

$\displaystyle\psi_{\pm}(\nu)=-\ln\left[b(\nu)\pm\sqrt{b^{2}(\nu)-r_{+}/r_{-}}\right],$
(23)

respectively. Because the two rates $r_{\pm}$ satisfy the detailed balance
condition, we easily prove that the two SCGFs (23) satisfy the fluctuation
theorem

$\displaystyle\psi_{+}(\nu)=\beta\omega_{0}-\psi_{-}(\nu).$ (24)

We depict these three SCGFs in Fig. (2)(a) and (b) for a set of parameters.

Figure 2: (a) The solid curve is the SCGF $\psi_{+}(\nu)$ of the variable that
counts collapses to the ground state. (b) The solid curve and dashed curves
are the SCGFs $\psi_{+}(\nu)$ and $\psi_{-}(\nu)$ of the other counting
variable, respectively. This random variable is related to the heat production
of the quantum jump trajectory. In both panels, the physical parameters are
$\omega_{0}=1$, $r_{-}=1$, $r_{+}=0.5$, and $\Omega=0.8$. The star symbols
denote the inverse functions of the SCGFs of the counting statistics for the
same counting variables.

###  IV.2 An inverse function method for calculating large deviations of
counting statistics

The SCGFs in Fig. (2) can be verified by depicting the inverse functions of
the SCGFs of the counting statistics for the same counting variables. As we
noted at the end of Sec. (III), the latter are obtained by solving for the
largest real roots of Eqs. (20) and (22) with respect to $\nu$ (a function of
$\lambda$) [1]; see the star symbols in Fig. (2). Obviously, the degrees of
the polynomials of the FPT and counting statistics are distinct: the former
are $1$ and $2$, whereas the latter are all 3. This inspires us to use a
numerical method to solve for the LDs of the counting statistics in certain
quantum systems. If the degree of the equation of poles of $z$ is less than or
equal to 3 and far smaller than that of $\nu$, we can first analytically solve
the SCGFs $\psi_{\pm}(\nu)$ of the FPT statistics and then perform an inverse
function transform, i.e., exchange the positions of $\nu$ and $\psi_{\pm}$ to
obtain the SCGFs of the counting statistics. An advantage of this method is
that it avoids the procedure of finding the largest real roots of higher-
degree polynomials 222Many efficient algorithms exist for finding the roots of
polynomials. Nevertheless, our method is simple.. For example, in the quantum
three-level system with one collapsed quantum state that is schematically
shown in Fig. (3), the degree of the equation of poles of $\nu$ is 6. In
contrast, the degree of the same equation of $z$ is $1$ [3]. We test the
method in the system, and the data are very satisfactory; see Fig. (3).

Figure 3: Schematic diagram of a driven quantum three-level system with one
collapsed quantum state $|1\rangle$ and the SCGFs of the counting statistics
for the random variable counting the number of collapses to the state
$|1\rangle$. The star and cross symbols represent the SCGF data obtained by
solving for the largest real root of a 6-degree polynomial, where two sets of
physical parameters are used [3]. The solid and dashed curves represent the
SCGF data obtained via the inverse function method. Note that the features of
the SCGFs depend on the values of the parameters [3, 16].

###  IV.3 Violations of classical uncertainty relations

In the past several years, quantum violations of the classical thermodynamic
uncertainty relation (TUR) [32, 33] and kinetic uncertainty relation (KUR)
[12] have attracted much interest. [34, 35, 36, 37, 38, 9, 39, 40]. Because
Gingrich and Horowitz noted that both uncertainty relations are conjugated in
the counting statistics and FPT statistics [14], we are interested in applying
the exact SCGFs, Eqs. (21) and (23), to study the same question. The first is
the classical KUR [41]. Its conjugation in the FPT statistics written in terms
of the SCGF is

$\displaystyle-\left.\frac{\psi_{+}^{\prime\prime}}{\psi_{+}^{\prime}}\right|_{\nu=0}\geq\frac{1}{\langle
k\rangle},$ (25)

where $\langle k\rangle$ represents the mean activity and prime (′) and double
prime (′′) denote the first and second derivatives with respect to $\nu$,
respectively. Here, the random counting variable of interest is the number of
collapses to the ground state $|2\rangle$. On the basis of Eq. (21), we easily
calculate the left-hand side of Eq. (25), and the result is

$\displaystyle\frac{2\Omega^{2}+r^{2}+4r_{+}r_{-}}{r_{-}(\Omega^{2}+rr_{+})}-\frac{6r}{2\Omega^{2}+r^{2}}.$
(26)

The mean activity is equal to the reciprocal of the mean random variable
$\langle T/K\rangle$ in the large $K$ limit, where $K$ is the full counting
variable. We let the SCGF of the FPT statistics of the full counting variable
be $\psi_{k}(\nu)$. The corresponding equation of poles is obtained by setting
$w_{ij}=1$ for $i,j=1,2$ in Eq. (2), that is,

$\displaystyle 2r_{+}r_{-}\xi
z^{-2}+\Omega^{2}rz^{-1}-2\xi(\xi^{2}+4\mu^{2})=0.$ (27)

Then, the right-hand side of Eq. (25), or the classical kinetic bound, is

$\displaystyle\frac{1}{\langle
k\rangle}=-\psi_{k}^{\prime}(0)=\frac{2\Omega^{2}+r^{2}}{r(\Omega^{2}+2r_{+}r_{-})}.$
(28)

In Fig. (4)(a), using two independent dimensionless parameters, $r_{0}/\Omega$
and $\bar{n}$, we numerically depict the exact curve on which the inequality
(25) reduces to an equality. We see that quantum violation of the KUR is not
uncommon in the parameter space; see the shaded area therein. Equations (26)
and (28) simply explain the validity of the KUR at very small and large
$r_{0}/\Omega$. If the Rabi frequency is very large, $\it i.e.$,
$r_{0}/\Omega\ll 1$, the two equations are approximately $1/r_{-}$ and $1/r$,
respectively. Obviously, $1/r_{-}>1/r$. If the Rabi frequency is very small,
$\it i.e.$, $r_{0}/\Omega\gg 1$, the two equations are approximately the
equations for the classical KUR in an incoherent TLS. Under this situation,
the inequality must be valid [12]. An algebra also proves this. Note that
these two cases are independent of $\bar{n}$. We see that quantum violations
of the KUR mainly occurs at smaller $\bar{n}$ and intermediate Rabi
frequencies. To illustrate this observation, in Fig. (4)(b), we specifically
depict Eqs. (26) and (28) as functions of $r_{0}/\Omega$ at fixed
$\bar{n}=0.15$. There are two quantum bounds for the KUR in the literature. On
the basis of the sMP perspective on quantum jump trajectories, Carollo et al.
derived a factor $\chi$ to replace the mean activity [9],

$\displaystyle\chi=\sum_{\alpha,\beta=1}^{2}\frac{\sigma^{2}_{\beta\alpha}}{\tau^{2}_{\beta\alpha}}R_{\beta\alpha}c_{\alpha}.$
(29)

The precise meanings of the involved coefficients are discussed in Appendix B.
We show the numerical data for the quantum bound $1/\chi$ in Fig. (4)(b); see
the dashed curve. Another interesting quantum bound was proposed by Van Yu and
Saito [38]. They corrected the mean activity to

$\displaystyle\langle k\rangle+Q=\langle
k\rangle\left(1+\frac{8\Omega^{2}}{r^{2}}\right).$ (30)

Considering that Ref. [40] presented a careful derivation, we do not explain
the correction but only directly present the final expression. We depict the
quantum bound $1/(\langle k\rangle+Q)$ in Fig. (4)(b); see the dotted curve
therein. We see that these two quantum bounds are always valid. In particular,
they uniformly converge to the classical bound at larger $r_{0}/\Omega$, that
is, that of the incoherent TLS. At a larger Rabi frequency, however, their
predictions completely deviate. Apparently, the bound of Eq. (29) is better
than the bound of Eq. (30).

Figure 4: (a) The curve indicates the values of $r_{0}/\Omega$ ($x$-axis) and
$\bar{n}$ ($y$-axis) on which the classical KUR of the quantum TLS reduces to
equality. The uncertainty relation is violated in the area enclosed by the
curve and $x$-axis. The $\bar{n}$ value of the horizontal line is $0.15$. The
inset shows the curve on which the classical TUR of the same quantum system
reduces to equality. (b) The black solid curve and gray thin curve represent
the data calculated using the left-hand side and right-hand side (the
classical bound) of the classical KUR, Eqs. (26) and (28), respectively. The
dashed and dotted curves are the quantum bounds $1/\chi$ and $1/(\langle
k\rangle+Q)$, respectively.

We briefly discuss the classical TUR to close this work. The classical TUR of
an asymmetric current in the FPT statistics

expressed in terms of the SCGF is [14]

$\displaystyle-\left.\frac{\psi_{+}^{\prime\prime}}{\psi_{+}^{\prime}}\right|_{\nu=0}\geq\frac{2k_{B}}{\langle\sigma\rangle},$
(31)

where $k_{B}$ is the Boltzmann constant, $\langle\sigma\rangle$ is the mean
entropy production rate, and we assume that the mean current is positive. The
rate of the second counting variable mentioned in Sec. (IV.1) is an asymmetric
current of the quantum TLS. Accordingly, the general Eq. (31) is reduced to

$\displaystyle\beta\omega_{0}\left.\frac{\psi^{\prime\prime}_{+}}{(\psi^{\prime}_{+})^{2}}\right|_{\nu=0}\geq
2.$ (32)

Here, we use the relation
$k_{B}/\langle\sigma\rangle=-\psi_{+}^{\prime}(0)/\beta\omega_{0}$. By
substituting the exact Eq. (23) and performing simple algebraic operations, we
obtain the left-hand side of Eq. (32):

$\displaystyle\beta\omega_{0}\left[\frac{r}{\delta}-\frac{6\Omega^{2}\delta
r}{(2\Omega^{2}+r^{2})^{2}}\right].$ (33)

Similar to the case of the KUR, using the parameters $r_{0}/\Omega$ and
$\bar{n}$, we numerically determine the curve on which the inequality (32)
reduces to an equality; see the inset of Fig. (4)(a). We find that violation
of the classical TUR is also not uncommon. In particular, the area of
violation is very distinct from that in which the classical KUR is satisfied.
Menczel et al.. [36] investigated the causes of the violation in detail. They
also obtained the same Eq. (33) as a special case of a general equation in the
presence of detuning; see Eq. (10) therein. Note that the preceding
derivations were carried out for the counting statistics, in which a
characteristic polynomial technique was used [17]. Therefore, this consistency
again demonstrates the conjugation between these two statistics.

##  V Discussion and Conclusions

In this paper, we apply the sMP method to calculate the large deviations of
the FPT statistics in open quantum systems. We find that the SCGFs can be
obtained by solving an algebraic equation of poles and calculating the
logarithm of the solution. Because the same equation is used to solve for the
SCGFs of the counting statistics for the same counting variables, the
conclusion that large deviations of the FPT statistics and counting statistics
are conjugated and connected by inverse functions to each other is explicitly
obtained in the quantum regime. An intriguing finding of this paper is that
the degrees of the equation of poles are distinct with respect to the two
statistics, at least in the typical quantum system with which we are
concerned. Because the degree of the polynomial for the FPT statistics is
usually smaller than that for the counting statistics, analytically solving
for the former SCGFs is possible. Hence, instead of the conclusion that one
should first solve for the large deviations of the counting statistics and
then obtain the large deviations of the FPT statistics through the inverse
function, our result indicates that a converse procedure is more useful in
open quantum systems. In addition, we use the analytical SCGFs of the FPT
statistics to explore quantum violations of the classical KUR and TUR. The
relationships between the two statistics are shown again. Intriguingly, the
quantum bound based on the sMP perspective is tighter than other quantum
bounds are. Clarifying the cause and its implications for quantum TUR deserves
further investigation. We hope that in the near future, we will report this
progress.

Acknowledgments This work was supported by the National Natural Science
Foundation of China under Grants No. 12075016 and 11575016.

##  Appendix A Joint transition probabilities of counting statistics

In terms of the WTDs, the probability distribution of the trajectory (1) is

$\displaystyle{\cal P}[{\bf
X}(t,N)]=S_{\gamma}(t-t_{N})p_{\gamma|\alpha_{N-1}}(t_{N}-t_{N-1})\cdots
p_{\alpha_{2}|\alpha_{1}}(t_{2}-t_{1})p_{\alpha_{1}|\eta}(t_{1}),$ (34)

where the first term on the right-hand side of the equation is the SD

$\displaystyle
S_{\gamma}(t)=\int_{t}^{\infty}\sum_{\beta=1}^{M}p_{\beta|\gamma}(s)ds.$ (35)

The joint transition probability of the counting variable (2) is formally
expressed by the trajectory distribution as

$\displaystyle T_{\gamma\eta}(n|t)=\sum_{N=0}^{\infty}\sum_{{\bf X}(t,N)}{\cal
P}[{\bf X}(t,N)]\delta_{nC[{\bf X}(t,N)]}.$ (36)

Note that the second summation is presented in shorthand notation: it
represents the sums of all possible intermediate collapsed states
$\phi_{\alpha_{i}}$ and integrals for all times $t_{i}$, $0\leq
t_{1}\leq\cdots\leq t_{N}\leq t$. Equation (36) is complex. However, it can be
dramatically simplified by calculating the Laplace transform of time $t$ and
the $z$-transform of count $n$ simultaneously. The result is

$\displaystyle\bar{\hat{T}}_{\gamma\eta}(z|\nu)$ $\displaystyle=$
$\displaystyle{\hat{S}}_{\gamma}(\nu)\sum_{N=0}^{\infty}{\hat{p}}_{\gamma|\alpha_{N-1}}(\nu)z^{-w_{\gamma\alpha_{N-1}}}\cdots{\hat{p}}_{\alpha_{2}|\alpha_{1}}(\nu)z^{-w_{\alpha_{2}\alpha_{1}}}{\hat{p}}_{\alpha_{1}|\eta}(\nu)z^{-w_{\alpha_{1}\eta}}$
(37) $\displaystyle=$
$\displaystyle\left[\hat{{\mathbb{S}}}(\nu)\sum_{N=0}^{\infty}{\mathbb{W}}^{N}(\nu,z)\right]_{\gamma\eta}$
$\displaystyle=$ $\displaystyle\left[\hat{{\mathbb{S}}}(\nu)\hskip
2.84544pt\left(\mathbb{I-W}(\nu,z)\right)^{-1}\right]_{\gamma\eta}.$

Here, we defined an $M\times M$ matrix $\mathbb{W}$ and diagonal matrix
$\hat{\mathbb{S}}$, and their elements are given in Eqs. (15) and (16),
respectively. We emphasize that the spectral radius of the matrix $\mathbb{W}$
must be less than $1$ to ensure convergence of the sum of the power matrices.
In our preceding paper [1], we defined a matrix $\mathbb{G}$ to calculate the
generating function of the counting statistics:

$\displaystyle Z_{C}(z|\nu)$ $\displaystyle=$ $\displaystyle{\bf
1}^{T}{\mathbb{G}}^{-1}(z,\nu){\bf P}(0),$ (38)

where $z=\exp(\lambda)$ and $\lambda$ are the parameters of the counting
statistics of the counting variable (2). The elements of the matrix
$\mathbb{G}$ are

$\displaystyle[{\mathbb{G}}]_{\gamma\gamma}$ $\displaystyle=$
$\displaystyle\frac{1-\hat{p}_{\gamma|\gamma}(\nu)z^{-w_{\gamma\gamma}}}{\hat{S}_{\gamma}(\nu)}$
(39) $\displaystyle{\rm}[{\mathbb{G}}]_{\gamma\eta}$ $\displaystyle=$
$\displaystyle-\frac{\hat{p}_{\gamma|\eta}(\nu)z^{-w_{\gamma\eta}}}{\hat{S}_{\gamma}(\nu)},{\hskip
2.84544pt}\gamma\neq\eta,$ (40)

respectively. It is easy to see that

$\displaystyle{\mathbb{G}}(\nu,z)=\left[\mathbb{I-W}(\nu,z)\right]{\hat{\mathbb{S}}}^{-1}(\nu),$
(41)

and the matrix ${\mathbb{G}}^{-1}$ is just the joint transition probability
matrix $\mathbb{T}$ for the counting statistics. Several comments are on
order. First, Eq. (38) is a quantum extension of the equation derived by
Andrieux and Gaspard many years ago when they studied the current fluctuation
theorem in the classical sMPs of classical stochastic systems [42]. Second,
Eq. (41) implies that the SCGF of the counting statistics is obtained by
solving the same equation of pole (17) with respect to $\nu$ (a function of
$\lambda$). Finally, there is still another expression for the matrix
$\mathbb{G}$:

$\displaystyle{\mathbb{G}}(z,\nu)$ $\displaystyle=$
$\displaystyle\nu{\mathbb{I}}-{\mathbb{L}}(z,\nu)$ (42)

where the elements of the matrix $\mathbb{L}$ are

$\displaystyle[{\mathbb{L}}]_{\gamma\gamma}$ $\displaystyle=$
$\displaystyle\frac{\hat{p}_{\gamma|\gamma}(\nu)z^{-\omega_{\gamma\gamma}}-\sum_{\alpha=1}^{M}{\hat{p}}_{\alpha|\gamma}(\nu)}{\hat{S}_{\gamma}(\nu)}$
(43)

and $[{\mathbb{L}}]_{\gamma\eta}=[{\mathbb{G}}]_{\gamma\eta}$, with
$\gamma\neq\eta$. The matrix $\mathbb{L}$ is the quantum extension [1] of the
tilted generator of a classical sMP [43]. If all the weights $w_{\gamma\eta}$
and diagonal elements ${\hat{p}}_{\gamma\gamma}$ are zero, this matrix reduces
to the standard generator of the sMP in the frequency domain [44].

##  Appendix B The factor $\chi$ of Carollo et al. [9]

In Eq. (29), the coefficients $c_{\alpha}$ for $\alpha=1,2$ represent the
rates of the quantum system collapsing into quantum state $\phi_{\alpha}$ in
the steady state. They can be written in terms of the Laplace transforms of
the WTDs [1, 9] as

$\displaystyle c_{1}$ $\displaystyle=$
$\displaystyle\left.\frac{{\hat{p}}_{12}}{{\hat{p}}_{12}\tau_{1}+{\hat{p}}_{21}\tau_{2}}\right|_{\nu=0},$
(44) $\displaystyle c_{2}$ $\displaystyle=$
$\displaystyle\left.\frac{{\hat{p}}_{21}}{{\hat{p}}_{12}\tau_{1}+{\hat{p}}_{21}\tau_{2}}\right|_{\nu=0},$
(45)

where
$\tau_{\alpha}={\hat{p}}^{\prime}_{1\alpha}+{\hat{p}}^{\prime}_{2\alpha}$ is
the mean time of the quantum system starting from quantum state
$\phi_{\alpha}$ and continuously evolving without collapse, that is, the mean
age of deterministic evolution. The coefficient $R_{\beta\alpha}$ is the
transition probability of a Markov chain embedded in the sMP, which is simply
equal to $\left.\hat{p}_{\beta\alpha}\right|_{\nu=0}$. The coefficients
$\sigma^{2}_{\beta\alpha}$ and $\tau_{\beta\alpha}$ are the variance and mean
of the time that the quantum system starts from the quantum state
$\phi_{\alpha}$ and finally collapses to state $\phi_{\beta}$, respectively
[9]. Their ratio is

$\displaystyle\frac{\sigma^{2}_{\beta\alpha}}{\tau^{2}_{\beta\alpha}}=\left.\frac{{\hat{p}}^{\prime\prime}_{\beta\alpha}{\hat{p}}_{\beta\alpha}}{\hat{p}^{\prime
2}_{\beta\alpha}}\right|_{\nu=0}-1.$ (46)

By substituting the exact formulas for the Laplace transforms of the WTDs of
the quantum TLS [1], we can obtain exact expressions for each term in Eq.
(29). Because this is a straightforward procedure and several expressions are
lengthy, we do not present them here. To make this paper self-contained, the
Laplace transforms of the WTDs of the quantum TLS are listed below:

$\displaystyle\hat{p}_{1|1}(\nu)$ $\displaystyle=$ $\displaystyle
r_{+}\frac{\Omega^{2}}{2\xi(\nu)[\xi^{2}(\nu)+4\mu^{2}]},$
$\displaystyle\hat{p}_{1|2}(\nu)$ $\displaystyle=$ $\displaystyle
r_{+}\frac{\xi^{2}(\nu)+\delta\xi(\nu)/2+\Omega^{2}/2}{\xi(\nu)[\xi^{2}(\nu)+4\mu^{2}]}$
$\displaystyle\hat{p}_{2|1}(\nu)$ $\displaystyle=$ $\displaystyle
r_{-}\frac{\xi^{2}(\nu)-\delta\xi(\nu)/2+\Omega^{2}/2}{\xi(\nu)[\xi^{2}(\nu)+4\mu^{2}]},$
$\displaystyle\hat{p}_{2|2}(\nu)$ $\displaystyle=$ $\displaystyle
r_{-}\frac{\Omega^{2}}{2\xi(\nu)[\xi^{2}(\nu)+4\mu^{2}]}.$ (47)

## References

  * [1] Fei Liu.  Semi-markov processes in open quantum systems: Connections and applications in counting statistics.  Phys. Rev. E, 106:054152, Nov 2022.
  * [2] Fei Liu.  Semi-markov processes in open quantum systems. ii. counting statistics with resetting.  Phys. Rev. E, 108:064101, Dec 2023.
  * [3] Fei Liu.  Asymptotic large deviations of counting statistics in open quantum systems.  Phys. Rev. E, 108:064111, Dec 2023.
  * [4] Lee H. Levitov, L. S. and G. B Lesovik.  Electron counting statistics and coherent states of electric current.  J. Math. Phys, 1996.
  * [5] D. A. Bagrets and Yu. V. Nazarov.  Full counting statistics of charge transfer in coulomb blockade systems.  Phys. Rev. B, 67:085316, Feb 2003.
  * [6] Massimiliano Esposito, Upendra Harbola, and Shaul Mukamel.  Nonequilibrium fluctuations, fluctuation theorems, and counting statistics in quantum systems.  Rev. Mod. Phys., 81(4):1665, 2009.
  * [7] Samuel L. Rudge and Daniel S. Kosov.  Counting quantum jumps: A summary and comparison of fixed-time and fluctuating-time statistics in electron transport.  J. Chem. Phys., 151(3), 07 2019.  034107\.
  * [8] Hugo Touchette.  The large deviation approach to statistical mechanics.  Phys. Rep., 478(1-3):1–69, 2008.
  * [9] Federico Carollo, Robert L. Jack, and Juan P. Garrahan.  Unraveling the large deviation statistics of markovian open quantum systems.  Phys. Rev. Lett., 122(13):130605, April 2019.
  * [10] Calum A. Brown, Katarzyna Macieszczak, and Robert L. Jack.  Unraveling metastable markovian open quantum systems.  Phys. Rev. A, 109:022244, Feb 2024.
  * [11] K. Saito and A. Dhar.  Waiting for rare entropic fluctuations.  Europhys. Lett., 2016.
  * [12] Juan P. Garrahan.  Simple bounds on fluctuations and uncertainty relations for first-passage times of counting observables.  Phys. Rev. E, 95:032134, Mar 2017.
  * [13] Todd R Gingrich, Grant M Rotskoff, and Jordan M Horowitz.  Inferring dissipation from current fluctuations.  Journal of Physics A: Mathematical and Theoretical, 50(18):184004, April 2017.
  * [14] Todd R. Gingrich and Jordan M. Horowitz.  Fundamental bounds on first passage time fluctuations for currents.  Phys. Rev. Lett., 119:170601, Oct 2017.
  * [15] Joel L. Lebowitz and Herbert Spohn.  J. Stat. Phys., 95(1/2):333–365, 1999.
  * [16] Juan P. Garrahan and Igor Lesanovsky.  Thermodynamics of quantum jump trajectories.  Phys. Rev. Lett., 104(16):160601, Apr 2010.
  * [17] M. Bruderer, L. D. Contreras-Pulido, M. Thaller, L. Sironi, D. Obreschkow, and M. B. Plenio.  Inverse counting statistics for stochastic and open quantum systems: the characteristic polynomial approach.  New J. Phys., 16 (2014)(3):033030, 2014.
  * [18] Heinz-Peter Breuer and Francesco Petruccione.  The theory of open quantum systems.  Oxford university press, 2002.
  * [19] B. R. Mollow.  Pure-state analysis of resonant light scattering: Radiative damping, saturation, and multiphoton effects.  Phys. Rev. A, 12:1919–1943, Nov 1975.
  * [20] P. Zoller, M. Marte, and D. F. Walls.  Quantum jumps in atomic systems.  Phys. Rev. A, 35:198–207, Jan 1987.
  * [21] M.D. Srinivas and E.B. Davies.  Photon counting probabilities in quantum optics.  Opt. Acta., 28(7):981–996, 1981.
  * [22] Howard Carmichael.  An open systems approach to Quantum Optics, volume 18.  Springer, 1993.
  * [23] Klaus Mølmer, Yvan Castin, and Jean Dalibard.  Monte carlo wave-function method in quantum optics.  J. Opt. Soc. Am. B, 10(3):524–538, Mar 1993.
  * [24] M.B. Plenio and P.L. Knight.  The quantum-jump approach to dissipative dynamics in quantum optics.  Rev. Mod. Phys., 70(1):101, 1998.
  * [25] Sheldon M. Ross.  Stochastic Processes.  John Wiley & Sons, New York, 2nd edition edition, 1995.
  * [26] Alan V.oppenheim, Alan S. Willsky, and S. Hamid Nawab.  Signals and Systems.  Prentice-Hall, Inc.:Upper Saddle River, NJ., 2nd edition edition, 1997\.
  * [27] In this work, fortunately, the number of poles of the quantum TLS is indeed less than or equal to 2.
  * [28] Krzysztof Ptaszyński.  First-passage times in renewal and nonrenewal systems.  Phys. Rev. E, 97:012127, Jan 2018.
  * [29] Heinz-Peter Breuer.  Quantum jumps and entropy production.  Phys. Rev. A, 68:032105, Sep 2003.
  * [30] Fei Liu and Jingyi Xi.  Characteristic functions based on a quantum jump trajectory.  Phys. Rev. E, 94:062133, Dec 2016.
  * [31] Many efficient algorithms exist for finding the roots of polynomials. Nevertheless, our method is simple.
  * [32] Andre C. Barato and Udo Seifert.  Thermodynamic uncertainty relation for biomolecular processes.  Phys. Rev. Lett., 114:158101, Apr 2015.
  * [33] Todd R. Gingrich, Jordan M. Horowitz, Nikolay Perunov, and Jeremy L. England.  Dissipation bounds all steady-state current fluctuations.  Phys. Rev. Lett., 116:120601, Mar 2016.
  * [34] Krzysztof Ptaszyński.  Coherence-enhanced constancy of a quantum thermoelectric generator.  Phys. Rev. B, 98:085425, Aug 2018.
  * [35] Obinna Abah and Eric Lutz.  Efficiency of heat engines coupled to nonequilibrium reservoirs.  EPL (Europhysics Letters), 106(2):20001, April 2014.
  * [36] Paul Menczel, Eetu Loisa, Kay Brandner, and Christian Flindt.  Thermodynamic uncertainty relations for coherently driven open quantum systems.  Journal of Physics A: Mathematical and Theoretical, 54(31):314002, jul 2021.
  * [37] Alex Arash Sand Kalaee, Andreas Wacker, and Patrick P. Potts.  Violating the thermodynamic uncertainty relation in the three-level maser.  Phys. Rev. E, 104:L012103, Jul 2021.
  * [38] Tan Van Vu and Keiji Saito.  Thermodynamics of precision in markovian open quantum dynamics.  Phys. Rev. Lett., 128:140602, Apr 2022.
  * [39] Yoshihiko Hasegawa.  Quantum thermodynamic uncertainty relation for continuous measurement.  Phys. Rev. Lett., 125:050601, Jul 2020.
  * [40] Michael J. Kewming, Anthony Kiely, Steve Campbell, and Gabriel T. Landi.  First passage times for continuous quantum measurement currents, 2023\.
  * [41] J. P. Garrahan, R. L. Jack, V. Lecomte, E. Pitard, K. van Duijvendijk, and F. van Wijland.  Dynamical first-order phase transition in kinetically constrained models of glasses.  Phys. Rev. Lett., 98:195702, May 2007.
  * [42] D. Andrieux and P. Gaspard.  Fluctuation theorem for currents in semi-markov processes.  J. Stat. Mech.: Theor. Exp, page P11007, 2008.
  * [43] Massimiliano Esposito, Upendra Harbola, and Shaul Mukamel.  Fluctuation theorem for counting statistics in electron transport through quantum junctions.  Phys. Rev. B, 75:155316, Apr 2007.
  * [44] H. Qian and H. Wang.  Continuous time random walks in closed and open single-molecule systems with microscopic reversibility.  Europhys. Lett., 76(1):15–21, 2006.


# Learning Intrinsic Dimension via Information Bottleneck for Explainable
Aspect-based Sentiment Analysis

###### Abstract

Gradient-based explanation methods are increasingly used to interpret neural
models in natural language processing (NLP) due to their high fidelity. Such
methods determine word-level importance using dimension-level gradient values
through a norm function, often presuming equal significance for all gradient
dimensions. However, in the context of Aspect-based Sentiment Analysis (ABSA),
our preliminary research suggests that only specific dimensions are pertinent.
To address this, we propose the Information Bottleneck-based Gradient (IBG)
explanation framework for ABSA. This framework leverages an information
bottleneck to refine word embeddings into a concise intrinsic dimension,
maintaining essential features and omitting unrelated information.
Comprehensive tests show that our IBG approach considerably improves both the
models’ performance and interpretability by identifying sentiment-aware
features.

Keywords: Intrinsic dimension, Information Bottleneck, Explainable, Aspect-
based Sentiment Analysis

Learning Intrinsic Dimension via Information Bottleneck for Explainable
Aspect-based Sentiment Analysis

Zhenxiao Cheng1, Jie Zhou1,∗††thanks: ∗ Corresponding author,
<EMAIL_ADDRESS>Wen Wu1, Qin Chen1, Liang He1
---
1 School of Computer Science and Technology, East China Normal University,
Shanghai, China

Abstract content

## 1\. Introduction

The domain of natural language processing (NLP) has witnessed the rise of
neural models that offer remarkable capabilities. Yet, the intricacies of
these models often remain cloaked in layers of complexity, raising questions
about their interpretability Danilevsky et al. (2020); Ribeiro et al. (2016);
Lundberg and Lee (2017). Gradient-based explanation methods Simonyan et al.
(2014); Smilkov et al. (2017); Sundararajan et al. (2017) have emerged as a
prominent solution to this interpretability conundrum, offering insights into
how neural models function Doshi-Velez and Kim (2017), especially in terms of
their fidelity.

These methods pivot on the idea of ascertaining the importance of words by
utilizing dimension-level gradient values, processed through a norm function.
Formally, Gradient-based explanation methods estimate the contribution of
input $x$ towards output $y$ by computing the partial derivative of $y$ w.r.t
$x$. These saliency methods can be used to enable feature importance
explainability, especially on word/token-level features Aubakirova and Bansal
(2016); Karlekar et al. (2018). Then, Smooth Gradient Smilkov et al. (2017)
and Integrated Gradients Sundararajan et al. (2017) are proposed to improve
the original gradients. A prevalent assumption made during this process is the
uniform significance attributed to every gradient dimension.

However, while this might hold true for many applications, the nuances of
Aspect-based Sentiment Analysis (ABSA) present a more complex scenario. In our
preliminary analysis of aspect-based sentiment classification tasks (Section
2), we have found that this assumption is not always valid. First, not all
dimensions are equally significant. Second, while the number of important
dimensions varies across datasets, only a few dimensions prove essential
(intrinsic dimension Li et al. (2018)). Third, the key dimensions exhibit
similarity within a dataset but vary across different datasets. For example,
dimension 401 ranks among the top 100 important dimensions for 89% instances
in the Res14 dataset. Further elaboration on these findings can be found in
Section 2. This observation calls for a more discerning approach to gradient-
based explanations in the ABSA context.

Figure 1: IBG compresses the 100 noisy dimensions of token gradient into 8
intrinsic dimensions via information bottleneck.

(a) Lap14

(b) Res14

(c) Res15

(d) Res16

(e) Lap14

(f) Res14

(g) Res15

(h) Res16

Figure 2: Visualization of the gradients on hidden dimension over four classic
ABSA datasets.

In this paper, we aim to answer the question: “How can important dimensions be
dynamically selected?" We propose an Information Bottleneck-based Gradient
explanation framework (IBG) for ABSA to learn the intrinsic dimension. To be
specific, we propose an Information Bottleneck-based Intrinsic Learning (iBiL)
structure to distill word embeddings into an intrinsic dimension that is both
concise and replete with pertinent information, ensuring that irrelevant data
is judiciously pruned (Figure 1). Our model is model-agnostic, we integrate it
with several state-of-the-art baselines, such as BERT-SPC Kenton and Toutanova
(2019) and DualGCN Li et al. (2021). Our comprehensive evaluations and tests
provide substantial evidence of the effectiveness of the IBG framework. As
detailed in the following sections, IBG not only enhances the performance
metrics but also improves the clarity of interpretations, shedding light on
sentiment-aware features.

The key contributions of this paper are listed as follows111Our code is
publicly available at https://github.com/your-username/your-repository.

  * •

We propose the Information Bottleneck-based Gradient (IBG) explanation
framework to find the low-dimensional intrinsic space since we discover that
not all dimensions of the embedding are equally important in completing the
ABSA task through preliminary analyses.

  * •

We introduce the iBiL structure, forcing the model to learn its intrinsic
sentiment embedding by effectively removing irrelevant information while
retaining sentiment-related details via information bottleneck.

  * •

Through extensive experiments, we demonstrate that our framework is capable of
enhancing both the performance and the interpretability of the original model
significantly.

(a) Lap14

(b) Res14

(c) Res15

(d) Res16

Figure 3: The influence of top k hidden dimension

## 2\. Preliminary Analysis

In this section, we mainly conduct preliminary analysis to answer the
following three questions.

Question 1: Are all the hidden dimensions equally important? In Figure 2, we
visualize the gradients of the hidden dimensions. Specifically, we train a
sentiment classifier for aspect-based sentiment analysis, which inputs the
sentence and aspect into a classifier to predict the sentiment polarity. Then
we compute the gradients of the opinion words concerning the given aspects as
well as all the words in the sentence. The classifier we used is the Bert-SPC
model, which is a very classic and fundamental baseline.

Our observations from these figures can be summarized as follows: First, we
note substantial variations in the values along the dimensions. Several
gradient values are notably larger than others, indicating that only a small
fraction of dimensions consistently contributes significantly to the model’s
predictions for each sample. Most dimensions have limited influence. Second,
the values of the opinion words are considerably larger than those of general
words. This suggests that gradients can assist in identifying key words.

Question 2: How many dimensions are necessary? The fact that all dimensions do
not have equal importance implies that there exist certain dimensions that are
really important. Therefore, in this subsection, we will discuss what is the
exact number of key dimensions among all dimensions. Specifically, we employ
gradient-based explanation methods to select the top-k important dimensions
for predicting sentiment polarity with respect to a given aspect (see Figure
3).

Our observations reveal the following: First, utilizing approximately the top
100-300 dimensions often yields similar results to using all dimensions in
most cases. Second, in certain instances, employing the top-k dimensions can
lead to performance improvements compared to using all dimensions.

(a) Lap14

(b) Res14

(c) Res15

(d) Res16

Figure 4: Percentage of Samples Where Each Dimension is in the Top 100
Importance

Question 3: Which dimensions are important? Therefore, aiming to ascertain
whether certain dimensions consistently play a crucial role in ABSA, we
conduct a statistical analysis focusing on the top-100 important dimension
indices in each sample’s prediction, as detailed in Figure 4.

Our findings indicate the following: First, it is noteworthy that across more
than 50% of samples within each dataset, approximately 80 dimensions
consistently maintain their positions within the top-100 important dimension
indices. Second, several dimension indices consistently demonstrate
significance in nearly every sample within a dataset. For instance, the top-3
significant dimensions are (145, 124, 401), (443, 175, 401), (401, 168, 218)
and (419, 145, 219) in the Lap14, Res14, Res15 and Res16 datasets,
respectively. However, it is essential to recognize that the specific
important dimensions vary among different datasets.

Figure 5: The framework of our IBG approach.

## 3\. Our Approach

In this paper, we propose the IBG explanation framework for ABSA by learning
the low-dimensional intrinsic features via information bottleneck (Figure 5).
Our framework explains the sentiment classifier by extracting the aspect-aware
opinion words via the gradient method (Section 3.1). It calculates the
important weights according to the gradients on the embedding level, which
contains redundancy information. Thus, we introduce our self-designed
Information Bottleneck-based Intrinsic Learning (iBiL) structure between the
embedding layer and encoder layer of the traditional language model (Section
3.2). This incorporation utilizes the information bottleneck principle to
compress the embedding layer to get the intrinsic representations, thereby
eliminating redundant information in the original embedding and retaining
essential information.

Let $s$ be a sentence with words $\\{w_{1},w_{2},...,w_{|s|}\\}$, $a$ be an
aspect in the sentence $s$, and $y\in Y$ be the sentiment label of $a$. Given
a corpus $\mathcal{D}=\\{(s_{i},a_{i},y_{i})\\}_{i=1}^{|\mathcal{D}|}$, our
task is to predict the sentiment polarity $y\in\\{P,N,O\\}$ of the sentence
towards the given aspect $a$. $P$, $N$, $O$ represent positive, negative and
neutral, respectively. The word embeddings of the sentence $s$ are
$x=\\{x_{1},...,x_{i},...,x_{|s|}\\}$, where $x_{i}$ is the word embedding of
$w_{i}$. Moreover, we aim to explain the model by extracting the _aspect-aware
opinion words_ $o$ that express the sentiment w.r.t the aspect $a$.

### 3.1. Overview of IBG

In this section, we introduce the overall structure of our IBG. It is a
gradient-based explanation method based on an aspect-based sentiment
classifier, which computes the importance scores for each token based on the
embedding to find the aspect-specific opinion words (e.g., delicious) To
improve the interpretability, it incorporates our designed iBiL structure
between the embedding and encoder layers. This structure is employed to
acquire intrinsic representations at the embedding level, eliminating
redundant information while retaining emotion-related essential information.

The prevailing paradigm of gradient-based model interpretation methods in NLP
consists of two main steps. First, a pre-trained language model, including the
embedding layer, is fine-tuned for specific downstream tasks. Subsequently,
importance scores for each token are computed, primarily through the embedding
layer.

Particularly, we train a sentiment classifier using our iBiL for ABSA, which
aims to predict the sentiment of the sentence concerning the given aspect. Let
$\mathcal{F}$ be a sentiment classifier with embedding and encoder layers that
predicts the sentiment distribution $P(y|s,a)$ based on the sentence $s$ and
aspect $a$.

$P(y|s,a)=\mathcal{F}(s,a)$ (1)

It is worth noting that, given our framework’s model-agnostic nature, the
sentiment classifier $\mathcal{F}$ can be any existing ABSA model. In this
paper, we mainly conduct the experiments on BERT-SPC Kenton and Toutanova
(2019) and DualGCN-BERT Li et al. (2021).

After the words $w$ in the sentence $s$ passing through the embedding layer,
we obtain embeddings $x=\\{x_{1},...,x_{i},...,x_{|s|}\\}$, where
$x_{i}\in\mathcal{R}^{\mathrm{High}}$ is the high dimension embedding of word
$w_{i}$. Afterward, we employ iBiL structures to learn distinct intrinsic
dimensional representations via the information bottleneck structure, which
will be introduced in Section 3.2. Through iBiL, we also obtain the “intrinsic
representation" $\hat{x}=\\{\hat{x}_{1},...,\hat{x}_{i},...,\hat{x}_{|s|}\\}$,
where $\hat{x}_{i}\in\mathcal{R}^{\mathrm{Low}}$ is the low dimension
representation of word $w_{i}$. Here, $\mathrm{High}$ is the dimensionality of
the hidden layer in the original embedding, while $\mathrm{Low}$ represents
the size of the intrinsic dimensionality, where $\mathrm{High}$ (e.g., 768) is
much larger than $\mathrm{Low}$ (e.g., 10, 20). To input the final word
representation $x^{\prime}$ into the encoder layer, we upsample the intrinsic
representation into a hidden dimension.

$x^{\prime}=x+W_{up}\hat{x}$ (2)

where $W_{up}$ represents a learnable upsampling matrix. Throughout the entire
training process, $x$ remains frozen and unchanged.

Subsequently, a novel attribution method is employed to determine the
importance scores for each token based on both the original embeddings and
intrinsic representations, ultimately leading to improvements in models’
performance and interpretability. Similar to prior work Simonyan et al.
(2014), we compute importance scores by dotting the gradients on the model’s
prediction with the corresponding vectors:

$\begin{split}{\gamma}(w_{i})&=\left|x_{i}\times\frac{\partial\mathcal{F}(s,a)}{\partial
x_{i}}\right|\\\
\hat{\gamma}(w_{i})&=\left|\hat{x}_{i}\times\frac{\partial\mathcal{F}(s,a)}{\partial\hat{x}_{i}}\right|\end{split}$
(3)

where $\gamma$ and $\hat{\gamma}$ are the importance weights for each token in
the embedding layer and the intrinsic representation layer.

Finally, for each token’s $\gamma$ and $\hat{\gamma}$, we introduce a
hyperparameter $\alpha$ to balance the weights between the two scores. For
each word $w_{i}$, the final importance score is calculated as:

$\mathrm{FScore}(w_{i})=(1-\alpha)\gamma(w_{i})+\alpha\hat{\gamma}(w_{i})$ (4)

Finally, we use $\mathrm{FScore}(w_{i})$ to select the opinion words of
aspect.

### 3.2. Information Bottleneck-based Intrinsic Learning

In our proposed IBG framework, one of the key innovations lies in the design
of Information Bottleneck-based Intrinsic Learning (iBiL). This involves
compressing the pre-trained embedding $x$ into a lower-dimensional space
$\hat{x}$ to learn the intrinsic sentiment representation in the embedding
layer. The primary goal is to retain essential information in the embedding
while removing redundant noise.

According to the original IB theory Alemi et al. (2016), the main objective is
to learn a compressed representation $Z$ by maximizing the mutual information
between $Z$ and output $Y$ and minimizing the mutual information between $Z$
and input $X$. In our work, to learn the intrinsic representation $\hat{x}$,
we aim to preserve the information of $\hat{x}$ with respect to the sentiment
polarity $y$ while minimizing the mutual information between $\hat{x}$ and the
original word vectors $x$. Therefore, our goal is to minimize the following
loss:

$\mathcal{L}=\beta I(x;\hat{x})-I(y;\hat{x})$ (5)

where $I(.,.)$ represents the mutual information.

However, in practical operations, directly calculating the mutual information
$x$ and $\hat{x}$ is not feasible. We employ the Variational Inference method
Li and Eisner (2019) to estimate the final loss function. Specifically, the
upper bound of $I(x;\hat{x})$ can be obtained through the following method,

$\begin{split}I(x;\hat{x})\leq\int\,d\hat{x}dxp(x)p(x,\hat{x})\log\frac{p(\hat{x}\mid
x)}{q(\hat{x})}\end{split}$ (6)

Since

$\displaystyle KL[p(\hat{x}),q(\hat{x})]\geq 0$ (7) $\displaystyle\implies$
$\displaystyle\int\,d\hat{x}p(\hat{x})\log
p(\hat{x})\geq\int\,d\hat{x}p(\hat{x})\log q(\hat{x})$

In actual computation process, we first sample $\hat{x}$ from the original
word vectors using the reparameterization method Kingma and Welling (2014):

$p(\hat{x}\mid x)=f_{\mu}(x)+f_{\xi}(x)\cdot z$ (8)

where $z\sim\mathcal{N}(0,I)$. Here, $f_{\mu}$ and $f_{\xi}$ correspond to two
linear layers, which are trainable. And $q(\hat{x})$ in Equation 6 is assumed
to follow the standard normal distribution. Therefore, $I(x;\hat{x})$ in
Equation 5 can be directly replaced with the KL loss in Equation 6 and be
easily calculated.

The lower bound of $I(y;\hat{x})$ can also be obtained as follows,

$\small\begin{split}I(y;\hat{x})\geq\int\,dyd\hat{x}p(y,\hat{x})\log
q(y\mid\hat{x})-\int\,dyp(y)\log p(y)\end{split}$ (9)

Since

$\displaystyle KL[p(y\mid\hat{x}),q(y\mid\hat{x})]\geq 0$
$\displaystyle\implies$ (10) $\displaystyle\int\,dyp(y\mid\hat{x})\log
p(y\mid\hat{x})\geq$ $\displaystyle\int\,dyp(y\mid\hat{x})\log
q(y\mid\hat{x})$

The $\int\,dyp(y)\log p(y)$ in Equation 9 is a constant, and thus, we do not
need to pay attention to it during the optimization. If we consider
$q(y\mid\hat{x})$ as the subsequent dimensionality-expanding matrix and
encoder structure of the model, then the process of minimizing $I(y;\hat{x})$
can be directly approximated as optimizing the original sentiment
classification loss function. In other words, $I(y;\hat{x})$ can be
straightforwardly replaced by a cross-entropy loss $\mathcal{L}_{CE}$. The
final loss function can be written as follows.

$\mathcal{L}=\mathcal{L}_{CE}+\beta KL[p(\hat{x}\mid x),q(\hat{x})]$ (11)

## 4\. Experiment Setups

### 4.1. Datasets, Metrics and Settings

#### Datasets.

To evaluate the performance and the interpretability of our IBG framework, we
conduct a comprehensive series of experiments on four common datasets: Res14,
Lap14, Res15 and Res16 Fan et al. (2019), which labeled the opinion words for
each aspect.

#### Metrics.

In order to assess the performance of our framework, we employ the most widely
used metrics in the ABSA task: Accuracy and Macro F1-score. Moreover, to
verify the effectiveness of our framework in improving models’
interpretability, following Chen and Ji (2020), we adopt the area over the
perturbation curve (AOPC) Nguyen (2018); Samek et al. (2016) and Post-hoc
Accuracy (Ph-Acc) Chen et al. (2018). AOPC computes the average decrease in
accuracy when the model makes predictions after removing the top-k important
words for explanation. Ph-acc, on the other hand, retains only the top-k words
and masks the remaining words to assess whether the model can still make
accurate predictions.

#### Settings.

We use bert-base-uncased version Kenton and Toutanova (2019) and Adam
optimizer for the original BERT and the additional components we introduce
with learning rates 1e-5 and 1e-4, respectively. In the selection of intrinsic
dimensionality sizes, we consider dimensions of 5, 10, 20, 50, 100 and 300.
The hyperparameter $\alpha$ was set to 0.5.

4 Model | Lap14 | Res14 | Res15 | Res16
---|---|---|---|---
Acc | F1 | AOPC | Ph-Acc | Acc | F1 | AOPC | Ph-Acc | Acc | F1 | AOPC | Ph-Acc | Acc | F1 | AOPC | Ph-Acc
AEN-BERT | 81.80 | 56.07 | - | - | 88.59 | 72.69 | - | - | 86.44 | 63.73 | - | - | 88.60 | 65.06 | - | -
LCF-BERT | 81.83 | 58.23 | - | - | 90.00 | 67.91 | - | - | 85.94 | 67.53 | - | - | 89.91 | 69.98 | - | -
ASCM4ABSA | 81.93 | 57.34 | - | - | 89.96 | 70.85 | - | - | 86.81 | 66.32 | - | - | 88.98 | 68.04 | - | -
ChatGPT | 82.78 | 44.60 | - | - | 90.62 | 51.72 | - | - | 90.54 | 71.86 | - | - | 93.63 | 76.70 | - | -
BERT-SPC${}_{\texttt{IEGA}}$ | 82.28 | 62.93 | 15.04 | 42.18 | 90.62 | 72.75 | 11.13 | 69.18 | 85.40 | 59.39 | 08.26 | 70.78 | 88.56 | 62.60 | 10.48 | 75.49
RGAT-BERT${}_{\texttt{IEGA}}$ | 82.58 | 65.10 | 13.58 | 66.52 | 91.64 | 77.50 | 15.67 | 63.29 | 87.09 | 69.36 | 16.07 | 72.98 | 90.78 | 67.34 | 12.71 | 80.04
BERT-SPC${}_{\texttt{Grad}}$ | 81.80 | 56.31 | 11.13 | 67.02 | 90.47 | 72.01 | 09.88 | 62.71 | 88.02 | 61.73 | 10.60 | 76.73 | 90.45 | 70.32 | 10.53 | 79.17
BERT-SPC${}_{\texttt{InteGrad}}$ | - | - | 09.85 | 74.09 | - | - | 10.47 | 54.24 | - | - | 16.59 | 80.88 | - | - | 10.31 | 81.40
BERT-SPC${}_{\texttt{SmoothGrad}}$ | - | - | 09.21 | 69.81 | - | - | 12.59 | 69.65 | - | - | 12.67 | 78.11 | - | - | 09.43 | 81.58
DualGCN-BERT${}_{\texttt{Grad}}$ | 84.27 | 67.07 | 18.84 | 64.60 | 91.41 | 76.62 | 11.65 | 77.65 | 88.71 | 69.82 | 13.36 | 73.04 | 91.67 | 76.97 | 13.60 | 73.46
DualGCN-BERT${}_{\texttt{InteGrad}}$ | - | - | 14.35 | 59.46 | - | - | 13.06 | 74.35 | - | - | 14.29 | 74.19 | - | - | 14.91 | 71.71
DualGCN-BERT${}_{\texttt{SmoothGrad}}$ | - | - | 22.70 | 68.03 | - | - | 15.53 | 80.24 | - | - | 14.29 | 74.19 | - | - | 16.23 | 75.00
BERT-SPC${}_{\texttt{IBG}}$ | 83.30 | 65.34 | 17.77 | 75.16 | 91.29 | 76.33 | 28.35 | 84.24 | 89.40 | 78.32 | 17.28 | 81.34 | 92.32 | 79.57 | 20.39 | 83.33
DualGCN-BERT${}_{\texttt{IBG}}$ | 85.22 | 72.99 | 26.89 | 66.17 | 92.58 | 80.22 | 27.18 | 78.71 | 90.63 | 77.68 | 25.58 | 75.81 | 93.64 | 84.61 | 20.83 | 82.24
4 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Table 1: The main results of performance and interpretability.

### 4.2. Baselines

We compare our framework with the SOTA baselines to investigate its
performance and interpretability. To validate the performance, we select the
following baselines:

  * •

BERT-SPC Kenton and Toutanova (2019) simply concatenates the raw sentences
with the corresponding aspect terms, subsequently feeding these inputs
directly into a pre-trained BERT model for ABSA.

  * •

AEN-BERT Song et al. (2019) proposes an Attentional Encoder Network (AEN) and
enhances BERT with attention-based encoders to capture context-specific
information related to aspects.

  * •

LCF-BERT Zeng et al. (2019) designs a Local Context Focus (LCF) mechanism
which uses multi-head self-attention to force the model to pay attention to
the local context words.

  * •

RGAT-BERT Wang et al. (2020) is the first work that uses the Graph
Convolutional Network (GCN) in ABSA to utilize the syntactical dependency
structures in the sentences.

  * •

DualGCN-BERT Li et al. (2021) is a Dual GCN, employing two GCNs, which can
simultaneously capture both syntax and semantic information.

  * •

ASCM4ABSA Ma et al. (2022) proposes three aspect-specific input methods and
exploits these transformations to promote the language models to pay more
attention to the aspect-specific context in ABSA.

  * •

ChatGPT Wang et al. (2023) designs a prompt specifically for ABSA, using the
GPT-3.5-turbo model to generate the results and analyze its performance.

  * •

IEGA Cheng et al. (2023) is a model agnostic Interpretation-Enhanced Gradient-
based framework for ABSA, which guides the model’s attention towards important
words like opinion words.

In addition, to verify the interpretability, we select some classic gradient-
based explanation strategies for comparison.

  * •

Simple Gradient Simonyan et al. (2014) calculates the gradient of the model’s
output w.r.t. the input by taking the dot product of these gradients with the
corresponding feature values.

  * •

Smooth Gradient Smilkov et al. (2017) is an enhancement of the Simple Gradient
that reduces noise and provides more stable explanations by computing the
gradient at multiple points along the path to the actual input.

  * •

Integrated Gradients Sundararajan et al. (2017) considers a baseline input and
calculates the gradient at multiple points along the straight-line path,
connecting the baseline to the actual input.

4 Model | Lap14 | Res14 | Res15 | Res16
---|---|---|---|---
Acc | F1 | AOPC | Ph-Acc | Acc | F1 | AOPC | Ph-Acc | Acc | F1 | AOPC | Ph-Acc | Acc | F1 | AOPC | Ph-Acc
BERT-SPC${}_{\texttt{IBG}}$ | 83.30 | 65.34 | 17.77 | 75.16 | 91.29 | 76.33 | 28.35 | 84.24 | 89.40 | 78.32 | 17.28 | 81.34 | 92.32 | 79.57 | 20.39 | 83.33
w/o IB | 82.87 | 64.66 | 18.63 | 71.09 | 90.94 | 74.62 | 28.94 | 72.71 | 89.40 | 76.14 | 17.28 | 78.57 | 91.45 | 78.54 | 17.76 | 82.46
w/o iBiL | 81.80 | 56.31 | 11.13 | 67.02 | 90.47 | 72.01 | 09.88 | 62.71 | 88.02 | 61.73 | 10.60 | 76.73 | 90.45 | 70.32 | 10.53 | 79.17
4 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Table 2: The results of ablation studies.

## 5\. Results and Analyses

### 5.1. Main Results

To demonstrate that our framework can both enhance models’ performance and
interpretability, we conducted extensive comparative experiments (Table 1).
The results yield the following findings:

First, IBG outperforms other models in terms of both performance and
interpretability. Regarding performance, our framework performs better than
the strong models like RGAT and DualGCN, as well as the latest ChatGPT model.
As for interpretability, IBG also better captures meaningful words compared to
classical gradient-based strategies and model-agnostic methods like IEGA. This
indicates that IBG can effectively capture the sentiment-related low-
dimensional features.

Second, IBG is capable of further enhancing both the performance and
interpretability based on the existing models. We can see that our IBG results
in a 1-2 points improvement in accuracy on each dataset. Taking BERT-SPC as an
example, after integrating our framework, the Macro-F1 is improved by more
than 10 points on the Lap14, Res15 and Res16 datasets. Even though Dual-GCN
already performs well, Dual-GCN${}_{\texttt{IBG}}$ still achieves an
approximately 1% increase in accuracy and 5% improvement in F1 score over all
datasets. We also notice that our framework exhibits a higher improvement in
Macro-F1. This indicates that the representations obtained by information
bottleneck demonstrate higher sensitivity to different polarities, including
neutral sentiment. On the other hand, the effect of IBG on enhancing model
interpretability is also significant. The models exhibit substantial increases
in both AOPC and Ph-Acc after the incorporation of IBG. Compared to IEGA, the
AOPC of BERT-SPC${}_{\texttt{IBG}}$ surpasses BERT-SPC${}_{\texttt{IEGA}}$ by
17.22%, 9.02% and 9.91% on the Res14, Res15 and Res16 datasets, respectively.
These results further verify our IBG’s ability to better capture keywords like
opinion words through learning the intrinsic dimension.

Third, IBG offers a better explanation in comparison to traditional gradient-
based model interpretation strategies (Simple Gradient, Smooth Gradient and
Integrated Gradient). BERT-SPC with IBG demonstrates more than twofold
improvement in AOPC on the Res14 and Res16 datasets. Ph-Acc shows a remarkable
40.00% improvement on Res14 compared with the Integrated Gradient method.
Similarly, DualGCN-BERT${}_{\texttt{IBG}}$ exhibits noticeable enhancements in
AOPC compared to these three interpretation strategies, with the greatest
improvements of 12.54%, 15.53%, 12.22% and 7.23% on the four datasets.
However, the improvement in Ph-Acc on DualGCN-BERT is less pronounced. This
can be attributed to the fact that DualGCN leverages GCN to capture the
dependency relationships among words. Consequently, masking excessive words
within a sentence can disrupt the integrity of the graph’s structure, thus
impacting the final prediction. In summary, the incorporation of our framework
is capable of enhancing models’ ability to focus on contextually implicit
sentiment information in words through gradient analysis, thereby improving
its explanation.

### 5.2. Ablation Studies

We also conduct ablation experiments (Table 2) to validate our framework from
two perspectives: First, removing the iBiL (w/o iBiL) reduces performance,
which indicates that compressing the model into a low-dimensional space for
learning intrinsic dimension does benefit the ABSA task. Second, the
information bottleneck module has a significant impact on the model’s final
performance and interpretability. After removing the information bottleneck
structure (w/o IB), the model’s accuracy and F1-score both decline, even with
a decrease of 4.07% and 11.53% in Ph-Acc on the Lap14 and Res14 datasets,
respectively. This indicates that simply compressing embeddings to a lower
dimensionality introduces noise, which affects the model’s classification
ability. Introducing an information bottleneck, on the other hand, does enable
the model to forget irrelevant information and retain important features for
sentiment classification.

### 5.3. Further Analysis

#### The Influence of Compressed Size $\mathrm{Low}$.

We further explore the variation in model performance and interpretability
when compressing the pre-trained BERT embeddings into different dimensions.
Two key findings can be deduced from Figure 6: First, with the continuous
increase in dimensions, both models’ performance and interpretability exhibit
a trend of initially increasing and then decreasing. Second, the optimal state
of the model consistently reaches at 10 or 20 dimensions and the performance
is even better than the original model without compressing. This suggests that
our framework can effectively capture essential information in models’
embeddings and eliminate redundant information.

(a) Lap14

(b) Lap14

(c) Res16

(d) Res16

Figure 6: The influence of compressed size $\mathrm{Low}$.

#### The Influence of $\alpha$.

We conduct experiments to observe the impact of varying $\alpha$ values in
Equation 4, which are used to balance the weights between the important scores
obtained from intrinsic features and the original word embeddings (Figure 7).
We can see that both the intrinsic representations and the original word
embeddings are useful for the model’s explanation. Relatively, assigning a
higher weight to intrinsic dimension results in slightly higher AOPC and Ph-
Acc compared with assigning a higher weight to word embeddings. It shows
intrinsic vectors successfully learn the aspect-aware sentiment features,
which is important for ABSA.

(a) AOPC

(b) Ph-Acc

Figure 7: The influence of $\alpha$ on four datasets

## 6\. Related Work

### 6.1. Intrinsic Dimension

Oymak et al. (2021) showed that trained neural networks often exhibit a low-
rank property, giving rise to the concept of “intrinsic dimension" Li et al.
(2018), which represents the minimum subspace dimension capable of encoding
effective information or solving a problem. Following this concept, Pope et
al. (2020) applied dimension estimation techniques to high-dimensional image
data and discovered that natural image datasets have low intrinsic dimension.
Significantly, Aghajanyan et al. (2021) first introduced the intrinsic
dimension into the field of natural language processing, revealing that the
size of the large pre-trained language models’ intrinsic dimension is much
smaller compared to that of their whole parameters. After this, many studies
started to incorporate low-rank structures into large models, achieving
parameter-efficient learning by only tuning a small subspace based on the
intrinsic dimension Qin et al. (2021); Sun et al. (2022b, a); Houlsby et al.
(2019); Wang et al. (2022); Hu et al. (2021). Differing from predecessors’
attempts to compress or tune models using intrinsic dimension, our work learns
the intrinsic dimension of the large language models via information
bottleneck, exploring how to better interpret the model’s predictions in ABSA.

### 6.2. Gradient-based Explanation Algorithms

Literature on explanation and attribution methods has grown in the last few
years, with a few broad categories of approaches: perturbing the input Fong et
al. (2019); Ribeiro et al. (2016); utilizing gradient Baehrens et al. (2010);
Binder et al. (2016); Selvaraju et al. (2017); visualizing intermediate layers
Zeiler and Fergus (2014). Our work extends and improves upon the gradient-
based method Simonyan et al. (2014), a popular technique applicable to many
different types of models. Several works were proposed to improve the original
gradient-based methods, such as SmoothGrad Smilkov et al. (2017) and
Integrated Gradient Sundararajan et al. (2017). Different from them, we
optimize to compute gradient scores based on an intrinsic space, enabling a
more effective model interpretation method.

### 6.3. Information Bottleneck

A series of studies motivated us to utilize IB Li and Eisner (2019); Zhou et
al. (2021, 2022) to improve the explanations of gradient-based explanation
methods. Li and Eisner (2019) compressed the pre-trained embedding (e.g.,
BERT, ELMO), remaining only the information that helps a discriminative parser
through variational IB. Zhmoginov et al. (2021) utilized the IB approach to
discover the salient region. Some works Jiang et al. (2020); Chen et al.
(2018); Guan et al. (2019); Schulz et al. (2020); Bang et al. (2021) proposed
to identify vital features or attributions via IB. Moreover, Chen and Ji
(2020) designed a variational mask strategy to delete the useless words in the
text. In this paper, we utilize IB to learn the intrinsic space to improve the
models’ explanation.

### 6.4. Aspect-based Sentiment Analysis

Aspect-based Sentiment Analysis (ABSA) involves the extraction of aspect terms
and opinion words from a sentence and the prediction of sentiment polarity
Zhang et al. (2022a). In our study, we focus on the subtask Aspect-based
Sentiment Classification (ABSC), which entails predicting sentiment labels for
a given sentence and its associated aspect. To consider the complex contextual
relationships in sentences, some ABSC research combined attention mechanisms
with large pre-trained language models, such as BERT-SPC Kenton and Toutanova
(2019), AEN-BERT Song et al. (2019) and LCF-BERT Zeng et al. (2019). There is
another trend of combining dependency trees and Graph Convolutional Networks
(GCNs), exploiting syntax information explicitly, like RGAT Wang et al.
(2020), DualGCN Li et al. (2021) and SSEGCN Zhang et al. (2022b). However,
even when employing methods such as dependency trees to align aspect terms
with their corresponding opinion words, in practice, we still observe that the
model may focus on the wrong aspect, especially in sentences containing
multiple aspects. Therefore, research on Explainable Aspect-based Sentiment
Analysis is essential. Cheng et al. (2023) leveraged annotated opinion words
to force the model to pay greater attention to these words in terms of
gradients, enhancing the models’ interpretability, but the cost of annotation
is high. In this paper, we propose a model-agnostic framework to enhance both
performance and interpretability without additional labels.

## 7\. Conclusions and Further Work

This paper conducts preliminary experiments, demonstrating the uneven
importance of word embedding dimensions in ABSA. However, the current
gradient-based explanation methods do not take this difference into account.
Thus, we propose an Information Bottleneck-based Gradient (IBG) explanation
framework for ABSA, leveraging the information bottleneck principle to compel
the model to learn intrinsic information. By integrating our framework with
the latest models, we conduct extensive comparative experiments, confirming
that our proposed IBG framework significantly enhances both the performance
and interpretability of the original models. Through ablation experiments, we
demonstrate the beneficial impact of the information bottleneck structure and
the attempt to map the embedding layer to a low-dimensional intrinsic space.
Future research will explore applying IBG to large-scale language models
(e.g., LLaMA) and other NLP tasks.

## Acknowledge

The authors wish to thank the reviewers for their helpful comments and
suggestions. This research is funded by the National Key Research and
Development Program of China (No.2021ZD0114002), the National Natural Science
Foundation of China (No.62307028 and No.62377013), the Science and Technology
Commission of Shanghai Municipality Grant (No.22511105901, No.21511100402 and
No.21511100302), and Shanghai Science and Technology Innovation Action Plan
(No.23ZR1441800 and No.23YF1426100).

## References

##

  * Aghajanyan et al. (2021) Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ , pages 7319–7328.
  * Alemi et al. (2016) Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. 2016.  Deep variational information bottleneck.  In _Proceedings of the 4th International Conference on Learning Representations_.
  * Aubakirova and Bansal (2016) Malika Aubakirova and Mohit Bansal. 2016.  Interpreting neural networks to improve politeness comprehension.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2035–2041.
  * Baehrens et al. (2010) David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. 2010.  How to explain individual classification decisions.  _The Journal of Machine Learning Research_ , 11:1803–1831.
  * Bang et al. (2021) Seojin Bang, Pengtao Xie, Heewook Lee, Wei Wu, and Eric Xing. 2021.  Explaining a black-box by using a deep variational information bottleneck approach.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 11396–11404.
  * Binder et al. (2016) Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. 2016.  Layer-wise relevance propagation for neural networks with local renormalization layers.  In _Proceedings of the 25th International Conference on Artificial Neural Networks_ , pages 63–71.
  * Chen and Ji (2020) Hanjie Chen and Yangfeng Ji. 2020.  Learning variational word masks to improve the interpretability of neural text classifiers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_ , pages 4236–4251.
  * Chen et al. (2018) Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. 2018.  Learning to explain: An information-theoretic perspective on model interpretation.  In _Proceedings of the 35th International Conference on Machine Learning_ , pages 883–892.
  * Cheng et al. (2023) Zhenxiao Cheng, Jie Zhou, Wen Wu, Qin Chen, and Liang He. 2023.  Tell model where to attend: Improving interpretability of aspect-based sentiment classification via small explanation annotations.  In _Proceedings of the 2023 International Conference on Acoustics, Speech and Signal Processing_ , pages 1–5.
  * Danilevsky et al. (2020) Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. 2020.  A survey of the state of explainable AI for natural language processing.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_.
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards a rigorous science of interpretable machine learning.  _arXiv preprint arXiv:1702.08608_.
  * Fan et al. (2019) Zhifang Fan, Zhen Wu, Xin-Yu Dai, Shujian Huang, and Jiajun Chen. 2019.  Target-oriented opinion words extraction with target-fused neural sequence labeling.  In _Proceedings of the North American Chapter of the Association for Computational Linguistics_ , pages 2509–2518.
  * Fong et al. (2019) Ruth Fong, Mandela Patrick, and Andrea Vedaldi. 2019.  Understanding deep networks via extremal perturbations and smooth masks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2950–2958.
  * Guan et al. (2019) Chaoyu Guan, Xiting Wang, Quanshi Zhang, Runjin Chen, Di He, and Xing Xie. 2019\.  Towards a deep and unified understanding of deep neural models in nlp.  In _Proceedings of the 36th International Conference on Machine Learning_ , pages 2454–2463.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  In _Proceedings of the 36th International Conference on Machine Learning_ , pages 2790–2799.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021.  Lora: Low-rank adaptation of large language models.  In _Proceedings of the 9th International Conference on Learning Representations_.
  * Jiang et al. (2020) Zhiying Jiang, Raphael Tang, Ji Xin, and Jimmy Lin. 2020.  Inserting information bottleneck for attribution in transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 3850–3857.
  * Karlekar et al. (2018) Sweta Karlekar, Tong Niu, and Mohit Bansal. 2018.  Detecting linguistic characteristics of alzheimer’s dementia by interpreting neural models.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 701–707.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 4171–4186.
  * Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014.  Auto-encoding variational bayes.  In _Proceedings of the 2nd International Conference on Learning Representations_.
  * Li et al. (2018) Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018.  Measuring the intrinsic dimension of objective landscapes.  In _Proceedings of the 6th International Conference on Learning Representations_.
  * Li et al. (2021) Ruifan Li, Hao Chen, Fangxiang Feng, Zhanyu Ma, Xiaojie Wang, and Eduard Hovy. 2021\.  Dual graph convolutional networks for aspect-based sentiment analysis.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ , pages 6319–6329.
  * Li and Eisner (2019) Xiang Lisa Li and Jason Eisner. 2019.  Specializing word embeddings (for parsing) by information bottleneck.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_ , pages 2744–2754.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions.  In _Proceedings of the 31st international conference on neural information processing systems_ , pages 4768–4777.
  * Ma et al. (2022) Fang Ma, Chen Zhang, Bo Zhang, and Dawei Song. 2022.  Aspect-specific context modeling for aspect-based sentiment analysis.  In _Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing_ , pages 513–526.
  * Nguyen (2018) Dong Nguyen. 2018.  Comparing automatic and human evaluation of local explanations for text classification.  In _Proceedings of the North American Chapter of the Association for Computational Linguistics_ , pages 1069–1078.
  * Oymak et al. (2021) Samet Oymak, Mingchen Li, and Mahdi Soltanolkotabi. 2021.  Generalization guarantees for neural architecture search with train-validation split.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 8291–8301.
  * Pope et al. (2020) Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. 2020.  The intrinsic dimension of images and its impact on learning.  In _Proceedings of the 9th International Conference on Learning Representations_.
  * Qin et al. (2021) Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, et al. 2021.  Exploring universal intrinsic task subspace via prompt tuning.  _arXiv preprint arXiv:2110.07867_.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  " why should i trust you?" explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1135–1144.
  * Samek et al. (2016) Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. 2016.  Evaluating the visualization of what a deep neural network has learned.  _IEEE TNNLS_ , 28(11):2660–2673.
  * Schulz et al. (2020) Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. 2020.  Restricting the flow: Information bottlenecks for attribution.  In _Proceedings of the 8th International Conference on Learning Representations_.
  * Selvaraju et al. (2017) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 618–626.
  * Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In _Proceedings of 2nd International Conference on Learning Representations_.
  * Smilkov et al. (2017) Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. 2017.  Smoothgrad: removing noise by adding noise.  In _Proceedings of the International Conference on Machine Learning Workshop_.
  * Song et al. (2019) Youwei Song, Jiahai Wang, Tao Jiang, Zhiyue Liu, and Yanghui Rao. 2019.  Attentional encoder network for targeted sentiment classification.  _arXiv preprint arXiv:1902.09314_.
  * Sun et al. (2022a) Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuan-Jing Huang, and Xipeng Qiu. 2022a.  Bbtv2: towards a gradient-free future with large language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3916–3930.
  * Sun et al. (2022b) Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022b.  Black-box tuning for language-model-as-a-service.  In _Proceedings of the 39th International Conference on Machine Learning_ , pages 20841–20855.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , pages 3319–3328.
  * Wang et al. (2020) Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. 2020.  Relational graph attention network for aspect-based sentiment analysis.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3229–3238.
  * Wang et al. (2022) Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan, and Jianfeng Gao. 2022.  Adamix: Mixture-of-adaptations for parameter-efficient model tuning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5744–5760.
  * Wang et al. (2023) Zengzhi Wang, Qiming Xie, Zixiang Ding, Yi Feng, and Rui Xia. 2023.  Is chatgpt a good sentiment analyzer? a preliminary study.  _arXiv preprint arXiv:2304.04339_.
  * Zeiler and Fergus (2014) Matthew D Zeiler and Rob Fergus. 2014.  Visualizing and understanding convolutional networks.  In _Proceedings of the 13th European Conference on Computer Vision_ , pages 818–833.
  * Zeng et al. (2019) Biqing Zeng, Heng Yang, Ruyang Xu, Wu Zhou, and Xuli Han. 2019.  Lcf: A local context focus mechanism for aspect-based sentiment classification.  _Applied Sciences_ , 9(16):3389.
  * Zhang et al. (2022a) Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2022a.  A survey on aspect-based sentiment analysis: Tasks, methods, and challenges.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Zhang et al. (2022b) Zheng Zhang, Zili Zhou, and Yanna Wang. 2022b.  Ssegcn: Syntactic and semantic enhanced graph convolutional network for aspect-based sentiment analysis.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 4916–4925.
  * Zhmoginov et al. (2021) Andrey Zhmoginov, Ian Fischer, and Mark Sandler. 2021.  Information-bottleneck approach to salient region discovery.  In _Proceedings of the Machine Learning and Knowledge Discovery in Databases: European Conference_ , pages 531–546.
  * Zhou et al. (2021) Jie Zhou, Yuanbin Wu, Qin Chen, Xuan-Jing Huang, and Liang He. 2021.  Attending via both fine-tuning and compressing.  In _Findings of the Association for Computational Linguistics_ , pages 2152–2161.
  * Zhou et al. (2022) Jie Zhou, Qi Zhang, Qin Chen, Liang He, and Xuan-Jing Huang. 2022.  A multi-format transfer learning model for event argument extraction via variational information bottleneck.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 1990–2000.