# A New Spectral Conjugate Subgradient Method with Application in Computed
Tomography Image Reconstruction

M. Loretoa and T. Humphriesa and C. Raghavanb and K. Wuc and S. Kwakd CONTACT
M. Loreto. Email<EMAIL_ADDRESS>a University of Washington Bothell, USA; b
New York University, USA; c University of Washington, USA; d University of
Oregon, USA.

###### Abstract

A new spectral conjugate subgradient method is presented to solve nonsmooth
unconstrained optimization problems. The method combines the spectral
conjugate gradient method for smooth problems with the spectral subgradient
method for nonsmooth problems. We study the effect of two different choices of
line search, as well as three formulas for determining the conjugate
directions. In addition to numerical experiments with standard nonsmooth test
problems, we also apply the method to several image reconstruction problems in
computed tomography, using total variation regularization. Performance
profiles are used to compare the performance of the algorithm using different
line search strategies and conjugate directions to that of the original
spectral subgradient method. Our results show that the spectral conjugate
subgradient algorithm outperforms the original spectral subgradient method,
and that the use of the Polak–Ribière formula for conjugate directions
provides the best and most robust performance.

###### keywords:

Nonsmooth optimization; conjugate gradients; spectral subgradient; computed
tomography.

90C30, 90C56, 94A08

##  1 Introduction

The problem under consideration is the minimization of nonsmooth functions
without any constraints

$\min_{x\in\mathbb{R}^{n}}~{}~{}f(x),$ (1)

where $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is locally Lipschitz continuous
in its domain and differentiable on an open dense subset of $\mathbb{R}^{n}$.
We are interested in the specific case where $f$ is continuously
differentiable almost everywhere and not differentiable at minimizers.

To solve (1), Loreto et al. in [24] proposed the spectral subgradient method.
It combines a nonmonotone line search [16], a globalization scheme [10], and
the classical subgradient approach using the the Barzilai-Borwein (spectral)
step length [4]. The significant advantage of the spectral step is that its
computation is independent of the optimal function value. In [24], extensive
numerical experimentation was presented, comparing six classic subgradient
methods with the spectral subgradient with the method of performance profiles
[12]. The numerical experiments demonstrated the spectral subgradient method’s
superiority compared to these methods.

Moreover, Loreto and collaborators have explored combining the spectral step
with subgradients and other subdifferential approaches. In [9], they developed
the spectral projected subgradient method for the minimization of non-
differentiable functions on convex sets. Successful numerical results were
presented for two distinct problem types: set covering and generalized
assignment problems, and convergence results were shown later in [23] and
[25]. In [22], they developed and analyzed the nonsmooth spectral gradient
method for unconstrained problems in which the spectral step is combined with
subdifferential approaches, such as the simplex gradient and the sampling
gradient.

The spectral subgradient method represents an extension to the nonsmooth case
of the spectral gradient method developed by Raydan [29, 30]. In [30], the
author found that the use of gradient descent with spectral step length for
unconstrained (smooth) nonlinear optimization provided several advantages over
approaches based on the conjugate gradient method for this problem. Inspired
by this, Birgin & Martínez proposed a spectral conjugate gradient (SCG) method
[5], which combines the spectral step with conjugate gradient directions. They
studied several formulas for determining the conjugate directions (which we
describe in Section 2.1), and determined that the use of the spectral step
provided improved results over the standard conjugate gradient method.

Several modifications to SCG have been proposed subsequently. Andrei proposed
the SCALCG algorithm [2, 1], including a quasi-Newton step to ensure the
production of descent directions. In [35, 13], the authors propose instead
modifying the choice of spectral step to guarantee descent, and prove
convergence using both Armijo and Wolfe line searches. Subsequent authors have
built off these early works by proposing modified formulas for generating
search directions with desirable properties, e.g.[34, 8, 3, 21, 11, 20].

All of these extensions to SCG assume $f$ is continuously differentiable. It
is interesting to note that an extension of the original conjugate gradient
method to minimize nondifferentiable functions using subgradients was first
presented many years ago by Wolfe [32, 33]. In the interim, however, there
appears to have been little work done in the area of using conjugate
subgradients for nonsmooth problems, despite some recent papers [27, 19].

In this paper, we introduce the spectral conjugate subgradient (SCS) method as
a novel approach for solving nonsmooth problems. The inspiration for this
method stems from the SCG approach. The SCS algorithm represents the first
extension of spectral CG methods to nonsmooth problems, to the best of our
knowledge. Our primary objective is to expand the application of spectral
methods to nonsmooth problems, given the success of smooth spectral methods in
solving a wide range of practical problems (Birgin et al. [6, 7])

The organization of this paper is as follows: Section 2 provides a description
of the SCS method, while Section 3 introduces an application of interest which
involves the use of the nonsmooth total variation (TV) function as a
regularizer for several image reconstruction problems in computed tomography
(CT). Section 4 presents numerical results based on a set of nonsmooth
problems, including the CT problem, using performance profiles. Finally,
Section 5 offers some concluding remarks.

##  2 Methodology

###  2.1 Spectral Conjugate Gradient Method

The original spectral CG method for unconstrained optimization was proposed by
Birgin & Martínez [5], under the assumption that the objective function f is
continuously differentiable at all points. Letting $g_{k}$ denote the gradient
at the $k$th iterate $x_{k}$, the vectors $s_{k}=x_{k+1}-x_{k}$ and
$y_{k}=g_{k+1}-g_{k}$ are defined for $k=0,1,2,\dots$. Beginning from an
initial guess $x_{0}$, the subsequent iterates are given by

$x_{k+1}=x_{k}+\alpha_{k}d_{k},$

where $\alpha_{k}$ is selected using a Wolfe line search (described in the
next section), and the conjugate directions $d_{k}\in\mathbb{R}^{n}$ are
generated by

$d_{k+1}=-\theta_{k}g_{k+1}+\beta_{k}s_{k}.$ (2)

The spectral step parameter, denoted by $\theta_{k}$, was initially proposed
by Barzilai and Borwein [4] and subsequently refined by Raydan [29, 30]:

$\theta_{k}=\frac{s_{k}^{T}s_{k}}{s_{k}^{T}y_{k}},$ (3)

with $\theta_{0}$ typically chosen to be 1. As the algorithm requires
$\theta_{k}$ to be non-negative and bounded, a safeguarding procedure is often
used (e.g. [24]):

$\theta_{k+1}=\left\\{\begin{array}[]{ll}\min\\{\theta_{\max},\displaystyle\frac{1}{\parallel
s_{k}\parallel}\\}&\mathrm{if\ }s_{k}^{T}y_{k}\leq 0\\\
\min\\{\theta_{\max},\max\\{\theta_{\min},\displaystyle\frac{s_{k}^{T}s_{k}}{s_{k}^{T}y_{k}}\\}\\}&\mathrm{if\
}s_{k}^{T}y_{k}>0\end{array},\right.$ (4)

where $\displaystyle 0<\theta_{\min}<\theta_{\max}<\infty$.

The parameter $\beta_{k}$ can be chosen in several ways; in [5], the authors
proposed the following three choices:

$\displaystyle\beta_{k}$
$\displaystyle=\frac{(\theta_{k}y_{k}-s_{k})^{T}g_{k+1}}{s_{k}^{T}y_{k}}$ (5)
$\displaystyle\beta_{k}$
$\displaystyle=\frac{\theta_{k}y_{k}^{T}g_{k+1}}{\alpha_{k}\theta_{k-1}g_{k}^{T}g_{k}}$
(6) $\displaystyle\beta_{k}$
$\displaystyle=\frac{\theta_{k}g_{k+1}^{T}g_{k+1}}{\alpha_{k}\theta_{k-1}g_{k}^{T}g_{k}}$
(7)

noting that under certain assumptions, these are equivalent to the formula
introduced by Perry [28] and its further modifications by Polak–Ribière and
Fletcher–Reeves, respectively. With any of these choices, it is possible that
the search direction $d_{k+1}$ computed by (2) may fail to be a descent
direction; in this case, the authors proposed setting
$d_{k+1}=-\theta_{k}g_{k+1}$ to “restart” the algorithm.

In their paper, Birgin & Martínez studied the performance of several variants
of this approach, including the three choices for $\beta_{k}$, using
$\theta_{k}=1$ instead of the spectral step (corresponding to the classical
conjugate gradient approach), and a heuristic for the initial choice of
stepsize $\alpha$ when performing the Wolfe line search. Ultimately it was
determined that the Perry formula (5) using the spectral step and the
heuristic for Wolfe line search gave the best performance.

This work presents an extension of the spectral conjugate gradient method that
allows for $g_{k}$ to be a subgradient at points where $f$ is non-
differentiable. A subgradient of $f$ at a point $x$ is any vector $g$ that
satisfies the inequality $f(y)\geq f(x)+g^{T}(y-x)$ for all $y$. This
algorithm can also be interpreted as a generalization of the spectral
subgradient method [24], which corresponds to the special case where
$\beta_{k}=0$ in equation (2). Additionally, we investigate two different line
search strategies: the nonmonotone line search used by the spectral
subgradient method, and the Wolfe line search used in [5]. We describe these
two choices of line search in the next section.

###  2.2 Line Search Strategies

As mentioned previously, the proposed spectral conjugate subgradient can use
one of two line search strategies: nonmonotone, or Wolfe line searches.

####  2.2.1 Nonmonotone Line Search

Historically, the spectral step has been paired with the nonmonotone line
search described below, based on Raydan’s work [30]. The reason for this
pairing is the nonmonotone behavior of the step. So, it is natural to adjust
the line search scheme with the nonmonotone globalization strategy of Grippo
et al. [16] combined with the proposed globalization scheme of La Cruz et al.
[10].

$\displaystyle f(x_{k}+\alpha d_{k})$ $\displaystyle\leq\max_{0\leq
j\leq\min\\{k,M\\}}f(x_{k-j})+\gamma\alpha
g_{k}^{T}d_{k}+\eta_{k},\text{with}$ (8) $\displaystyle 0<\sum_{k}\eta_{k}$
$\displaystyle=\eta<\infty.$ (9)

The value of $\alpha>0$ is determined by a backtracking process beginning with
$\alpha=1$. For the sequence $\eta_{k}$, we use
$\displaystyle\eta_{k}=\frac{\eta_{0}}{k^{1.1}}$, which guarantees that (9)
holds. This sequence helps the line search to be satisfied at early
iterations. $M$ is a fixed integer with $M\geq 0$, and $0<\gamma<1$ is a small
sufficient decrease parameter.

The nonmonotone behavior of the line search is induced by the terms
$\displaystyle\max_{0\leq j\leq\min\\{k,M\\}}f(x_{k-j})$ and $\eta_{k}$. The
parameter $M$ allows the function value to increase or remain stationary for
up to $M$ iterations, which is suitable for the behavior of the spectral step,
since the spectral step is associated with the eigenvalues of the Hessian at
the minimum point, rather than the function values, as noted by [14, 15].

####  2.2.2 Wolfe Line Search

In [5], the authors use the Wolfe conditions to guarantee convergence.
Although the Wolfe conditions assume that $f$ is differentiable, we included
the Wolfe line search option in our algorithm to be compared with the
nonmonotone line search, since we assume $f$ is differentiable almost
everywhere. Given $0<\gamma<\sigma<1$, we require that the step size $\alpha$
satisfy

$f(x_{k}+\alpha d_{k})\leq f(x_{k})+\gamma\alpha g_{k}^{T}d_{k}$ (10)

and

$\nabla f(x_{k}+\alpha d_{k})^{T}d_{k}\geq\sigma g_{k}^{T}d_{k}$ (11)

in every iteration, where $g_{k}$ is a subgradient of $f$ at $x_{k}$.

###  2.3 Spectral Conjugate Subgradient Algorithm

Using the definitions of $s_{k}$ and $y_{k}$ from Section 2.1, but where
$g_{k}$ is now a subgradient, we define the spectral conjugate subgradient
(SCS) algorithm as follows:

Algorithm SCS:

Let $x_{0}\in\mathbb{R}^{n}$ be given, and let $g_{0}$ represent a subgradient
at $f(x_{0})$. We define $d_{0}=-g_{0}$, and set $k=0$ and $MAXITER$ as the
maximum allowable number of iterations. Let $M$ be an integer with $M\geq 0$,
let $0<\theta_{\min}<\theta_{\max}<\infty$, let
$\eta_{0}=\max(f(x_{0}),\parallel g_{0}\parallel)$, and let
$0<\gamma<\sigma<1$. Then:

Repeat until $k=MAXITER$

  1. 1.

Compute $\alpha$ based on the line search, either nonmonotone (8), or Wolfe
((10) and (11))

  2. 2.

Define $\alpha_{k}=\alpha$ and $x_{k+1}=x_{k}+\alpha_{k}d_{k}$

  3. 3.

Compute $\theta_{k}$ by (4) and $\beta_{k}$ by (5),(6), or (7)

  4. 4.

Define $d=-\theta_{k}g_{k+1}+\beta_{k}s_{k}$

  5. 5.

If $d^{T}g_{k+1}\leq-10^{-3}\parallel d\parallel_{2}\parallel
g_{k+1}\parallel_{2}$

then $d_{k+1}=d$

else $d_{k+1}=-\theta_{k}g_{k+1}$

  6. 6.

$k=k+1$

Remarks:

  * •

Step 5 is the heuristic suggested by Birgin & Martínez to guarantee $d$ is a
descent direction. If the angle between $d$ and $-g_{k+1}$ lies outside of
$[-\frac{\pi}{2},\frac{\pi}{2}]$ (to some small tolerance), the algorithm is
restarted to use the direction of the negative subgradient.

  * •

For values of $x_{k+1}$ where $f$ is differentiable, the subgradient $g_{k+1}$
is equal to the gradient $\nabla f(x_{k+1})$.

  * •

When $\beta=0$ and the nonmonotone line search (8) is used, the SCS algorithm
becomes the spectral subgradient method described in [24].

  * •

When using the Perry formula (5) for $\beta_{k}$, it is possible in nonsmooth
problems for the denominator $s_{k}^{T}y_{k}$ to be zero. In this case we
simply set $\beta_{k}=0$, equivalent to the spectral subgradient method.

  * •

For the Wolfe line search ((10) and (11)), we adopted the implementation by
Taviani [31]

  * •

The algorithm terminates upon reaching the maximum iteration count
($MAXITER$), and the best value $f_{\min}$ is reported, along with the
corresponding point $x$.

##  3 CT problem with Total Variation

In computed tomography (CT) imaging, a two- or three-dimensional image of a
compactly supported function is recovered from a discrete sampling of the so-
called X-ray transform of the function. This transform consists of integrals
along every possible line through the function; in the two-dimensional case,
which we consider in this article, it is equivalent to the better-known Radon
transform. The simplest type of two-dimensional discretization, known as
parallel-beam geometry, is obtained by equal spacing of both the affine and
angular parameters in the X-ray transform, producing a set of measurements
known as a sinogram. Figure 1 shows an example of a simple two-dimensional
object (the Shepp-Logan phantom) and its sinogram.

Discretizing both the domain of the function and the X-ray transform results
in a linear system of equations:

$Ax+\eta=b,$ (12)

where $x\in\mathbb{R}^{n}$ is the non-negative image being reconstructed,
$b\in\mathbb{R}^{m}$ is the sinogram, and $A$ is the $m\times n$ system matrix
representing the geometry of the CT system. The vector $\eta$ represents
measurement noise that arises naturally during the real-life CT imaging
process. When $\|\eta\|$ is small, and the X-ray transform of the object is
well-sampled, the image can be reconstructed by solving the non-negative
least-squares problem:

$\min_{x\in\mathbb{R}^{n}_{+}}~{}~{}\frac{1}{2}\|Ax-b\|^{2},$ (13)

typically using an iterative method, due to the large size of the matrices
involved. In more challenging circumstances, one must instead solve a
regularized problem:

$\min_{x\in\mathbb{R}^{n}_{+}}~{}~{}\frac{1}{2}\|Ax-b\|^{2}+\mu\phi(x),$ (14)

where $\phi:\mathbb{R}^{n}\to\mathbb{R}$ is an appropriate regularization
function, and $\mu>0$ is a weighting parameter. Two scenarios of this type are
if $\|\eta\|$ is large, or if the angular sampling of the X-ray transform is
insufficient to recover the high-frequency components of $x$. Both of these
situations arise as a result of trying to reduce dose to a patient in medical
CT imaging; the former is often called low-dose imaging, and the latter is
sparse-view imaging. In both cases, solving the unregularized problem (13)
results in poor quality images, as shown in Figure 1.

Figure 1: CT imaging example. Left: true $400\times 400$ pixel digital Shepp-
Logan phantom. Center-left: parallel-beam sinogram corresponding to 360 views
taken over 180∘; the affine parameter is the $y$-axis and the angular
parameter the $x$-axis. Center-right: low-dose image reconstructed using
unregularized least-squares with 20% Gaussian noise added to $b$. Right:
Sparse-view mage reconstructed using unregularized least squares with only 60
views taken over 180∘

A common choice of regularizer $\phi(x)$ is the isotropic total variation (TV)
function:

$\phi_{TV}(x)=\sum_{1\leq i,j\leq
N-1}\sqrt{\left(x_{i,j+1}-x_{i,j}\right)^{2}+\left(x_{i+1,j}-x_{i,j}\right)^{2}},$
(15)

in which $x$ is reshaped as an $N\times N$ image (with $N^{2}=n$), indexed by
row and column. A routine computation gives the partial derivatives of
$\phi_{TV}$ as:

$\displaystyle\frac{\partial}{\partial x_{i,j}}\phi_{TV}(x)$
$\displaystyle=\frac{2x_{i,j}-x_{i+1,j}-x_{i,j+1}}{\sqrt{\left(x_{i+1,j}-x_{i,j}\right)^{2}+\left(x_{i,j+1}-x_{i,j}\right)^{2}}}$
(16)
$\displaystyle~{}~{}~{}+\frac{x_{i,j}-x_{i,j-1}}{\sqrt{(x_{i,j}-x_{i,j-1})^{2}+(x_{i+1,j-1}-x_{i,j-1})^{2}}}$
$\displaystyle~{}~{}+\frac{x_{i,j}-x_{i-1,j}}{\sqrt{(x_{i,j}-x_{i-1,j})^{2}+(x_{i-1,j+1}-x_{i-1,j})^{2}}}.$

Clearly the derivative is undefined whenever the denominator of any of the
three terms in (16) is zero. This situation arises frequently as it occurs
when pixels have the same values as their neighbors; furthermore, such images
are likely to be minimizers of (14) as they correspond to images with small
TV. Note, however, that the numerator of the respective term must also be zero
whenever the denominator is zero. Following [26], we therefore define a
subgradient by simply setting any term in (16) with a denominator of zero
equal to zero.

##  4 Numerical Results

We provide numerical results that demonstrate the performance of the proposed
method using a collection of widely recognized nonsmooth benchmark problems
frequently employed for evaluation purposes. Additionally, we showcase an
application in CT reconstruction with the utilization of total variation, as
outlined in Section 3.

###  4.1 Nonsmooth test problems

The collection of problems consists of 10 nonsmooth minimization problems,
taken from [18] and detailed in Table 1. This table provides information such
as the optimal value $f_{*}=f(x_{*})$ and the dimension $n$. For further
details on this problem set, including suggested initial points, please refer
to [18].

Table 1: Problem Set Prob | $f_{*}$ | n
---|---|---
P1: MAXQ | 0.0 | 20
P2: MXHILB | 0.0 | 50
P3: Chained LQ | $-(n-1)2^{1/2}$ | 2
P4: Chained CB3 I | $2(n-1)$ | 20
P5: Chained CB3 II | $2(n-1)$ | 20
P6: Activefaces | 0.0 | 2
P7: Brown 2 | 0.0 | 2
P8: Mifflin 2 | -34.795 | 50
P9: Crescent I | 0.0 | 2
P10:Crescent II | 0.0 | 2

We are comparing a total of 8 versions of the spectral conjugate subgradient
method, corresponding to two choices of line search (nonmonotone or Wolfe) and
four different approaches to calculate the parameter $\beta$ in (2). We let
$\beta_{0}=0$ correspond to the original spectral subgradient method, and
$\beta_{1}$, $\beta_{2}$ and $\beta_{3}$ correspond to the formulas (5), (6),
and (7) respectively. Each algorithm was run for $MAXITER=1000$ iterations,
and the the best function value, called $f_{\min}$, was reported.

We evaluate the performance of each algorithm by analyzing the results using
performance profiles [12]. A detailed explanation of how we adopted the
profiles is given in [24]. Considering a set of solvers $S$, which in this
scenario includes the 8 variations of the aforementioned algorithm, and the
set of problems $P$ given in Table 1, the performance ratio for each solver-
problem pair is defined as follows:

$r_{p,s}=\frac{t_{p,s}}{\min\\{t_{p,s}:s\in S\\}},$

where the performance measure $t_{p,s}$ represents a specific metric, e.g.
computation time required by solver $s$ to solve problem $p$. The performance
of solver $s$ is then assessed by computing a function

$\rho_{s}(\tau)=\frac{1}{n_{p}}size\\{p\in P:r_{p,s}\leq\tau\\}.$

Here $n_{p}$ denotes the total number of problems in the set $P$, and $\tau$
is a parameter ranging between 1 and $r_{M}$. The max value $r_{M}$ is
determined to satisfy $r_{M}\geq r_{p,s}$ for all $p$ in the set $P$ and $s$
in the set $S$. This function is then plotted for all solvers $s$ to produce
the performance profiles; the best-performing algorithms have profiles in the
upper left of the plot, and worse-performing algorithms in the lower right.
Typically a log scale is used for $\tau$ for ease of visualization; then the
value of $\rho_{s}(\tau)$ at zero (when $\tau=1$) provides the percentage of
problems in which solver $s$ achieved the best performance.

In our experiments, we focus on three different performance metrics: the
relative error between $f_{\min}$ and $f_{*}$ (if $f_{*}$ is non-zero), or the
absolute error if $f_{*}$ is zero; the total number of function evaluations
needed to obtain $f_{\min}$; and total CPU time consumed by the algorithm. An
algorithm is considered to solve a problem $p$ if the error between $f_{\min}$
and $f_{*}$ is less than $10^{-1}$. If solver $s$ fails to solve problem $p$,
$r_{p,s}$ is set equal to $r_{M}$.

To represent the 8 methods in the performance profiles, we use the notation
$WB_{i}$ for the Wolfe line search and the four choices of $\beta$,
($i=1,\ldots 4$). Likewise, we set the notation $NMB_{i}$ for the nonmonotone
line search.

The performance profile using error as the performance measure is shown in
Figure (2). The plot shows the algorithm with Wolfe line search and parameter
$\beta=0$, $WB_{0}$, as the most precise solver overall since it solves
accurately $60\%$ of the problems, and $70\%$ of them if a small error is
tolerated. However, $WB_{0}$ didn’t solve the other $30\%$ of the problems
based on our criteria. This makes $WB_{0}$ the most accurate algorithm but the
least robust.

On the other hand, $NMB_{2}$ solves $100\%$ of the problems when some error is
tolerated, though only $20\%$ with maximum accuracy. This makes $NMB_{2}$ the
most robust algorithm. It is worth mentioning $WB_{2}$ as a good alternative
to combine accuracy and robustness, solving accurately $50\%$ of the problems,
and $90\%$ of them when some error is tolerated. We note that $NMB_{0}$,
originally presented at [24], is outperformed by both $WB_{0}$ and $NMB_{2}$
in terms of accuracy and robustness, respectively.

Figure 2: Performance profile based on error between $f_{\min}$ and $f_{*}$
using the nonmonotone line search with M=7, and the Wolfe line search per each
parameter $\beta$. Figure 3: Performance profile based on number of function
evaluations using the nonmonotone line search with M=7, and the Wolfe line
search per each parameter $\beta$

.

Figure 3 shows the performance profile based on total number of function
evaluations needed to find $f_{m}in$. It shows the spectral subgradient method
$NMB_{0}$[24] as the most efficient algorithm. $NMB_{0}$ solves $80\%$ of the
problems with the fewest function evaluations. Similarly, $NMB_{1}$ presents
as an efficient option, slightly worse than $NMB_{0}$. However, $NMB_{0}$ and
$NMB_{1}$ were unable to solve one problem to within the specified threshold.

On the other hand, $NMB_{2}$ solves $100\%$ of the problems to within the
specified threshold, while requiring only somewhat more function evaluations
than $NMB_{0}$ and $NMB_{1}$. This would make the trade-off robustness versus
efficiency worthy, making $NMB_{2}$ the best alternative in our opinion.

Figure 3 also shows that the nonmonotone line search considerably outperforms
the Wolfe line search with respect to the number of function evaluations
required. Indeed, solvers using Wolfe line search are clearly on the bottom
right side of the chart, meaning they solve the fewest problems at the highest
cost. It is worth mentioning that Figure 2 showed $WB_{0}$ as the most precise
solver and $WB_{2}$ as a suitable alternative in terms of precision. But
Figure 3 shows them both as the most expensive in terms of function
evaluations, which makes these options unattractive.

Figure 4: Performance profile based on CPU using the nonmonotone line search
with M=7, and the Wolfe line search per each parameter $\beta$

With respect to CPU time, Figure 4 shows quite similar behavior when compared
to the profile based on function evaluations (Figure 3). This outcome is
anticipated since evaluating the function is the primary factor that drives
the computational effort of these algorithms. $NMB_{2}$ seems to be the best
alternative, being more robust than $NMB_{0}$ and $NMB_{1}$, given it solves
$100\%$ of the problems, although slightly worse performance.

After analyzing all performance profiles, we can conclude that $NMB_{2}$ is
the best alternative when factoring in all the criteria. This is based on its
high robustness, as it was the only method that solved all ten problems to
within the specified threshold. $NMB_{2}$ also provides solutions with
accuracy comparable with the rest of the solvers, although clearly
outperformed by $WB_{0}$ on this measure. And, $NMB_{2}$ is in the group of
top performing solvers based on computation time, with only slightly poorer
performance than $NMB_{0}$ and $NMB_{1}$.

In conclusion, we can state that the Wolfe line search tends to improve the
accuracy of the algorithms, but clearly at expense of many more function
evaluations, and hence computational effort.

###  4.2 CT problem with total variation

In this section, we study the effect of the choice of parameter $\beta$ when
applying Algorithm SCS to a set of TV-regularized CT reconstruction problems.
All experiments were run using the AIRTools II Matlab toolbox [17] to generate
parallel-beam CT system matrices $A$ and the corresponding sinograms $b$. An
image size of $400\times 400$ pixels was used, resulting in a problem size of
$n=160,000$. We then applied Algorithm SCS to the regularized least squares
problem (14), using TV (15) as the regularizer, whose subgradient was computed
using (16) and the subsequently discussed method.

A total of 24 reconstruction problems were considered, corresponding to:

  * •

Three different images (phantoms) from the AIRTools II toolbox: shepplogan,
threephases, and grains. All three phantoms are piecewise constant, making
them well-suited to total variation regularization. They are also all elements
of the $n$-dimensional box $[0,1]^{n}\subset\mathbb{R}^{n}$.

  * •

Four different imaging scenarios: two low-dose scenarios with Gaussian noise
of 5% and 10% added to $b$, and two sparse-view scenarios using 60 and 30
angular views. In the two low-dose scenarios, a total of 360 views and 566
lines through the object were simulated, giving a sinogram size of
$m=203,760$; this is reduced by a factor of 6 and 12, respectively, for the
two sparse-view scenarios. The sparse-view scenarios did not include noise.

  * •

Two different values of $\mu$, controlling the weighting of the regularization
term in (14). For the two low-dose scenarios, these were $\mu=25$ and
$\mu=250$, while for the sparse-view scenarios, we used $\mu=0.5$ and $\mu=5$.
These values were chosen empirically to give generally good results, while
providing different weighting on the nonsmooth (TV) component of the function.
Note that in the sparse-view scenarios, the size of the first term in (14) is
smaller than in the low-dose scenarios (since the sinogram size is smaller),
which is why a smaller value of $\mu$ can be used.

Additionally, because the desired solution to each problem is a vector
$x\in[0,1]^{n}$, we modified Algorithm SCS to include projection onto this set
(denoted $\Omega$). This is accomplished as follows: before every iteration of
Step 1, redefine $d_{k}$ as

$d_{k}=P_{\Omega}(x_{k}+d_{k})-x_{k},$ (17)

where $P_{\Omega}(x)=\min\\{\max\\{x,0\\},1\\}$ is the projection onto
$\Omega$. This guarantees that $x_{k}\in\Omega$ for all $k$, provided that the
step size $\alpha_{k}$ does not exceed 1. As this is not the case for the
Wolfe line search, in this section we only use the nonmonotone line search
with $M=7$. We, therefore, consider only four algorithms, namely NM$\beta_{0}$
to NM$\beta_{3}$ from the previous section. We note that the use of Wolfe line
search is prohibitively expensive for this problem in any event, due to the
high computational cost of function evaluations. Algorithm SCS was run for 200
total iterations for every case, beginning from $x_{0}=0\in\mathbb{R}^{n}$.

Figure 5 shows the performance profile based on the lowest objective function
value, $f_{\min}$ found by each solver for each problem. Note that this
differs from the measure used for the nonsmooth test problems (relative or
absolute difference between $f_{\min}$ and the true optimum $f_{*}$), since
$f_{*}$ is unknown for these problems. A solver was considered to have solved
the problem only if $f_{\min}$ was within 10% of the lowest value found. We
observe that the choice of $\beta_{2}$ provides the best overall results,
solving 80% of the problems to a high degree of accuracy; furthermore, it was
the only solver to solve all problems within the 10% threshold. The choice of
$\beta_{3}$ was the next best-performing, followed by $\beta_{1}$ and finally
$\beta_{0}$ (the spectral subgradient method), which was competitive on only
about 45% of the problems, and failed to solve nearly 50% to the required
tolerance.

Figure 5: Performance profile for CT reconstruction problems, based on lowest
objective function value $f_{\min}$ found by Algorithm SCS using nonmonotone
line search with each choice of $\beta$.

Figure 6 shows the performance profile generated for the four solvers, using
the total number of function evaluations as the performance measure. The
profile based on CPU time is omitted, as the results were essentially the
same. Unlike for the nonsmooth test problems (Figure 3), we observe that the
SCS algorithm using $\beta_{2}$ required significantly fewer function
evaluations than when using $\beta_{1}$ or $\beta_{3}$. As a total of 200
iterations was run for each solver, this indicates that the direction chosen
as a result of using $\beta_{2}$ generally resulted in less backtracking than
the other methods. We note that while the choice of $\beta_{0}$ typically
resulted in the fewest number of function evaluations needed, it was also
unsuccessful in solving nearly half the problems, as discussed in the previous
paragraph.

Figure 6: Performance profile for CT reconstruction problems, based on number
of function evaluations used by Algorithm SCS using nonmonotone line search
with each choice of $\beta$.

Figure 7 shows the best images reconstructed by Algorithm SCS using
$\beta_{2}$. We can observe that the reconstructed images are generally of
high quality, although there are still noticeable artifacts in some of the
images corresponding to more challenging scenarios (third and fifth columns).
It is possible that increasing the value of the weighting parameter, $\mu$,
would improve image quality in these cases.

Figure 7: Best images reconstructed by Algorithm SCS using $\beta_{2}$. The
first column shows the true phantom images used to generate the data, while
the remaining four columns show images reconstructed for the four scenarios
(low-dose imaging with 5% and 10% Gaussian noise, and sparse-view imaging with
60 and 30 views, respectively). The larger of the two $\mu$ values is used for
each scenario

##  5 Final Remarks

In this article, we develop a new Spectral Conjugate Subgradient (SCS) method
for nonsmooth unconstrained optimization, by combining conjugate directions
proposed by Birgin & Martínez [5] and the spectral subgradient method [24]. We
investigate the use of two different line search strategies and several
choices of the parameter $\beta$ used to determine conjugate directions. We
present the results of numerical experiments with standard nonsmooth test
problems, as well as problems from CT imaging using total variation
regularizations. The results are examined utilizing performance profiles,
which enable the comparison of various approaches in terms of precision and
computational cost.

The numerical results show that the combination of the nonmonotone line search
with the parameter $\beta_{2}$ (analogous to the Polak–Ribière formula), which
we denote by $NMB_{2}$, was the most successful approach. This is based on its
high robustness, given that it solved all the problems to within a specified
threshold, for both the nonsmooth test problems and the CT imaging problems.
Additionally, the computational effort required by $NMB_{2}$ was on par with
the least expensive approaches in both sets of experiments.

It is interesting to note that in Birgin & Martínez’ original article they
found that the choice of $\beta_{1}$ (Perry’s formula) was the best option,
while the best option in our experiments was $\beta_{2}$. It is difficult to
compare the results directly, given that their experiments were only on smooth
problems, using Wolfe line search. However, a possible explanation for why the
choice of $\beta_{2}$ works better in our experiments is that the formula for
$\beta_{1}$ has a denominator of $s_{k}^{T}y_{k}$, which is more often equal
to zero when solving nonsmooth problems than for differentiable problems.
Indeed, in our numerical experimentation we noticed this term is often zero
for some problems. In this case, we set $\beta_{1}=0$, defaulting to the
spectral subgradient approach. On the other hand, when the problem is
differentiable, the Wolfe line search ensures that $s_{k}^{T}y_{k}>0$,
avoiding this issue.

Finally, further investigation into the convergence properties of the SCS
method would be worthwhile. Loreto et al. [24] presented convergence results
on the spectral subgradient method, and so did Birgin for the spectral
conjugate gradient method. A combination of these existing convergence
results, with some additional hypotheses could be a potential path to prove
the convergence of the SCS.

## Disclosure statement

The authors do not have any significant financial or non-financial interests
to declare.

## Funding

The National Science Foundation provided support for this work (grant
DMS-2150511).

## Notes on contributor(s)

M. Loreto and T. Humphries co-authored the manuscript. C. Raghavan, K. Wu and
S. Kwak participated in a Research Experience for Undergraduates (REU) funded
by the above-mentioned NSF Grant in Summer 2022, providing initial
implementation of the spectral conjugate subgradient method and preliminary
experimentation.

## Data Availability Statement

The authors can provide the MATLAB code and data used in this work upon
request.

##  6 References

## References

  * Andrei [2007a] Andrei, Neculai. 2007a. “A scaled BFGS preconditioned conjugate gradient algorithm for unconstrained optimization.” _Applied Mathematics Letters_ 20 (6): 645–650. https://www.sciencedirect.com/science/article/pii/S0893965906002382.
  * Andrei [2007b] Andrei, Neculai. 2007b. “Scaled conjugate gradient algorithms for unconstrained optimization.” _Computational Optimization and Applications_ 38 (3): 401–416.
  * Andrei [2010] Andrei, Neculai. 2010. “New accelerated conjugate gradient algorithms as a modification of Dai-Yuan’s computational scheme for unconstrained optimization.” _Journal of Computational and Applied Mathematics_ 234 (12): 3397–3410.
  * Barzilai and Borwein [1988] Barzilai, J., and J. M. Borwein. 1988. “Two point step size gradient methods.” _IMA Journal of Numerical Analysis_ 8: 141–148.
  * Birgin and Martínez [2001] Birgin, E. G., and J. M. Martínez. 2001. “A Spectral Conjugate Gradient Method for Unconstrained Optimization.” _Applied Mathematics Optimization_ 43: 117–128.
  * Birgin, Martínez, and Raydan [2009] Birgin, E. G., J. M. Martínez, and M. Raydan. 2009. “Spectral Projected Gradient methods.” _encyclopedia of optimization, (2nd Edn)_ 3652–3659.
  * Birgin, Martínez, and Raydan [2014] Birgin, E. G., J. M. Martínez, and M. Raydan. 2014. “Spectral Projected Gradient methods: Review and Perspectives.” _Journal of Statistical Software_ 60 (3): 1–21.
  * Cheng and Liu [2010] Cheng, Wanyou, and Qunfeng Liu. 2010. “Sufficient descent nonlinear conjugate gradient methods with conjugacy condition.” _Numerical Algorithms_ 53 (1): 113–131.
  * Crema, Loreto, and Raydan [2007] Crema, A., M. Loreto, and M. Raydan. 2007. “Spectral Projected Subgradient with a momentum term for the Lagrangean dual approach.” _Computers and Operation Research_ 34: 3174–3186.
  * Cruz, Martínez, and Raydan [2006] Cruz, W. La, J. M. Martínez, and M. Raydan. 2006. “Spectral residual method without gradient information for solving large-scale nonlinear systems.” _Math. of Comp._ 75: 1449–1466.
  * Deng, Wan, and Chen [2013] Deng, Songhai, Zhong Wan, and Xiaohong Chen. 2013. “An improved spectral conjugate gradient algorithm for nonconvex unconstrained optimization problems.” _Journal of optimization theory and applications_ 157 (3): 820–842.
  * Dolan and Moré [2002] Dolan, E. D., and J. J. Moré. 2002. “Benchmarking optimization software with performance profiles.” _Math. Program_ 91: 201 – 213.
  * Du and Chen [2008] Du, Shou-qiang, and Yuan-yuan Chen. 2008. “Global convergence of a modified spectral FR conjugate gradient method.” _Applied Mathematics and Computation_ 202 (2): 766–770.
  * Fletcher [1990] Fletcher, R. 1990. “Low storage methods for unconstrained optimization.” _Lectures in Applied Mathematics (AMS)_ 26: 165–179.
  * Glunt, Hayden, and Raydan [1993] Glunt, W, Tom L Hayden, and Marcos Raydan. 1993. “Molecular conformations from distance matrices.” _Journal of Computational Chemistry_ 14 (1): 114–120.
  * Grippo, Lampariello, and Lucidi [1986] Grippo, L., F. Lampariello, and S. Lucidi. 1986. “A nonmonotone line search technique for Newton’s method.” _SIAM Journal on Numerical Analysis_ 4 (23): 707–716.
  * Hansen and Jørgensen [2018] Hansen, P. C., and J. S. Jørgensen. 2018. “AIR Tools II: algebraic iterative reconstruction methods, improved implementation.” _Numerical Algorithms_ 79 (1): 107–137.
  * Karmitsa [2007] Karmitsa, N. 2007. “Test Problems for Large-Scale Nonsmooth Minimization.” .
  * Krutikov, Samoilenko, and Meshechkin [2019] Krutikov, Vladimir N, Natalia S Samoilenko, and Vladimir V Meshechkin. 2019. “On the properties of the method of minimization for convex functions with relaxation on the distance to extremum.” _Automation and remote control_ 80 (1): 102–111.
  * Liu, Feng, and Zou [2019] Liu, JK, YM Feng, and LM Zou. 2019. “A spectral conjugate gradient method for solving large-scale unconstrained optimization.” _Computers & Mathematics with Applications_ 77 (3): 731–739.
  * Livieris and Pintelas [2012] Livieris, Ioannis E., and Panagiotis Pintelas. 2012. “Globally convergent modified Perry’s conjugate gradient method.” _Applied Mathematics and Computation_ 218 (18): 9197–9207. https://www.sciencedirect.com/science/article/pii/S0096300312002263.
  * Loreto et al. [2017] Loreto, M., H. Aponte, D. Cores, and M. Raydan. 2017. “Nonsmooth spectral gradient method for unconstrained optimization.” _EURO J. Comput. Optim._ 5 (4): 529–553.
  * Loreto and Crema [2015] Loreto, M., and A. Crema. 2015. “Convergence Analysis for the Modified Spectral Projected Subgradient Method.” _Optimization Letters_ 9: 915–929.
  * Loreto, Xu, and Kotval [2019] Loreto, M., Y. Xu, and D. Kotval. 2019. “A numerical study of applying spectral-step subgradient method for solving nonsmooth unconstrained optimization problems.” _Computers and Operation Research_ 104: 90–97.
  * Loreto et al. [2016] Loreto, Milagros, Samantha Clapp, Charles Cratty, and Breeanna Page. 2016. “Modified spectral projected subgradient method: convergence analysis and momentum parameter heuristics.” _Bulletin of Computational Applied Mathematics, CompAMa_ 4: 28–54.
  * Neto and De Pierro [2010] Neto, Elias Salomão Helou, and Álvaro Rodolfo De Pierro. 2010. “Incremental Subgradients for Constrained Convex Optimization: A Unified Framework and New Methods.” _SIAM Journal on Optimization_ 20 (3): 1547–1572. https://doi.org/10.1137/070711712.
  * Nurminskii and Tien [2014] Nurminskii, Evgeni A, and David Tien. 2014. “Method of conjugate subgradients with constrained memory.” _Automation and Remote Control_ 75 (4): 646–656.
  * Perry [1978] Perry, A. 1978. “A modified conjugate gradient algorithm.” _Operation Research_ 26: 1073––1078.
  * Raydan [1993] Raydan, M. 1993. “On the Barzilai and Borwein choice of steplength for the gradient method.” _IMA Journal of Numerical Analysis_ 13: 321–326.
  * Raydan [1997] Raydan, M. 1997. “The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem.” _SIAM Journal on Optimization_ 7: 26–33.
  * Taviani [n.d.] Taviani, Davide. n.d. “Line Search Algorithm with Strong Wolfe conditions.” https://gist.github.com/Heliosmaster/1043132. Created: 2011-06-23.
  * Wolfe [1974] Wolfe, Philip. 1974. “Note on a method of conjugate subgradients for minimizing nondifferentiable functions.” _Mathematical Programming_ 7 (1): 380–383.
  * Wolfe [1975] Wolfe, Philip. 1975. “A method of conjugate subgradients for minimizing nondifferentiable functions.” In _Nondifferentiable optimization_ , 145–173. Springer.
  * Zhang and Zhou [2008] Zhang, Li, and Weijun Zhou. 2008. “Two descent hybrid conjugate gradient methods for optimization.” _Journal of Computational and Applied Mathematics_ 216 (1): 251–264. https://www.sciencedirect.com/science/article/pii/S0377042707002348.
  * Zhang, Zhou, and Li [2006] Zhang, Li, Weijun Zhou, and Donghui Li. 2006. “Global convergence of a modified Fletcher-Reeves conjugate gradient method with Armijo-type line search.” _Numerische mathematik_ 104 (4): 561–572.


* Alahyari et al. (2017) Hiva Alahyari, Richard Berntsson Svensson, and Tony Gorschek. 2017.  A study of value in agile software development organizations.  _Journal of Systems and Software_ 125 (2017), 271–288.
  * Alami et al. (2022) Adam Alami, Oliver Krancher, and Maria Paasivaara. 2022.  The journey to technical excellence in agile software development.  _Information and Software Technology_ 150 (2022), 106959.
  * Albusays et al. (2021) Khaled Albusays, Pernille Bjorn, Laura Dabbish, Denae Ford, Emerson Murphy-Hill, Alexander Serebrenik, and Margaret-Anne Storey. 2021.  The diversity crisis in software development.  _IEEE Software_ 38, 2 (2021), 19–25.
  * Anthropic (2023) Anthropic. 2023.  Claude AI.  https://claude.ai.  Accessed: 13-April-2024.
  * Asia (2024) Microsoft News Center Asia. 2024.  _Microsoft in Asia: Empowering a new global innovation engine that is transforming economies and societies_.  https://news.microsoft.com/apac/features/microsoft-in-asia-empowering-a-new-global-innovation-engine-that-is-transforming-economies-and-societies/ Accessed: 2024-06-11.
  * Baltes and Ralph (2022) Sebastian Baltes and Paul Ralph. 2022.  Sampling in software engineering research: A critical review and guidelines.  _Empirical Software Engineering_ 27, 4 (2022), 94.
  * Barke et al. (2023) Shraddha Barke, Michael B James, and Nadia Polikarpova. 2023.  Grounded copilot: How programmers interact with code-generating models.  _Proceedings of the ACM on Programming Languages_ 7, OOPSLA1 (2023), 85–111.
  * Barkhi et al. (2006) Reza Barkhi, Ali Amiri, and Tabitha L James. 2006.  A study of communication and coordination in collaborative software development.  _Journal of Global Information Technology Management_ 9, 1 (2006), 44–61.
  * Beganovic et al. (2023) Adna Beganovic, Muna Abu Jaber, and Ali Abd Almisreb. 2023.  Methods and applications of chatgpt in software development: A literature review.  _Southeast Europe Journal of Soft Computing_ 12, 1 (2023), 08–12.
  * Borsci et al. (2022) Simone Borsci, Alessio Malizia, Martin Schmettow, Frank Van Der Velde, Gunay Tariverdiyeva, Divyaa Balaji, and Alan Chamberlain. 2022.  The chatbot usability scale: the design and pilot of a usability scale for interaction with AI-based conversational agents.  _Personal and ubiquitous computing_ 26 (2022), 95–119.
  * Brade et al. (2023) Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023.  Promptify: Text-to-image generation through interactive prompt exploration with large language models. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_. 1–14.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Clance and Imes (1978) Pauline Rose Clance and Suzanne Ament Imes. 1978.  The imposter phenomenon in high achieving women: Dynamics and therapeutic intervention.  _Psychotherapy: Theory, research & practice_ 15, 3 (1978), 241.
  * CSO Online (2023) CSO Online. 2023.  The Biggest Data Breach Fines, Penalties, and Settlements So Far.  _CSO Online_ (2023).  https://www.csoonline.com/article/567531/the-biggest-data-breach-fines-penalties-and-settlements-so-far.html Accessed: 25 May 2024.
  * Cummings and Berger (1976) Larry L Cummings and Chris J Berger. 1976.  Organization structure: How does it influence attitudes and performance?  _Organizational Dynamics_ 5, 2 (1976), 34–49.
  * Dakhel et al. (2023) Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023.  Github copilot ai pair programmer: Asset or liability?  _Journal of Systems and Software_ 203 (2023), 111734.
  * de Souza Santos and Ralph (2022) Ronnie E de Souza Santos and Paul Ralph. 2022.  A grounded theory of coordination in remote-first and hybrid software teams. In _Proceedings of the 44th International Conference on Software Engineering_. 25–35.
  * Diaz ([n. d.]) Maria Diaz. [n. d.].  ChatGPT vs. ChatGPT Plus: Is a paid subscription still worth it?  https://www.zdnet.com/article/chatgpt-vs-chatgpt-plus-is-a-paid-subscription-still-worth-it/
  * Díaz-Rodríguez et al. (2023) Natalia Díaz-Rodríguez, Javier Del Ser, Mark Coeckelbergh, Marcos López de Prado, Enrique Herrera-Viedma, and Francisco Herrera. 2023.  Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation.  _Information Fusion_ 99 (2023), 101896.
  * DPM (2021) DPM. 2021.  Luxembourg DPA issues €746 Million GDPR Fine to Amazon.  https://dataprivacymanager.net/luxembourg-dpa-issues-e746-million-gdpr-fine-to-amazon/
  * Du et al. (2024) Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024.  Evaluating large language models in class-level code generation. In _Proceedings of the IEEE/ACM 46th International Conference on Software Engineering_. 1–13.
  * Finocchiaro (2023) Giusella Finocchiaro. 2023.  The regulation of artificial intelligence.  _AI & SOCIETY_ (2023), 1–8.
  * Froeschl et al. (2020) Leslie Froeschl, Scott Wells, and Marques Bland. 2020.  Best Practices in Project Management in a Large, Distributed Organization: Lessons Learned from XSEDE.  In _Practice and Experience in Advanced Research Computing_. 386–392.
  * Gama and Lacerda (2023) Kiev Gama and Aline Lacerda. 2023.  Understanding and Supporting Neurodiverse Software Developers in Agile Teams. In _Proceedings of the XXXVII Brazilian Symposium on Software Engineering_. 497–502.
  * GitHub (2023) GitHub. 2023.  GitHub Copilot- Your AI pair programmer.  https://copilot.github.com.  Accessed: 12-Nov-2023.
  * GitLab (2023) GitLab. 2023.  2023 Global DevSecOps Report Series.  https://about.gitlab.com/developer-survey/.  Accessed: 2024-03-20.
  * Graetsch et al. (2023) Ulrike M. Graetsch, Hourieh Khalajzadeh, Mojtaba Shahin, Rashina Hoda, and John Grundy. 2023.  Dealing with Data Challenges when Delivering Data-Intensive Software Solutions.  http://arxiv.org/abs/2209.14055 arXiv:2209.14055 [cs].
  * Gunatilake et al. (2024) Hashini Gunatilake, John Grundy, Rashina Hoda, and Ingo Mueller. 2024.  Enablers and Barriers of Empathy in Software Developer and User Interactions: A Mixed Methods Case Study.  _ACM Transactions on Software Engineering and Methodology_ 33, 4 (May 2024), 1–41.  https://doi.org/10.1145/3641849
  * Hacker et al. (2023) Philipp Hacker, Andreas Engel, and Marco Mauer. 2023.  Regulating ChatGPT and other large generative AI models. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_. 1112–1123.
  * Harvard Business Review (2023) Harvard Business Review. 2023.  How global companies use AI to prevent supply chain disruptions.  _Harvard Business Review_ (November 2023).  https://hbr.org/2023/11/how-global-companies-use-ai-to-prevent-supply-chain-disruptions Accessed: 25 May 2024.
  * Herbold et al. (2023) Steffen Herbold, Annette Hautli-Janisz, Ute Heuer, Zlata Kikteva, and Alexander Trautsch. 2023.  A large-scale comparison of human-written versus ChatGPT-generated essays.  _Scientific Reports_ 13, 1 (2023), 18617.
  * Hidellaarachchi et al. (2023) Dulaji Hidellaarachchi, John Grundy, Rashina Hoda, and Ingo Mueller. 2023.  Understanding the influence of motivation on requirements engineering-related activities.  _arXiv preprint arXiv:2304.08074_ (2023).
  * Hoda (2021) Rashina Hoda. 2021.  Socio-technical grounded theory for software engineering.  _IEEE Transactions on Software Engineering_ 48, 10 (2021), 3808–3832.
  * Huang et al. (2021) Yu Huang, Denae Ford, and Thomas Zimmermann. 2021.  Leaving my fingerprints: Motivations and challenges of contributing to OSS for social good. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 1020–1032.
  * Imai (2022) Saki Imai. 2022.  Is github copilot a substitute for human pair-programming? an empirical study. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion).
  * Itkonen et al. (2012) Juha Itkonen, Mika V Mäntylä, and Casper Lassenius. 2012.  The role of the tester's knowledge in exploratory software testing.  _IEEE Transactions on Software Engineering_ 39, 5 (2012), 707–724.
  * Jaworski and Piotrkowski (2023) Mateusz Jaworski and Dariusz Piotrkowski. 2023.  Study of software developers' experience using the Github Copilot Tool in the software development process.  _arXiv preprint arXiv:2301.04991_ (2023).
  * Jiang et al. (2022) Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2022.  Discovering the syntax and strategies of natural language programming with generative language models. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–19.
  * Jo (2023) Hyeon Jo. 2023.  Decoding the ChatGPT mystery: A comprehensive exploration of factors driving AI language model adoption.  _Information Development_ (2023), 02666669231202764.
  * Johnson et al. (2023) Brittany Johnson, Christian Bird, Denae Ford, Nicole Forsgren, and Thomas Zimmermann. 2023.  Make Your Tools Sparkle with Trust: The PICSE Framework for Trust in Software Tools. In _2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 409–419.
  * Kalliamvakou (2022) Eirini Kalliamvakou. 2022.  Research: quantifying GitHub Copilot’s impact on developer productivity and happiness.  https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/
  * Li et al. (2022) Annie Li, Madeline Endres, and Westley Weimer. 2022.  Debugging with stack overflow: Web search behavior in novice and expert programmers. In _Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Software Engineering Education and Training_. 69–81.
  * Liang et al. (2024) Jenny T Liang, Chenyang Yang, and Brad A Myers. 2024.  A large-scale survey on the usability of ai programming assistants: Successes and challenges. In _Proceedings of the 46th IEEE/ACM International Conference on Software Engineering_. 1–13.
  * Liang et al. (2022) Jenny T Liang, Thomas Zimmermann, and Denae Ford. 2022.  Understanding skills for oss communities on github. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 170–182.
  * Lingard (2023) Lorelei Lingard. 2023.  Writing with ChatGPT: An illustration of its capacity, limitations & implications for academic writers.  _Perspectives on medical education_ 12, 1 (2023), 261.
  * Liu and Chilton (2022) Vivian Liu and Lydia B Chilton. 2022.  Design guidelines for prompt engineering text-to-image generative models. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–23.
  * Madampe et al. (2023a) Kashumi Madampe, Rashina Hoda, and John Grundy. 2023a.  The Emotional Roller Coaster of Responding to Requirements Changes in Software Engineering.  _IEEE Transactions on Software Engineering_ 49, 3 (March 2023), 1171–1187.  https://doi.org/10.1109/TSE.2022.3172925 Conference Name: IEEE Transactions on Software Engineering.
  * Madampe et al. (2023b) Kashumi Madampe, Rashina Hoda, and John Grundy. 2023b.  A Framework for Emotion-Oriented Requirements Change Handling in Agile Software Engineering.  _IEEE Transactions on Software Engineering_ 49, 5 (May 2023), 3325–3343.  https://doi.org/10.1109/TSE.2023.3253145 Conference Name: IEEE Transactions on Software Engineering.
  * Matt and Rauch (2020) Dominik T Matt and Erwin Rauch. 2020.  SME 4.0: The role of small-and medium-sized enterprises in the digital transformation.  _Industry 4.0 for SMEs: Challenges, opportunities and requirements_ (2020), 3–36.
  * Mergel (2015) Ines Mergel. 2015.  Open collaboration in the public sector: The case of social coding on GitHub.  _Government Information Quarterly_ 32, 4 (2015), 464–472.
  * Meta (2023) Meta. 2023.  LLaMA: Language Model.  https://llama.meta.com.  Accessed: 13-April-2024.
  * Meyer et al. (2019) André N Meyer, Earl T Barr, Christian Bird, and Thomas Zimmermann. 2019.  Today was a good day: The daily life of software developers.  _IEEE Transactions on Software Engineering_ 47, 5 (2019), 863–880.
  * Michailova and Hutchings (2006) Snejina Michailova and Kate Hutchings. 2006.  National cultural influences on knowledge sharing: A comparison of China and Russia.  _Journal of management studies_ 43, 3 (2006), 383–405.
  * Moore and McKenna (1999) Geoffrey A Moore and Regis McKenna. 1999.  Crossing the chasm.  (1999).
  * Nguyen and Nadi (2022) Nhan Nguyen and Sarah Nadi. 2022.  An empirical evaluation of GitHub copilot's code suggestions. In _Proceedings of the 19th International Conference on Mining Software Repositories_. 1–5.
  * OpenAI (2022) OpenAI. 2022.  ChatGPT: Optimizing Language Models for Dialogue.  https://www.openai.com.  Accessed: 12-Nov-2023.
  * Pant et al. (2024a) Aastha Pant, Rashina Hoda, Simone V Spiegler, Chakkrit Tantithamthavorn, and Burak Turhan. 2024a.  Ethics in the age of AI: an analysis of ai practitioners’ awareness and challenges.  _ACM Transactions on Software Engineering and Methodology_ 33, 3 (2024), 1–35.
  * Pant et al. (2024b) Aastha Pant, Rashina Hoda, Chakkrit Tantithamthavorn, and Burak Turhan. 2024b.  Navigating Fairness: Practitioners' Understanding, Challenges, and Strategies in AI/ML Development.  _arXiv preprint arXiv:2403.15481_ (2024).
  * Pascual (2023) Manuel G. Pascual. 2023.  Geoffrey Hinton: ‘We need to find a way to control artificial intelligence before it’s too late’.  https://english.elpais.com/science-tech/2023-05-12/geoffrey-hinton-we-need-to-find-a-way-to-control-artificial-intelligence-before-its-too-late.html Section: Science & Tech.
  * Patton (2002) Michael Quinn Patton. 2002.  _Qualitative research & evaluation methods_.  sage.
  * Peng et al. (2023) Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023.  The impact of ai on developer productivity: Evidence from github copilot.  _arXiv preprint arXiv:2302.06590_ (2023).
  * Ponzanelli et al. (2014) Luca Ponzanelli, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, and Michele Lanza. 2014.  Mining stackoverflow to turn the ide into a self-confident programming prompter. In _Proceedings of the 11th working conference on mining software repositories_. 102–111.
  * Porter (2023) J Porter. 2023.  Chatgpt continues to be one of the fastest-growing services ever.  _Retrieved December_ 3 (2023).
  * Raghavan R et al. (2020) Srinivasa Raghavan R, Jayasimha KR, and Rajendra V Nargundkar. 2020.  Impact of software as a service (SaaS) on software acquisition process.  _Journal of Business & Industrial Marketing_ 35, 4 (2020), 757–770.
  * Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021.  Prompt programming for large language models: Beyond the few-shot paradigm. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–7.
  * Roller and Lavrakas (2015) Margaret R. Roller and Paul J. Lavrakas. 2015.  _Applied qualitative research design: A total quality framework approach_.  The Guilford Press, New York, NY, US.  Pages: xviii, 398.
  * Ross et al. (2023) Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. 2023.  The programmer’s assistant: Conversational interaction with a large language model for software development. In _Proceedings of the 28th International Conference on Intelligent User Interfaces_. 491–514.
  * Russo (2023) Daniel Russo. 2023.  Navigating the complexity of generative ai adoption in software engineering.  _ACM Transactions on Software Engineering and Methodology_ (2023).
  * Ryan et al. (2021) Mark Ryan, Josephina Antoniou, Laurence Brooks, Tilimbe Jiya, Kevin Macnish, and Bernd Stahl. 2021.  Research and practice of AI ethics: a case study approach juxtaposing academic discourse with organisational reality.  _Science and Engineering Ethics_ 27 (2021), 1–29.
  * Sadowski et al. (2015) Caitlin Sadowski, Kathryn T Stolee, and Sebastian Elbaum. 2015.  How developers search for code: a case study. In _Proceedings of the 2015 10th joint meeting on foundations of software engineering_. 191–201.
  * Saldaña (2021) Johnny Saldaña. 2021.  The coding manual for qualitative researchers.  (2021).
  * Seong et al. (2024) Jeongmin Seong, Chris Bradley, and Angela Buensuceso. 2024.  _Asia at the forefront of global change: On the cusp of a new era_.  https://www.mckinsey.com/featured-insights/future-of-asia/future-of-asia-podcasts/asia-at-the-forefront-of-global-change-on-the-cusp-of-a-new-era Accessed: 2024-06-11.
  * Sirur et al. (2018) Sean Sirur, Jason R. C. Nurse, and Helena Webb. 2018.  Are we there yet? Understanding the challenges faced in complying with the General Data Protection Regulation (GDPR).  _arXiv:1808.07338 [cs]_ (Aug. 2018).  arXiv: 1808.07338.
  * Strauss and Corbin (1990) Anselm Strauss and Juliet Corbin. 1990.  _Basics of qualitative research_.  Sage publications.
  * Tang et al. (2021) Jenny Tang, Hannah Shoemaker, Ada Lerner, and Eleanor Birrell. 2021.  Defining privacy: How users interpret technical terms in privacy policies.  _Proceedings on Privacy Enhancing Technologies_ (2021).
  * The New York Times (2018) The New York Times. 2018.  Cambridge Analytica Scandal Fallout.  _The New York Times_ (4 April 2018).  https://www.nytimes.com/2018/04/04/us/politics/cambridge-analytica-scandal-fallout.html Accessed: 25 May 2024.
  * Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022.  Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In _Chi conference on human factors in computing systems extended abstracts_. 1–7.
  * Warren (2024) Zach Warren. 2024.  Legalweek 2024: Current US AI regulation means adopting a strategic — and communicative — approach.  https://www.thomsonreuters.com/en-us/posts/legal/legalweek-2024-ai-regulation/
  * Weyssow et al. (2023) Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, and Houari Sahraoui. 2023.  Exploring parameter-efficient fine-tuning techniques for code generation with large language models.  _arXiv preprint arXiv:2308.10462_ (2023).
  * Williams and Adamson (2022) Branden Williams and James Adamson. 2022.  _PCI Compliance: Understand and implement effective PCI data security standard compliance_.  CRC Press.
  * Wu et al. (2023) Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. 2023.  A brief overview of ChatGPT: The history, status quo and potential future development.  _IEEE/CAA Journal of Automatica Sinica_ 10, 5 (2023), 1122–1136.
  * Wu et al. (2019) Yuhao Wu, Shaowei Wang, Cor-Paul Bezemer, and Katsuro Inoue. 2019.  How do developers utilize source code from stack overflow?  _Empirical Software Engineering_ 24 (2019), 637–673.
  * Xia et al. (2017) Xin Xia, Lingfeng Bao, David Lo, Pavneet Singh Kochhar, Ahmed E Hassan, and Zhenchang Xing. 2017.  What do developers search for on the web?  _Empirical Software Engineering_ 22 (2017), 3149–3185.
  * Xu et al. (2022) Frank F Xu, Bogdan Vasilescu, and Graham Neubig. 2022.  In-ide code generation from natural language: Promise and challenges.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 31, 2 (2022), 1–47.
  * Yahoo Finance (2024) Yahoo Finance. 2024.  Is AI Moving Too Fast to Keep Up With?  _Yahoo Finance_ (23 May 2024).  https://finance.yahoo.com/news/ai-moving-too-fast-keep-190958680.html? Accessed: 25 May 2024.
  * Yang et al. (2016) Di Yang, Aftab Hussain, and Cristina Videira Lopes. 2016.  From query to usable code: an analysis of stack overflow code snippets. In _Proceedings of the 13th International Conference on Mining Software Repositories_. 391–402.
  * Yang (2024) Zeyi Yang. 2024.  Four things to know about China’s new AI rules in 2024.  https://www.technologyreview.com/2024/01/17/1086704/china-ai-regulation-changes-2024/
  * Yetistiren et al. (2022) Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022.  Assessing the quality of GitHub copilot’s code generation. In _Proceedings of the 18th international conference on predictive models and data analytics in software engineering_. 62–71.
  * Zhang et al. (2023) Beiqi Zhang, Peng Liang, Xiyu Zhou, Aakash Ahmad, and Muhammad Waseem. 2023.  Practices and challenges of using github copilot: An empirical study.  _arXiv preprint arXiv:2303.08733_ (2023).
  * Ziegler et al. (2022) Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022.  Productivity assessment of neural code completion. In _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_. 21–29.


# Adaptive Knowledge Distillation between Text and Speech
Pre-trained Models

###### Abstract

Learning on a massive amount of speech corpus leads to the recent success of
many self-supervised speech models. With knowledge distillation, these models
may also benefit from the knowledge encoded by language models that are pre-
trained on rich sources of texts. The distillation process, however, is
challenging due to the modal disparity between textual and speech embedding
spaces. This paper studies metric-based distillation to align the embedding
space of text and speech with only a small amount of data without modifying
the model structure. Since the semantic and granularity gap between text and
speech has been omitted in literature, which impairs the distillation, we
propose the Prior-informed Adaptive knowledge Distillation (PAD) that
adaptively leverages text/speech units of variable granularity and prior
distributions to achieve better global and local alignments between text and
speech pre-trained models. We evaluate on three spoken language understanding
benchmarks to show that PAD is more effective in transferring linguistic
knowledge than other metric-based distillation approaches.

††∗Work done during an internship at Speech Lab, Alibaba Group,
Singapore.††∗Supported by the Agency for Science, Technology and Research
(A*STAR) under its AME Programmatic Funding Scheme (Project #A18A2b0046).

Index Terms—  Knowledge Distillation, Pre-trained Models, Spoken Language
Understanding

##  1 Introduction

Recent years witness a great success of pre-trained models in various domains,
e.g., natural language [1, 2, 3, 4, 5] and speech [6, 7, 8]. The pre-trained
models learn to encode the input into high-level features via self-supervised
learning tasks such as masked language modeling (MLM), next sentence
prediction, etc., and bring considerable performance gain to the downstream
tasks [1, 2, 9]. Spoken Language Understanding (SLU) is a set of downstream
tasks in the speech domain intending to learn models to understand the content
of speech input. State-of-the-art SLU models are mostly achieved by finetuning
on the speech pre-trained models [10].

Despite that the pre-trained speech encoder is powerful in preserving content-
based information, it still falls short in encoding linguistic information due
to the absence of symbolic representation. To complement the speech pre-
training, a speech encoder could benefit from transferring knowledge from
language models that are pre-trained on unlabelled texts. However, the
knowledge obtained during the pre-training stage of speech and text has a
natural gap caused by the modal bias and the difference in pre-training
methods [11]. This motivates us to study the problem of distilling knowledge
from text pre-trained models to speech ones to enhance their performance on
understanding tasks.

We categorize existing work on leveraging text knowledge for end-to-end SLU
into two classes: one-tower and two-tower. The one-tower approaches [11, 12]
encode uni-modal inputs via a shared encoder and is flexible in taking either
uni-modal or multi-modal inputs at inference stage. The two-tower methods use
modal-specific encoders for speech and text learning, and rely on either
additional layers/codebooks [11, 13, 14, 15, 16] or alignment losses to align
the multi-modal embedding space.

This work falls under the two-tower category aligning the text and speech
embedding space by designing metric objectives [17, 18], which we call metric-
based distillation. Compared with the one-tower methods, metric-based
distillation requires only a small amount of data to distill text knowledge
[18] with an explicit alignment objective. It does not introduce any
additional parameters, which ensures the model’s efficiency and separability.
However, due to the large semantic and granularity gap between speech and
text, it is challenging to design appropriate metrics for metric-based
distillation. Prior works [17, 18] use simple metrics (L2 distance and cosine
similarity) to perform metric-based distillation from text to speech ignoring
the semantic and granularity difference between them, which harms the
distillation effectiveness. To this end, we propose the Prior-informed
Adaptive knowledge Distillation (PAD) that aligns text-speech representations
at adaptive granularity, with the alignment process being informed by the
prior knowledge reflecting the semantic significance.

We made the following contributions:

  * •

We propose to apply the Attention-based Significance Priors (ASP) to ease the
semantic knowledge transferring from texts to speech.

  * •

We propose the Anchor-based Adaptive Span Aggregation algorithm (AASA) that
narrows the modal granularity gap of alignments.

  * •

To the best of our knowledge, we are the first that evaluate multiple
different alignment strategies beyond vanilla global and local alignments to
study the feasibility of metric-based speech-text distillations. The results
on three spoken language understanding benchmarks verify our assumptions and
claims.

##  2 Methodology

###  2.1 Preliminary: Global and Local Alignment

Metric-based distillation distills text knowledge by aligning the
representation space of speech and text pre-trained models. In general, there
are two aligning strategies: global and local alignments. Global-level
alignment narrows the gap between sequence-level representations and local-
level alignment similarizes the local unit representations. Once the alignment
training is finished, the speech representation space is closer to that of the
text and thus text knowledge is transferred. Given a pair of speech and text
data, the output of speech and text pre-trained models are denoted as
$\mathrm{\mathbf{S}}=[\mathrm{\mathbf{s}}_{1},...,\mathrm{\mathbf{s}}_{n}]$
and
$\mathrm{\mathbf{T}}=[\mathrm{\mathbf{t}}_{1},...,\mathrm{\mathbf{t}}_{n}]$
respectively, where $\mathrm{\mathbf{s}}_{m}\in\mathbb{R}^{d_{s}}$ and
$\mathrm{\mathbf{t}}_{m}\in\mathbb{R}^{d_{t}}$. $d_{s}$ and $d_{t}$ are the
model dimension of speech and text modules.

#### Global-level alignment.

Sequence-level representations can be extracted from the output
representations of the pre-trained models. We denote the global-level output
representation of speech and text model as
$\mathrm{\mathbf{\hat{s}}}\in\mathbb{R}^{d_{s}}$ and
$\mathrm{\mathbf{\hat{t}}}\in\mathbb{R}^{d_{t}}$, which are downsampled
embeddings that represents the sentence such as the $cls$ embedding in BERT or
the averaged embeddings. The L1 distance of the two sequence representations
is computed for alignment:

$\mathcal{L}_{Glob}=||\mathrm{\mathbf{\hat{s}}}-\mathrm{\mathbf{\hat{t}}}||_{1}$
(1)

During global-level alignment training, the text module is frozen and only the
speech module is updated.

#### Local-level alignment.

A finer alignment can be conducted locally. we cannot use the L1 distance as
in Eq.1 because the speech and text tokens do not have a one-to-one mapping.
The local token units of speech and text module are aligned by maximizing the
sum of the maximum text-speech similarities regarding all text units:

$\mathcal{L}_{Loc}=-\frac{1}{n}\sum_{j=0}^{n}max_{i}\phi(\mathrm{\mathbf{s}}_{i},\mathrm{\mathbf{t}}_{j})$
(2)

Where $\phi$ denotes the similarity metric such as cosine similarity. Same as
the global-level alignment, the text module is fixed and only the speech
module is updated.

Fig. 1: The global and local PAD. The global and local alignments (Sec.2.1)
are both informed by the ASP (Sec.2.2) to narrow the semantic gap. The AASA
(Sec.2.3) adaptively reorganizes the speech sequence to narrow the granularity
gap.

###  2.2 Attention-based Significance Priors

Both global and local alignments introduced in Sec.2.1 treat tokens in a
sequence equally. However, the modal bias of speech and text causes some
alignments to be meaningless, e.g., blank and noise signals of acoustic
sequences cannot be found in text, and trying to align these signals with text
tokens may not be beneficial to model’s understanding ability. Thus, we assume
that the alignment between acoustic features and their text counterpart should
be focused on only the most semantic-relevant tokens or frames, for example,
the non-blank speech tokens or the keywords in the text sequence. As a result,
we need to properly score the significance of tokens (or frames). [18] used
the inverse document frequency (idf) to re-weight the significance of frequent
token-frame pairs. However, idf has two problems: (a) it is fixed for each
dataset, whereas the significance of a token varies with different contexts;
(b) it is not applicable to speech features that are continuous-valued. We
propose Attention-based Significance Priors (ASP) that extracts the
significance distribution $\mathcal{P}_{sig}\in\mathbb{R}^{n}$ of an output
hidden sequence
$\mathrm{\mathbf{H}}=[\mathrm{\mathbf{h}}_{1},..,\mathrm{\mathbf{h}}_{n}]$
from its self-attention map of the pre-trained transformer models‡‡‡We find
that the last-layer attention map achieves a better estimation of the ASP in
some cases, which led to better performance. (Fig.1):

$\small\mathcal{P}_{sig}(\mathrm{\mathbf{H}})=\frac{1}{L_{0}}\sum_{l=1}^{L_{0}}\frac{1}{\mathrm{\mathbf{e}}A(\mathrm{\mathbf{H}})^{l}\mathrm{\mathbf{e}}^{T}}\sum_{m=1}^{n}A(\mathrm{\mathbf{H}})^{l}_{m}$
(3)

The significance score of $\mathrm{\mathbf{h}}_{m}$ is a normalized mean over
the attention weights of the units attending to $\mathrm{\mathbf{h}}_{m}$.
$A(\mathrm{\mathbf{H}})^{l}=[A(\mathrm{\mathbf{H}})^{l}_{1},...,A(\mathrm{\mathbf{H}})^{l}_{n}]$
denotes the $l$-th layer self-attention map;
$A(\mathrm{\mathbf{H}})^{l}_{m}\in\\{a\in\mathbb{R}|0<a<1\\}^{n}$ denotes the
attention weights of the units that attend to the $m$-th unit of
$\mathrm{\mathbf{H}}$, which is the $m$-th row of
$A(\mathrm{\mathbf{H}})^{l}$; $L_{0}$ is the number of layers of the
transformer model; $\mathrm{\mathbf{e}}=\\{1\\}^{n}$. The prior distribution
is contextual and we will show that the priors of both speech and text modules
narrow the modal semantic gap and contribute to better alignment results.

For global alignments, we obtain $\mathrm{\mathbf{S}}^{\prime}$ and
$\mathrm{\mathbf{T}}^{\prime}$ after applying the significance prior to the
output sequence of speech and text module:
$\mathrm{\mathbf{S}}^{\prime}=\mathrm{\mathbf{S}}\odot\mathcal{P}_{sig}(\mathrm{\mathbf{S}})$,
$\mathrm{\mathbf{T}}^{\prime}=\mathrm{\mathbf{T}}\odot\mathcal{P}_{sig}(\mathrm{\mathbf{T}})$.
For local alignments, according to Eq.2, we obtain the similarity sequence
$\mathrm{\mathbf{\Phi}}=[\mathrm{\mathbf{\varphi}}_{1},...,\mathrm{\mathbf{\varphi}}_{n}]$
regarding the text output $\mathrm{\mathbf{T}}$, and we obtain
$\mathrm{\mathbf{\Phi}}^{\prime}$ constrained by prior:
$\mathrm{\mathbf{\Phi}}^{\prime}=\mathrm{\mathbf{\Phi}}\odot\mathcal{P}_{sig}(\mathrm{\mathbf{\Phi}})$.
For PAD, we replace the original sequences with their prior constrained ones
in Eq.1 and Eq.2.

###  2.3 Anchor-based Adaptive Span Aggregation

In reality, a text token may correspond to many speech tokens since a word
consists of multiple phonemes. However, the local alignment defined in Sec.2.1
aligns each text token with the corresponding speech token, which is not
consistent in granularity. Such alignment is possible because the attention
representations are contextual - each speech embedding potentially represents
the contextualized phoneme. However, attention scores are still highly
localized for transformers [19, 20], meaning that the embeddings tend to only
represent their close neighbors. As a result, there still remains a
granularity gap between the representations of speech and text tokens, thus
directly aligning them is not optimal.

Algorithm 1 Anchor-based Adaptive Span Aggregation

1:Procedure AASA ($\mathrm{\mathbf{S}},\xi$) $\triangleright$ Input:
$\mathrm{\mathbf{S}}\in\mathbb{R}^{n\times d_{s}}$

2:Anchor points $\mathrm{\mathbf{\Gamma}}\leftarrow[id_{e0}]\,$; span pools
$\tilde{\mathrm{\mathbf{S}}}\leftarrow[\,\ ]\,$.

3:$P_{sig}^{\prime}(\mathrm{\mathbf{S}})$, $Index_{sorted}$ $\leftarrow$
Descending Sort $P_{sig}(\mathrm{\mathbf{S}})$.

4:$\triangleright$ Select anchor pts by their significance scores (Eq.3).

5:for $id$ in $Index_{sorted}$ do

6: if $\forall id_{e}\in\mathrm{\mathbf{\Gamma}},|id-id_{e}|>\xi/2$ then
$\triangleright$ Control density.

7: $span_{m}\leftarrow\\{\mathrm{\mathbf{s}}_{id-
s_{m}},...,\mathrm{\mathbf{s}}_{id+s_{m}}\\}$ for $m\in{1,...,c}$

8:$\triangleright$ $s_{m}\in\\{\xi/2,\xi,2\xi,...\\}$; $\xi$ denotes the base
scale.

9:
$\mathrm{\mathbf{\tilde{s}}}_{id}\leftarrow\\{(pooling(span_{1}),...,pooling(span_{c})\\}$

10: Add $id$ to $\mathrm{\mathbf{\Gamma}}$; add
$\mathrm{\mathbf{\tilde{s}}}_{id}$ to $\mathrm{\mathbf{\tilde{S}}}$.

11:Return $\mathrm{\mathbf{\tilde{S}}}$ $\triangleright$ Output:
$\tilde{S}\in\mathbb{R}^{k\times c\times d_{s}}$

We propose to adaptively generate local spans where each span is aggregated by
several speech embeddings to narrow the granularity gap. The alignment process
is the same as Eq.2, only being different in that the original speech sequence
is replaced with the local span pools generated by the Anchor-based Adaptive
Span Aggregation (AASA) procedure (Algorithm 1). We denote the local span
pools of a speech sequence $\mathrm{\mathbf{S}}$ as
$\mathrm{\mathbf{\tilde{S}}}=[\mathrm{\mathbf{\tilde{S}}}_{1},...,\mathrm{\mathbf{\tilde{S}}}_{k}]$,
where $\mathrm{\mathbf{\tilde{S}}}_{m}\in\mathbb{R}^{c\times d_{s}}$. Each
span pool $\mathrm{\mathbf{\tilde{S}}}_{m}$ consists of $c$ spans of different
scales, and the absolute locations of the span pools are decided by the anchor
points. In line with local-level alignment (Eq.2), each text token aligns with
one of the spans $\mathrm{\mathbf{\tilde{s}}}_{m}$ in
$\mathrm{\mathbf{\tilde{S}}}$, maximizing the sum of the maximum text-span
similarities regarding all text tokens.

##  3 Results and Analysis

#### Experimental setup.

We use wav2vec2-base and bert-base-uncased checkpoints from the Hugging
Face§§§https://huggingface.co as the speech and text modules. We choose 768
dimensions for both models and cut down the BERT vocabulary size to 5000.
Following [18], the alignment training uses a 10-hour transcripted data
randomly selected from the train-clean-360 subset of Librispeech 960 [21]
dataset. For the evaluation of all downstream tasks, we follow the SUPERB [10]
benchmark using S3PRL¶¶¶https://github.com/s3prl/s3prl to achieve fair
comparison. We evaluate three SLU tasks: Intent Detection (IC), Emotion
Recognition (ER), and Slot Filling (SF) on Fluent Speech Commands, IEMOCAP,
and Audio SNIPS datasets, respectively.

SUPERB freezes the upstream pre-trained model and controls all other settings
to be identical. [18] finetune the whole upstream model, and they compare with
baselines without specifying the settings to be identical. Thus, their
alignment effectiveness is not easily comparable and our results based on
SUPERB are fairer and more reproducible for future work. We re-implement their
alignment methods for comparison.

Table 1: Results on IC, ER, and SF in terms of accuracy and F1 score. Our PAD outperforms all the alignment baselines. Models |  Align Variant |  | Intent
---
Classification
| Emotion
---
Recognition
| Slot
---
Filling
Acc $\uparrow$ |  Acc $\uparrow$ |  F1 $\uparrow$
wav2vec 2.0 Base [7] | - | 94.40 | 63.02 | 87.94
Glob - cls [18] | G | 97.39 | 64.19 | 86.30
TLocal - idf [18] | T | 94.91 | 58.49 | 82.04
Glob - avr | G | 97.30 | 64.65 | 86.00
TLocal | T | 95.90 | 59.18 | 82.27
TLocal - CW | T | 96.81 | 60.27 | 84.88
TLocal - CG | T | 96.76 | 61.97 | 84.89
TLocal - OR | T | 90.30 | 59.32 | 77.36
SLocal - OR | S | 97.57 | 63.87 | 85.12
PAD-Glob | G | 97.68 | 64.91 | 87.67
PAD-TLocal | T | 95.90 | 62.46 | 88.40
PAD-SLocal | S | 97.86 | 62.74 | 87.19

#### Results against various alignment methods.

We report the results of three levels of PAD against the baselines in Tab.1.
G, T, and S denote global, token-level local, and span-level local alignment,
respectively. For global-level alignment (Eq.1), we compare with 2 variants of
sentence embeddings: $cls$ embedding (Glob-cls) [18] and averaged pooling
(Glob-avr). For token-level local alignment, we compare with 5 variants:
vanilla token-level alignment (TLocal) as in Eq.1; token-level alignment using
idf to ignore the frequent words (TLocal-idf) [18]; first training a speech
token recognizer with CTC loss using the same 10-hour data, then the
prediction probabilities on text vocabulary are used to weight (TLocal-CW, by
constraining the alignment as ASP) or guide (TLocal-CG, achieve a speech-text
mapping by max probability, and align them according to the mapping instead of
the max similarity of Eq.2) the alignment; TLocal-OR treats the similarity
distribution of Eq.2 as the probability distribution in CTC calculation and
computes the alignment loss as a CTC loss to achieve alignments based on
maximum expectations. For span-level local alignment, we compare with the
span-level ordered align (SLocal-OR), which is the same as the TLocal-OR
except for its granularity.

The results in Tab.1 show that our three alignments outperform all baselines
in the downstream tasks. Whereas there exist performance trade-offs on
different tasks for global and local alignments. The models under global
alignments are better at classification tasks (Intent Classification and
Emotion Recognition), whereas the ones under local alignments are better at
sequence generation tasks (Slot Filling). Classification tasks require better
global-level semantics whereas sequence generation tasks require finer token-
level semantics. Compared with the token-level alignment, the span-level
alignment achieves good performance on both kinds of tasks, which illustrates
the effectiveness of AASP that narrows the gap between speech and text
granularities.

Table 2: Ablation study on IC, ER, and SF by removing the features one by one. Align Variants |  Align Variant |  | Intent
---
Classification
| Emotion
---
Recognition
| Slot
---
Filling
Acc $\uparrow$ |  Acc $\uparrow$ |  F1 $\uparrow$
PAD-Glob | G | 97.68 | 64.91 | 87.67
\- w/o s prior | G | 97.55 | 64.39 | 87.17
\- w/o t prior | G | 97.34 | 64.14 | 87.59
\- w/o both | G | 97.30 | 63.97 | 86.29
PAD-TLocal | T | 95.90 | 62.46 | 88.40
\- w/o prior | T | 95.90 | 59.18 | 82.27
PAD-SLocal | S | 97.86 | 62.74 | 87.19
\- w/o prior | S | 97.52 | 60.37 | 83.04
\- w/o span pool | S | 97.86 | 62.72 | 87.06
\- w/o anch pts | S | 96.73 | 60.88 | 86.10

#### Ablation Study.

We evaluate the impact of the various choices we made for PAD by ablating its
features (Tab.2). For PAD-Glob, we remove the speech and text priors one by
one and the consistent performance drop verifies their importance. Similarly,
we also verify the effectiveness of the significance priors for PAD-TLocal and
PAD-SLocal. Additionally, for PAD-SLocal, we find that either (a) removing the
span pools and using fixed spans or (b) using even-stride spans instead of
anchor-based ones decreases the performance.

#### Analysis of joint alignments.

We also evaluate all the joint alignment combinations that perform multiple
alignments concurrently (Tab.3). We find that all of them perform worse than
the combined results of separate alignments. Furthermore, we observe that the
performances of PAD-G&S and PAD-T&S $>$ PAD-all joint $>$ PAD-G&T, which is in
line with our assumptions: the global-level and token-level alignment may be
incompatible in essence because of their granularity gap, whereas the
granularity of both of them are close to the span-level alignment because of
the adaptive aggregation of AASA (Eq.1).

Table 3: Analysis of the joint alignment combinations. Align Variants |  Align Variant |  | Intent
---
Classification
| Emotion
---
Recognition
| Slot
---
Filling
Acc $\uparrow$ |  Acc $\uparrow$ |  F1 $\uparrow$
PAD-Glob | G | 97.68 | 64.91 | 87.67
PAD-TLocal | T | 95.90 | 62.46 | 88.40
PAD-SLocal | S | 97.86 | 62.74 | 87.19
PAD-G&T | G+T | 68.63 | 61.42 | 77.98
PAD-G&S | G+S | 97.86 | 62.93 | 87.25
PAD-T&S | T+S | 97.68 | 62.91 | 86.49
PAD-all joint | G+T+S | 97.50 | 62.40 | 85.47

#### Analysis of alignment loss.

Apart from the downstream task accuracies, the variation of alignment loss
before and after the alignment is a rough metric for comparing the alignment
effectiveness since they reflect the homogenization amount of embedding space.
With all entries being properly scaled, we plot the validation loss curves of
global and local alignment training from epoch 1 in Fig.2. Fig.2(a) shows that
the three alignments constrained by the prior show higher loss deduction
compared with the vanilla version, which is in line with the results obtained
above, illustrating the effectiveness of the speech and text priors. In
Fig.2(b), the local alignments that are constrained by the priors or leverage
the adaptive spans show higher loss deduction, which is also in line with the
effectiveness of the significance prior and adaptive span in local-level
alignment experiments.

Fig. 2: The global and local-level alignment loss on dev set. S and T denote
speech and text, respectively.

##  4 Conclusion

We propose a method that leverages adaptive spans and the in-stored
significance prior distribution to achieve better alignments between text and
speech pre-trained models at both global and local levels. We also observe
that the trade-off between global and local alignments always exists because
they may be contradictory in essence. We plan to investigate how to alleviate
this trade-off in future work.

## References

  * [1] Jacob Devlin et al.,  “BERT: pre-training of deep bidirectional transformers for language understanding,”  in Proceedings of the NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1. 2019, pp. 4171–4186, Association for Computational Linguistics.
  * [2] Zhenzhong Lan et al.,  “ALBERT: A lite BERT for self-supervised learning of language representations,”  in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. 2020, OpenReview.net.
  * [3] Yinhan Liu et al.,  “Roberta: A robustly optimized BERT pretraining approach,”  2019\.
  * [4] Colin Raffel et al.,  “Exploring the limits of transfer learning with a unified text-to-text transformer,”  J. Mach. Learn. Res., vol. 21, pp. 140:1–140:67, 2020.
  * [5] Alec et al. Radford,  “Language models are unsupervised multitask learners,”  2019\.
  * [6] Steffen Schneider et al.,  “wav2vec: Unsupervised pre-training for speech recognition,”  in Interspeech 2019, Graz, Austria, 15-19 September 2019, Gernot Kubin and Zdravko Kacic, Eds. 2019, pp. 3465–3469, ISCA.
  * [7] Alexei Baevski et al.,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [8] Wei-Ning Hsu et al.,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE ACM Trans. Audio Speech Lang. Process., vol. 29, pp. 3451–3460, 2021.
  * [9] Mauajama Firdaus, Asif Ekbal, and Erik Cambria,  “Multitask learning for multilingual intent detection and slot filling in dialogue systems,”  Information Fusion, vol. 91, pp. 299–315, 2023.
  * [10] Shu-Wen Yang et al.,  “SUPERB: speech processing universal performance benchmark,”  in Interspeech 2021, Brno, Czechia, 30 August - 3 September 2021, Hynek Hermansky, Honza Cernocký, Lukás Burget, Lori Lamel, Odette Scharenborg, and Petr Motlícek, Eds. 2021, pp. 1194–1198, ISCA.
  * [11] Junyi Ao et al.,  “Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing,”  in ACL 2022, Dublin, Ireland, May 22-27, 2022, Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, Eds. 2022, pp. 5723–5738, Association for Computational Linguistics.
  * [12] Sachidananda et al.,  “Calm: Contrastive aligned audio-language multirate and multimodal representations,”  arXiv preprint arXiv:2202.03587, 2022.
  * [13] Ankur et al. Bapna,  “Slam: A unified encoder for speech and language modeling via speech-text joint pre-training,”  arXiv preprint arXiv:2110.10329, 2021.
  * [14] Ankur et al. Bapna,  “mslam: Massively multilingual joint pre-training for speech and text,”  arXiv preprint arXiv:2202.01374, 2022.
  * [15] Bhuvan et al. Agrawal,  “Tie your embeddings down: Cross-modal latent spaces for end-to-end spoken language understanding,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7157–7161.
  * [16] Alexander et al. Liu,  “Cross-modal discrete representation learning,”  arXiv preprint arXiv:2106.05438, 2021.
  * [17] Pavel Denisov and Ngoc Thang Vu,  “Pretrained semantic speech embeddings for end-to-end spoken language understanding via cross-modal teacher-student learning,”  in Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020. 2020, pp. 881–885, ISCA.
  * [18] Yu-An Chung et al.,  “SPLAT: speech-language joint pre-training for spoken language understanding,”  in NAACL-HLT 2021, Online, June 6-11, 2021. 2021, pp. 1897–1907, Association for Computational Linguistics.
  * [19] Liunian Harold Li et al.,  “What does BERT with vision look at?,”  in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, Eds. 2020, pp. 5265–5275, Association for Computational Linguistics.
  * [20] Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han,  “Lite transformer with long-short range attention,”  in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. 2020, OpenReview.net.
  * [21] Vassil Panayotov et al.,  “Librispeech: An ASR corpus based on public domain audio books,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015. 2015, pp. 5206–5210, IEEE.