# Lineax: unified linear solves and linear least-squares in JAX and Equinox

Jason Rader
Oxford University
<EMAIL_ADDRESS>
Terry Lyons
Oxford University
Patrick Kidger
Google X
<EMAIL_ADDRESS>

###### Abstract

We introduce Lineax, a library bringing linear solves and linear least-squares
to the JAX+Equinox scientific computing ecosystem. Lineax uses general linear
operators, and unifies linear solves and least-squares into a single,
autodifferentiable API. Solvers and operators are user-extensible, without
requiring the user to implement any custom derivative rules to get
differentiability. Lineax is available at https://github.com/google/lineax.

##  1 Introduction

JAX is an autodifferentiatiable Python framework popular for machine learning
and scientific computing [4, 9, 12, 16]. Equinox [20] is a popular JAX library
[8, 15], targeting the same use cases, that adds additional support for
parameterised functions. Solving linear systems, whether well-posed linear
solves or ill-posed linear least-squares problems, is a central sub-problem in
scientific computing [14, 27]. For example, linear solves and least-squares
appear as subroutines in nonlinear optimisation [21], finite-difference
schemes [26], and signal processing [22]. As such, we introduce Lineax, a
library built in JAX and Equinox for linear solves and linear least-squares.

Lineax presents a single, differentiable interface for solving well-posed,
underdetermined, and overdetermined linear systems. It also allows users to
write custom differentiable linear solvers or least-squares solvers, and
introduces a linear operator abstraction.

Overall, we intend for Lineax to integrate well with the existing JAX
scientific ecosystem. This ecosystem is growing, and includes packages for
differentiable rigid-body physics simulation [11], computational fluid
dynamics [3, 6], protein structure prediction [10], ordinary and stochastic
differential equations [19], and probabilistic modeling [25]. We are beginning
to see some use of Lineax in this ecosystem already. This includes for linear
subroutines in ocean dynamics [18] and optimal transport [5]. Further, Diffrax
[19] plans to adopt Lineax in the near future for linear subroutines in
differential equations solves.

###  1.1 Main contributions

The main contributions of Lineax are:

  * •

A general linear operator abstraction, as implemented by dense matrices,
linear functions, Jacobians, etc.

  * •

Stable and fast gradients through least-squares solves. This includes through
user-defined solvers, without requiring extra effort from the user.

  * •

PyTree-valued 111JAX terminology for arbitrarily nested container ’node’ types
(tuples/dictionaries/lists/custom types) containing ’leaves’ (every other
Python/JAX type.) We exclusively consider PyTrees who’s leaves are JAX arrays.
operators and vectors.

Comparisons to existing JAX APIs

The operator abstraction introduced in Lineax offers a flexibility not found
in core JAX, which supports only dense matrices or matrix-vector product
representations of operators. Lineax introduces new solvers over core JAX,
such as `lineax.Tridiagonal`. Lineax also offers a consistent API between
operators and solvers, which is what allows for extensibility to user-
specified custom operator and solvers.

Compilation times for most Lineax solvers are essentially identical to JAX
native solvers; Lineax’s iterative solvers (CG, GMRES, …) compile roughly
twice as fast. The ‘benchmarks’ folder on GitHub provides a quantitative
comparison.

We emphasise the stable and fast gradients to contrast with the existing JAX
implementation, which as of version 0.4.16 exhibits instability or incorrect
gradients in some exceptional cases.

For these reasons, JAX is actually considering deprecating some of its own
APIs in favour of Lineax [29].

###  1.2 Classical linear solve example

Consider solving $Ax=b$ for a random matrix $A\in\mathbb{R}^{10\times 10}$
against a random vector $b\in\mathbb{R}^{10}$. This can be done via

⬇

1 import jax.random as jr

2 import lineax as lx

3

4 A_key, b_key = jr.split(jr.PRNGKey(0))

5 A = lx.MatrixLinearOperator(jr.normal(A_key, (10, 10)))

6 b = jr.normal(b_key, (10,))

7 solution = lx.linear_solve(A, b)

##  2 Performing linear solves and least-squares

The main entry point to linear solves and least-squares in Lineax is

⬇

lineax.linear_solve(A, b, solver)

for a linear operator $A$ and PyTree $b$. This performs a linear solve $Ax=b$
(for well-posed systems), or returns a least-squares solution
$\min_{x}\|Ax-b\|_{2}$ (for overdetermined systems), or returns a minimum norm
solution $\min_{x}\|x\|_{2}$ subject to $Ax=b$ (for underdetermined systems).
This is a lot of operations to unify together, and it may initially seem
strange to do so. The common thread – and our justification for unifying these
operations – is that mathematically, all the above operations correspond to
the pseudoinverse solution to $Ax=b$, ie. the solution arising from using the
Moore-Penrose pseudoinverse $x=A^{\dagger}b$ [1, 23].

The user can specify which solver they’d like to use via the `solver`
argument. This is helpful when the user already knows which solvers should
work well for a problem. Not every solver is capable of handling every
problem. For example, `lineax.CG` handles positive definite operators [21,
section 5]. Using a solver with an incompatible problem will result in an
error.

##  3 General linear operators

In Lineax, we represent $A$ more generally than as an $n\times m$ matrix.
Instead, we represent $A$ as a linear operator $A:X\to Y$, where $X$ and $Y$
are spaces of PyTrees of arrays. At an implementation level, a linear operator
is an object which subclasses

⬇

lineax.AbstractLinearOperator.

When $A$ is a dense matrix, $A\in\mathbb{R}^{\dim(Y)\times\dim(X)}$, it can be
treated as a Lineax linear operator via

⬇

lineax.MatrixLinearOperator(A).

Lineax operators themselves form a vector space, and are closed under
addition, scalar multiplication, and composition. Each linear operator $A$
must implement a method to:

  * •

Compute the matrix-vector product: $Ax$ for $x\in X$.

  * •

Compute the transpose of the operator: $A^{T}:Y\to X$.

  * •

Materialise the operator as a matrix:
$A\text{.as\\_matrix()}\in\mathbb{R}^{\dim(X)\times\dim(Y)}$.

  * •

Retrieve the input/output PyTree structure, as well as the input/output
dimensions. ie. the functions $\text{domain}(A)=X$ and $\text{codomain}(A)=Y$.

This increased generality comes with increased flexibility. For example:
large, sparse matrices can use data-efficient formats and utilise linear
solves which use only the matrix-vector product, such as GMRES [24] or
BiCGStab [28]. For example, a linear function $f:X\to Y$ can be made into a
linear operator with

⬇

lineax.FunctionLinearOperator(f, in_structure)

where `in_structure` describes the PyTree structure of the input of $f$
(equivalently, the PyTree structure of the elements $x\in X$.) Similarly, a
nonlinear function $g:X\to Y$ can be linearised at a point $x\in X$ and use
its Jacobian at $x$ as a linear operator via

⬇

lineax.JacobianLinearOperator(g, x)

The `lineax.AbstractLinearOperator` base class is available for users to
subclass and create their own linear operator types.

###  3.1 Operator tags

Tags are an optional argument to most linear operators, and indicate
properties of the operator $A$. For example, if $A\in\mathbb{R}^{n\times n}$
is positive semidefinite, then $A$ can be marked as a positive semidefinite
linear operator with

⬇

lineax.MatrixLinearOperator(A, lineax.positive_semidefinite_tag)

This indicates to any solver which uses $A$ that it is positive semidefinite.
For example, if $A$ is also nonsingular, then it can be used safely with
`lineax.CG`. Tags are also used to select the appropriate solver in the
polyalgorithm `lineax.AutoLinearSolver` detailed in section 6.

##  4 Computing gradients

In JAX, derivatives are built from Jacobian-vector products (JVPs) and vector-
Jacobian products (VJPs) for forward-mode and reverse-mode automatic
differentiation respectively [12]. The JVP of a function
$f:\mathbb{R}^{a}\to\mathbb{R}^{b}$ maps an input-tangent pair
$(x,v)\in\mathbb{R}^{a}\times\mathbb{R}^{a}$ to $(f(x),\partial
f(x)(v))\in\mathbb{R}^{b}\times\mathbb{R}^{b}$ where $\partial
f(x):\mathbb{R}^{a}\to\mathbb{R}^{b}$ is the Jacobian of $f$ at $x$. The VJP
maps an input-cotangent pair $(x,c)\in\mathbb{R}^{a}\times\mathbb{R}^{b}$ to
$(f(x),\partial f(x)^{T}c)\in\mathbb{R}^{b}\times\mathbb{R}^{a}$, where
$\partial f(x)^{T}:\mathbb{R}^{b}\to\mathbb{R}^{a}$ is the transpose of
$\partial f(x)$.

The major contribution of Lineax over existing linear solve and least-squares
software is the efficient computation of JVPs for pseudoinverse solutions.
That is, differentiation through both well-posed linear solves and ill-posed
least-squares solves are performed in the same manner as each other. In
particular, we may special case when operators have full row or column rank in
order to obtain improved performance, as we now show.

###  4.1 JVPs and forward-mode autodifferentiation

In this section, let $\mathcal{L}(A,b)$ denote the linear solve
`lineax.linear_solve`. For a primal problem $Ax=b$, then
$\mathcal{L}(A,b)=A^{\dagger}b$ where $A^{\dagger}$ be the Moore–Penrose
pseudoinverse of $A$, as mentioned in section 2. Here we discuss how the to
compute the JVP $\partial\mathcal{L}(A,b)(V,v)$, where $(V,v)$ is the tangent
pair consisting of a tangent operator $V$ and tangent vector $v$.

It is possible to compute the JVP through either argument. For example, the
tangent computation of a linear solve as a function of $v$ alone is

$\partial\mathcal{L}(A,b)(0,v)=A^{\dagger}v$

where $0$ represents the $0$ tangent operator.

Meanwhile, computing the JVP for $\partial\mathcal{L}(A,b)(V,0)$ requires
differentiating through a pseudoinverse, which has the explicit formula [13]

$\partial\mathcal{L}(A,b)(V,0)=(-A^{\dagger}VA^{\dagger}+A^{\dagger}(A^{\dagger})^{T}V^{T}(I-AA^{\dagger})+(I-A^{\dagger}A)V^{T}(A^{\dagger})^{T}A^{\dagger})A^{\dagger}b.$

Letting

$\displaystyle x$ $\displaystyle=A^{\dagger}b$ $\displaystyle z$
$\displaystyle=V^{T}(A^{\dagger})^{T}x,$

and adding the above two equations together and using the linearity of the
Jacobian, we have the total JVP with respect the primal pair $(A,b)$ and
tangent pair $(V,v)$ for `lineax.linear_solve` is

$\partial\mathcal{L}(A,b)(V,v)=A^{\dagger}\left(-Vx+(A^{\dagger})^{T}V^{T}(b-Ax)-Az+v\right)+z.$
(1)

If $A$ has linearly independent columns, then $A^{\dagger}A=I$ [14, section
5.5.2] and the term $z-A^{\dagger}Az=0$, giving

$\partial\mathcal{L}(A,b)(V,v)=A^{\dagger}\left(-Vx+(A^{\dagger})^{T}V^{T}(b-Ax)+v\right).$
(2)

When $A$ has linearly independent rows, then $(b-Ax)=0$ and

$\partial\mathcal{L}(A,b)(V,v)=A^{\dagger}\left(-Vx-Az+v\right)+z.$ (3)

Together, if $A$ has linearly independent rows and columns, then $A$ is well-
posed, $A^{\dagger}=A^{-1}$ is a true inverse, and

$\partial\mathcal{L}(A,b)(V,v)=A^{-1}\left(-Vx+v\right).$ (4)

We then select between equations (4), (3), (2), or (1) depending on whether we
know at compile time that $A$ has linearly independent rows and columns, has
only independent rows, has only independent columns, or has both dependent
rows and columns. Despite being a property of the operator, at compile time
the main way the JVP rule is dispatched via the choice of solver. This is
because not every solver supports dependent rows/columns (section 5), and will
return nan values if used in a solve with an unsupported operator. So, if a
solver does not support dependent rows/columns, we can be sure we will not get
a solution given an operator with dependent rows/columns in the JVP.

For example, `lineax.QR` [27, section 2] can handle dependent rows if the
number of rows is greater than the number of columns (or dependent columns if
the number of columns is greater than the number of rows) and will dispatch to
either equation (2) or (3). If the number of columns and rows of $A$ are the
same, then `lineax.QR` will dispatch to equation (4).

###  4.2 VJPs and backpropogation via transposition

Reverse-mode autodifferentiation of a function
$f:\mathbb{R}^{a}\to\mathbb{R}^{b}$ is not built on JVPs, but rather on
vector-Jacobian products (VJPs) $v^{T}\partial f(x)=\partial f(x)^{T}v$. As
suggested by the definition, VJPs are constructed via a JVP and transposition
[12]. This is how VJPs are implemented in JAX, and thus in Lineax as well. The
Jacobian $\partial\mathcal{L}(A,b)\colon\mathbb{R}^{m\times
n}\times\mathbb{R}^{m}\to\mathbb{R}^{n}$ is a linear function, see also the
explicit form in equation (1). Therefore, it has a transpose
$\partial\mathcal{L}(A,b)^{T}\colon\mathbb{R}^{n}\to\mathbb{R}^{m\times
n}\times\mathbb{R}^{m}$.

The transpose rule for the linear solve is implemented as a custom JAX
primitive. See [12] for more details.

##  5 User-defined solvers

A user can implement a custom solver by subclassing

⬇

lineax.AbstractLinearSolver

which requires the methods: `init`, `compute`, `transpose`,
`allow_dependent_rows`, and `allow_dependent_columns`.

Many direct linear solvers for $Ax=b$ use two stages of computation. First,
factor $A$ into a form amenable to computation (eg. LU factorisation [14,
section 3], QR factorisation, SVD factorisation, etc.) Then, use this
factorisation to solve for a given right hand side $b$. The factorisation of
$A$ does not depend on the right hand side $b$, and can be reused with various
choices of $b$. This saves computation cost when solving $Ax=b$ for many right
hands $b$.

For a solver using such a two-stage factorisation approach, `init` computes
the factorisation, and `compute` performs the solve for the specific right
hand $b$. `transpose` computes the transpose of the factorisation provided by
`init`, which allows us to skip computing the factorisation of the transpose
operator directly (`init(transpose(operator))`), as it is commonly the case
that we can cheaply derive this from `init(operator)` alone. This is needed
when computing VJPs as discussed in the previous section. The methods
`allow_dependent_rows` and `allow_dependent_columns` determine which equation
(1-4) is used in the differentiation, as discussed in section 4.

This greatly simplifies the process of writing a differentiable linear solver
or least-squares solver. In the core JAX library, it is somewhat cumbersome to
write a differentiable solver. It requires using `jax.lax.custom_linear_solve`
and implementing a solver transposition rule `transpose_solve` if the user
would like to use reverse-mode autodifferentiation. In Lineax, differentiation
comes for free once a solver is implemented, whether the solver is a linear
solve or a least-squares algorithm.

##  6 The AutoLinearSolver polyalgorithm

If the user does not provide a solver to `lineax.linear_solve`, then the
default linear solve is `lineax.AutoLinearSolver`. `lineax.AutoLinearSolver`
is a polyalgorithm which selects a solver automatically at compile time
depending on the structure of $A$, as indicated through its operator tag
(discussed in section 3.1). `lineax.AutoLinearSolver` takes the argument
`well_posed`, which indicates whether the system is expected to solve a least-
squares/minimum norm problem, or only handle well-posed linear solves.

`lineax.AutoLinearSolver(well_posed=True)` selects a solver depending upon the
operator structure, and throws an error when it encounters an underdeteremined
or overdetermined system. `lineax.AutoLinearSolver(well_posed=False)` solves
well-posed linear solves as well as linear least squares, but often at an
additional computation cost. Finally,
`lineax.AutoLinearSolver(well_posed=None)` solves a least-squares problem only
if it is not expensive to do so.

The specific polyalgorithms for well_posed=True, well_posed=False, and
well_posed=None are shown in figure 1.

Figure 1: The AutoLinearSolver polyalgorithm. Read from left-to-right, so that
well_posed=True starts at "$A$ is square?", well_posed=None starts at "$A$ is
diagonal?", and well_posed=False starts at "$A$ is diagonal?" in the
well_posed=False section.

###  6.1 Choosing a solver at compile time

We must choose between two paradigms for the implementation of
`lineax.AutoLinearSolver`: make the algorithm selection at run time, or make
the algorithm selection at compile time. This is a trade-off, as determining
which solver to use at run time means checking the elements of the matrix.
This incurs a run time overhead of $\mathcal{O}(n^{2})$ for an $n\times n$
matrix, which is relatively small compared to the $\mathcal{O}(n^{3})$ run
time of most linear solve algorithms. However, run time checking also incurs a
greater cost in compile times. Since it is not known at compile time which
branch of the polyalgorithm will run, the compiler is forced to compile all
branches. Thus, compilation cost scales with the logic of the polyalgorithm:
as more branches are included, compile times increase. This can limit the
extensibility of the polyalgorithm.

Compile time selection avoids these performance issues, and is faster both in
run time and compile time when used correctly. Further, it simplifies tying
solves to GPU hardware, as there is no possibility of taking separate branches
for different batch elements. However, compile time selection requires the
user to a-priori know the structure of the operator, and can result in using a
suboptimal solver if the operator has exploitable structure which the user
does not indicate.

We choose the compile time approach, and require the user to pass the
structure of an operator explicitly via the operator tag (section 3.1.) We
choose this approach primarily to minimise compilation times, and to avoid the
tradeoff between extensibility and compile time inherent in run time checking.

Our approach is in contrast to MATLAB’s `mldivide`, a (nondifferentiable)
unified linear solve and least-squares solver which uses a run time approach
[17]. Both the Julia and MATLAB languages offer methods for nonsingular linear
solves – the infix `\` operation in Julia and `linsolve` in MATLAB – which
accept compile time tags, but still perform run time checks if the user passes
no tags [2, 7, 17]. Therefore, both suffer from the additional overhead of the
run time approach in many cases.

##  7 Conclusion

We have introduced Lineax, a differentiable JAX+Equinox library unifying
linear solves and linear least-squares. We have demonstrated that users can
extend base Lineax operators and solvers and use them within our unified API,
without the need to write any custom derivative rules. We hope to see adoption
of Lineax solves within the JAX+Equinox scientific computing and machine
learning ecosystem.

##  8 Acknowledgements

This publication is based on work supported by the EPSRC Centre for Doctoral
Training in Mathematics of Random Systems: Analysis, Modelling and Simulation
(EP/S023925/1)

## References

  * [1] Adi Ben-Israel and Thomas N. E. Greville.  Generalized Inverses: Theory and Applications.  Springer New York, 2003.
  * [2] Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah.  Julia: A fresh approach to numerical computing.  SIAM Review, 59(1):65–98, 2017.  URL: https://epubs.siam.org/doi/10.1137/141000671, doi:10.1137/141000671.
  * [3] Deniz A. Bezgin, Aaron B. Buhendwa, and Nikolaus A. Adams.  JAX-fluids: A fully-differentiable high-order computational fluid dynamics solver for compressible two-phase flows.  Computer Physics Communications, 282:108527, 2023.
  * [4] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James, Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  Jax: composable transformations of python+numpy programs, 2018.  URL: http://github.com/google/jax.
  * [5] Marco Cuturi, Laetitia Meng-Papaxanthos, Yingtao Tian, Charlotte Bunne, Geoff Davis, and Olivier Teboul.  Optimal transport tools (ott): A jax toolbox for all things wasserstein.  arXiv preprint arXiv:2201.12324, 2022.
  * [6] Gideon Dresdner, Dmitrii Kochkov, Peter Norgaard, Leonardo Zepeda-Núñez, Jamie A. Smith, Michael P. Brenner, and Stephan Hoyer.  Learning to correct spectral methods for simulating turbulent flows.  arXiv preprint arXiv:2207.00556, 2022.
  * [7] Chris Rackauckas et al.  Linearsolve.jl: High-performance unified linear solvers, 2021.  Accessed 2023.  URL: https://github.com/SciML/LinearSolve.jl.
  * [8] David Hall et al.  Haliax.  Accessed 2023, 2023.  URL: https://github.com/stanford-crfm/haliax.
  * [9] Igor Babuschkin et al.  The deepmind jax ecosystem, 2020.  URL: http://github.com/deepmind.
  * [10] John Jumper et al.  Highly accurate protein structure prediction with alphafold.  Nature, 596:583–589, 8 2021.
  * [11] C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem.  Brax - a differentiable physics engine for large scale rigid body simulation, 2021.
  * [12] Roy Frostig, Matthew J. Johnson, Dougal Maclaurin, Adam Paszke, and Alexey Radul.  Decomposing reverse-mode automatic differentiation, 2021.  arXiv:2105.09469.
  * [13] Gene H. Golub and V. Pereyra.  The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate.  SIAM Journal on Numerical Analysis, 10(2):413–432, 1973.
  * [14] Gene H. Golub and Charles F. Van Loan.  Matrix Computations.  The Johns Hopkins University Press, third edition, 1996.
  * [15] David Hall, Ivan Zhou, and Percy Liang.  Levanter — legible, scalable, reproducible foundation models with jax.  Accessed 2023, 2023.  URL: https://github.com/stanford-crfm/levanter.
  * [16] Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee.  Flax: A neural network library and ecosystem for JAX, 2023.  URL: http://github.com/google/flax.
  * [17] The MathWorks Inc.  Matlab version: 9.13.0 (r2022b), 1984.  URL: https://www.mathworks.com.
  * [18] J. Emmanuel Johnson and Takaya Uchida.  Jaxsw: Jax approximate ocean models, 2023.  Accessed 2023.  URL: https://github.com/jejjohnson/jaxsw.
  * [19] Patrick Kidger.  On Neural Differential Equations.  PhD thesis, University of Oxford, 2021.
  * [20] Patrick Kidger and Cristian Garcia.  Equinox: neural networks in JAX via callable PyTrees and filtered transformations.  Differentiable Programming workshop at Neural Information Processing Systems 2021, 2021.
  * [21] Jorge Nocedal and Stephen Wright.  Numerical Optimization (Second Edition).  Springer New York, 2006.
  * [22] Alan V. Oppenheim and Ronald W. Schafer.  Digital Signal Processing.  Prentice-Hall, 1975.
  * [23] R. Penrose.  A generalized inverse for matrices.  Mathematical Proceedings of the Cambridge Philosophical Society, 51(3):406–413, 1955.
  * [24] Youcef Saad and Martin H. Schultz.  Gmres: A generalized minimal residual algorithm for solving nonsymmetric linear systems.  SIAM Journal on Scientific and Statistical Computing, 7(3):856–869, 1986.
  * [25] Miloš Stanojević and Laurent Sartran.  SynJax: Structured Probability Distributions for JAX.  arXiv preprint arXiv:2308.03291, 2023.
  * [26] J. W. Thomas.  Numerical Partial Differential Equations: Finite Difference Methods.  Springer New York, 1995.
  * [27] Lloyd N. Trefethen and David Bau.  Numerical Linear Algebra.  Society for Industrial and Applied Mathematics, 1997.
  * [28] H. A. van der Vorst.  Bi-cgstab: A fast and smoothly converging variant of bi-cg for the solution of nonsymmetric linear systems.  SIAM Journal on Scientific and Statistical Computing, 13(2):631–644, 1992.
  * [29] Jake Vanderplas.  Jep 18137: Scope of jax numpy & scipy wrappers.  Github pull request, created and accessed 2023, 2023.  URL: https://github.com/google/jax/pull/18137.


# Simplifying Content-Based Neural News Recommendation:
On User Modeling and Training Objectives

Andreea Iana  University of MannheimMannheimGermany andreea.iana@uni-
mannheim.de ,  Goran Glavaš  CAIDAS, University of WürzburgWürzburgGermany
<EMAIL_ADDRESS>and  Heiko Paulheim  University of
MannheimMannheimGermany<EMAIL_ADDRESS>
(2023)

###### Abstract.

The advent of personalized news recommendation has given rise to increasingly
complex recommender architectures. Most neural news recommenders rely on user
click behavior and typically introduce dedicated user encoders that aggregate
the content of clicked news into user embeddings (early fusion). These models
are predominantly trained with standard point-wise classification objectives.
The existing body of work exhibits two main shortcomings: (1) despite general
design homogeneity, direct comparisons between models are hindered by varying
evaluation datasets and protocols; (2) it leaves alternative model designs and
training objectives vastly unexplored. In this work, we present a unified
framework for news recommendation, allowing for a systematic and fair
comparison of news recommenders across several crucial design dimensions: (i)
candidate-awareness in user modeling, (ii) click behavior fusion, and (iii)
training objectives. Our findings challenge the status quo in neural news
recommendation. We show that replacing sizable user encoders with parameter-
efficient dot products between candidate and clicked news embeddings (late
fusion) often yields substantial performance gains. Moreover, our results
render contrastive training a viable alternative to point-wise classification
objectives.

neural news recommendation, user modeling, late fusion, training objectives,
contrastive learning, evaluation

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: ; July 23–27, 2023; Taipei, Taiwan††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information systems Recommender
systems

##  1\. Introduction

In recent years, content-based news recommendation has seen increasingly
complex neural recommender architectures that aim to customize suggestions to
users’ interests (Li and Wang, 2019; Wu et al., 2023). Most neural news
recommendation (NNR) models commonly comprise (i) a dedicated news encoder
(NE) and (ii) a user encoder (UE) (Wu et al., 2021, 2023). NEs – instantiated
as a convolutional network (Wang et al., 2018; Wu et al., 2019b, a), self-
attention network (Wu et al., 2019d, 2020b; Qi et al., 2021b), graph attention
network (Qi et al., 2021a), or, most recently, as a pretrained transformer
network (Wu et al., 2021; Yu et al., 2022) – convert input features (e.g.
titles, categories, entities) into the news embedding. UEs aggregate
embeddings of clicked news into a user-level representation by means of
sequential (An et al., 2019; Qi et al., 2020; Wang et al., 2022) or attentive
(Wu et al., 2019b, d; Wang et al., 2018) encoders that contextualize
embeddings of clicked news based on patterns in clicking behavior (Okura et
al., 2017; An et al., 2019; Wu et al., 2022b). We dub this predominant
paradigm early fusion (EF) because it aggregates representations of clicked
news (i.e., builds user representation) before comparison with the
recommendation candidate.

Most NNR models encode users and candidate news separately, in a candidate-
agnostic manner (An et al., 2019; Wu et al., 2019b, d). Candidate-aware models
(Wang et al., 2020; Qi et al., 2021c; Zhang et al., 2021; Qi et al., 2022), in
contrast, acknowledge that not all clicked news are equally informative w.r.t.
the relevance of the candidate (e.g., a candidate is often representative of
only a subset of a user’s preferences), and contextualize representations of
clicked news with the embedding of the candidate in user-level aggregation
with UE. Finally, the candidate’s embedding (output of NE) is compared against
the user embedding (output of UE): the candidate’s recommendation score is
computed directly as the dot product of the two embeddings (Wu et al., 2019b)
or with a feed-forward scorer (Wang et al., 2018). NNR models are
predominantly trained via standard classification objectives (Wang et al.,
2018; Wu et al., 2019b, d, 2021) with negative sampling (Huang et al., 2013;
Wu et al., 2019c).

The existing body of work has two main shortcomings. First, despite general
design homogeneity, direct comparisons between recent NNRs are hindered by
lack of transparency and adoption of ad-hoc evaluation protocols (Iana et al.,
2022; Raza and Ding, 2022). In particular, a vast majority of personalized
news recommenders are evaluated on proprietary datasets (e.g. MSN News (Wu et
al., 2019b, d), Bing News (Wang et al., 2018), NewsApp (Qi et al., 2022)).
Even the few models evaluated using the publicly available datasets such as
Adressa (Gulla et al., 2017) or MIND (Wu et al., 2020a) cannot be directly
compared due to different dataset splits and evaluation protocols (e.g., model
selection strategy) (Zhang et al., 2021; Gong and Zhu, 2022; Wu et al., 2021;
Wang et al., 2022). Secondly, simpler and arguably more intuitive design
alternatives have largely been left unexplored. First, the existing work
adopts EF as default architecture, proposing increasingly complex user
encoding components (An et al., 2019; Qi et al., 2022), often with little
empirical justification for added complexity. Second, only a small fraction of
NNRs leverage contrastive learning objectives (Wu et al., 2022a; Yu et al.,
2022), despite such training criteria being proven highly effective in closely
related retrieval and recommendation tasks (Li et al., 2021; Wei et al., 2021;
Yang et al., 2022; Xie et al., 2022).

In this work, we remedy the above shortcomings of current NNRs and shed new
light on user modeling and training objectives.111Disclaimer: In this work we
focus exclusively on NNR models that do not resort to graph-based modeling of
relations between users. 1) Concretely, we introduce a unified framework for
neural news recommendation, facilitating systematic and fair comparison of NNR
models across three crucial design dimensions: (i) candidate-awareness in user
modeling, (ii) click behavior fusion, and (iii) training objectives. 2) We
propose to replace user modeling with complex user encoders (i.e., early
fusion) with simple pooling of dot-product scores between candidate and
clicked news embeddings (i.e. late fusion). We show that, despite conceptual
simplicity, LF brings substantial performance gains over EF-based NNR,
rendering complex UEs empirically unjustified. 3) Finally, we demonstrate the
benefits of supervised contrastive training as a viable alternative to point-
wise classification. Our work fundamentally challenges the status quo of NNR
by introducing simpler and more effective alternatives to the established
paradigm based on complex user modeling.

##  2\. Methodology

Figure 1 depicts our unified evaluation framework for NNR, focusing on three
critical dimensions of news recommendation. Given input data, comprising news
and user behaviors, we analyze (i) candidate-agnostic (C-AG) vs. candidate-
aware (C-AW) user modeling under (ii) two click behavior fusion strategies,
namely EF and LF, where each model can be (iii) trained by minimizing either
the standard cross-entropy loss (CE) or a supervised contrastive objective
(SCL). Next, we describe the models selected for evaluation and formalize the
concrete design choices.

###  2.1. User Modeling

Candidate-Agnostic (C-AG) Models. For these models, the UE produces the user
embedding from embeddings of clicked news without contextualization against
the candidate. We evaluate the following C-AG models, mutually differing in
their NE component (i.e., how they embed the clicked news): (1) NPA (Wu et
al., 2019c) uses a personalized attention module to aggregate the
representations of the users’ clicked news, with projected embeddings of the
users IDs as attention queries; (2) NAML (Wu et al., 2019b) uses additive
attention (Bahdanau et al., 2014) to encode users’ preferences; (3) NRMS (Wu
et al., 2019d) learns user representations with a two-layer encoder that
consists of multi-head self-attention (Vaswani et al., 2017) and additive
attention; (4) LSTUR (Wu et al., 2020a) learns user representations with
recurrent networks: a short-term user embedding is produced from the clicked
news with a GRU (Cho et al., 2014), and combined with a long-term embedding,
consisting of a randomly initialized and fine-tuned part; the final user
embedding is then obtained either (i) as the final hidden state of the short-
term GRU, initialized with the long-term embedding (LSTURini), or (ii) by
simply concatenating the short- and long-term user embeddings (LSTURcon); (5)
CenNewsRec (Qi et al., 2020) adopts a similar UE architecture as LSTUR, but
learns long-term user vectors from clicked news using a sequence of multi-head
self-attention and attentive pooling networks, as opposed to storing an
explicit embedding per user; (6) MINS (Wang et al., 2022) encodes users
through a combination of multi-head self-attention, multi-channel GRU-based
recurrent network, and additive attention.

Figure 1. Illustration of the unified NNR framework, focusing on three crucial
design dimensions: (i) candidate-awareness in user modeling (green box), (ii)
click behavior fusion (orange box), and (iii) training objectives (purple
box).

Candidate-Aware (C-AW) Models. UEs in candidate-agnostic models produce the
same user embedding, regardless of the content of the candidate news. In
contrast, UEs of candidate-aware models, two of which we include in our
empirical analysis, produce user embeddings dependent on the candidate. (7)
DKN (Wang et al., 2018) computes candidate-aware representations of users as
the weighted sum of their clicked news embeddings, with weights being produced
by an attention network that takes as input the embeddings of the candidate
and of the clicked news, as produced by the NE. More recently, (8) CAUM (Qi et
al., 2022) combines (i) a candidate-aware self-attention network to model
long-range dependencies between clicked news, conditioned on the candidate,
and (ii) a candidate-aware convolutional network (CNN) to capture short-term
user interests from adjacent clicks, again conditioned with the candidate’s
content; the candidate-aware user embedding is finally obtained by attending
over the long-range and short-term representations.

News Encoders. The NNR models included in our evaluation primarily use news
titles as input, which they typically embed via pretrained word embeddings
(Pennington et al., 2014). NAML, LSTUR, MINS, and CAUM additionally leverage
category information, with categories embedded with a linear layer. CAUM
additionally encodes title entities and DKN exploits knowledge graph
embeddings (Ji et al., 2015). The shallow word and entity embeddings are
contextualized either using a combination of multi-head self-attention (in
NRMS, MINS, CAUM), or a sequence of CNN (Kim, 2014) and additive attention
networks (in NAML, LSTUR). NPA (Wu et al., 2019c) also utilizes a CNN to
contextualize word embeddings, followed by a personalized attention module,
analogous to the one used in its user encoder, whereas DKN employs a word-
entity-aligned knowledge-aware CNN (Wang et al., 2018). CenNewsRec (Qi et al.,
2020) combines the CNN network with multi-head self-attention and additive
attention modules. Models with multiple feature vectors produce final news
embeddings by simply concatenating them (LSTUR, CAUM), or by attending over
them (NAML, MINS).

###  2.2. Click Behavior Fusion

We question whether the design and computational complexity of early fusion
(EF), i.e., existence of dedicated user encoders in state-of-the-art NNR
models, is justified. To this end, we propose, as a lightweight alternative,
the late fusion (LF) approach that replaces the elaborate user encoders with
mean-pooling of dot-product scores between the embedding of the candidate
$n^{c}$ and the embeddings of the clicked news $n^{u}_{i}$. Given a candidate
news $n^{c}$ and a sequence of news clicked by the user
$H={n^{u}_{1},...,n^{u}_{N}}$, we compute the relevance score of the candidate
news with regards to the user $u$’s history as
$s(\mathbf{n}^{c},u)=\frac{1}{N}\sum_{i=1}^{N}\mathbf{n}^{c}\cdot\mathbf{n}^{u}_{i}$,
where $\mathbf{n}$ denotes the embedding of a news learned by the news encoder
and $N$ the history length.

Although LF suggests that explicitly encoding user behavior may not be
necessary for click prediction, user embeddings are still needed in
collaborative-filtering models (Li and Wang, 2019). Note that the LF
formulation above is equivalent to the dot product between the candidate
embedding $\mathbf{n}^{c}$ and the mean of embeddings of the user’s clicked
news $\mathbf{n}^{u}_{i}$,
$s(\mathbf{n}^{c},u)=\mathbf{n}^{c}\cdot\left(\frac{1}{N}\sum_{i=1}^{N}\mathbf{n}^{u}_{i}\right)$.
This means that LF can also seamlessly provide user embeddings (simply as
averages of clicked news embeddings) if needed. LF can thus been seen as a
parameterless user encoder, i.e., a computationally efficient alternative to
complex parameterized UEs in existing EF models. Because (i) we produce
embeddings of candidate and clicked news independently, and (ii) yield user
embeddings as averages of clicked news embeddings, LF models are candidate
agnostic (C-AG).

###  2.3. Training Objectives

The vast majority of existing NNR work, regardless of the concrete user
modeling architecture, tunes the parameters by minimizing the arguably most
straightforward classification objective, cross-entropy loss (with negative
sampling; see Figure 1), and largely fails to explore effective alternatives,
foremost contrastive objectives (Oord et al., 2018; Wu et al., 2022a). This
prevents understanding of models effectiveness under different training
regimes. We address this limitation by training all models (see §2.1) not only
with (1) common cross-entropy loss (with negative sampling), but also via (2)
a contrastive learning objective, in particular supervised contrastive loss
(Khosla et al., 2020).

##  3\. Experimental Setup

Data. We conduct experiments on the MINDsmall and MINDlarge datasets,
introduced by Wu et al. (2020a). Table 1 summarizes their main statistics.
Since Wu et al. (2020a) do not release test set labels, we use the respective
validation portions for testing, and split the respective training sets into
temporally disjoint training (first four days of data) and validation portions
(the last day).

Table 1. Statistics of the MINDsmall and MINDlarge datasets. | MINDsmall | MINDlarge
---|---|---
Statistic | Train | Test | Train | Test
# News | 51,282 | 42,416 | 101,527 | 72,023
# Users | 49,108 | 48,593 | 698,365 | 248,973
# Impressions | 153,727 | 70,938 | 2,186,683 | 365,201
# Categories | 17 | 17 | 18 | 17
# Subcategories | 264 | 252 | 285 | 269

Implementation and Optimization Details. We use 300-dimensional pretrained
Glove embeddings (Pennington et al., 2014) and 100-dimensional TransE
embeddings (Bordes et al., 2013) pretrained on Wikidata to initialize
respectively the word and entity embeddings of the NNR models under
comparison. We set the maximum history length to 50. Following Wu et al.
(2022a), our negative sampling creates four negatives per positive example. We
find the optimal temperature for SCL using the validation performance,
sweeping the interval $[0.08,0.3]$ with a 0.02 step. We train with batch size
of 512 for all C-AG models, 256 for DKN and only 64 for CAUM (due to
computational limitations). We set all other model-specific hyperparameters,
to optimal values reported in the respective papers. We train all models with
mixed precision, under a fixed computational budget: for 25 epochs on
MINDsmall and 10 epochs on MINDlarge. We optimize with the Adam algorithm
(Kingma and Ba, 2014), with the learning rate set to 1e-4. We repeat each
experiment five times (with different random seeds) and report averages (and
std. deviation) for common metrics: AUC, MRR, nDCG@5, and nDCG@10. Each model
is trained on a single NVIDIA Tesla V100 GPU with 32GB memory. Our
implementation is publicly available.222Code available at:
https://github.com/andreeaiana/simplifying_nnr

##  4\. Results and Discussion

Table 2. Recommendation performance of the compared models under combinations
of click behavior fusion (CBF), and training objectives. We report averages
and standard deviations across five different runs.

|  | MINDsmall | MINDlarge
---|---|---|---
|  | AUC | MRR | nDCG@5 | nDCG@10 | AUC | MRR | nDCG@5 | nDCG@10
Model | CBF | CE | SCL | CE | SCL | CE | SCL | CE | SCL | CE | SCL | CE | SCL | CE | SCL | CE | SCL
| EF | 54.7$\pm$0.6 | 56.5$\pm$0.7 | 29.0$\pm$0.7 | 28.4$\pm$0.6 | 26.9$\pm$0.8 | 26.6$\pm$0.6 | 33.2$\pm$0.8 | 32.6$\pm$0.4 | 56.8$\pm$0.2 | 58.1$\pm$0.8 | 31.4$\pm$0.5 | 30.0$\pm$0.6 | 29.5$\pm$0.4 | 27.7$\pm$0.6 | 35.9$\pm$0.4 | 34.3$\pm$0.6
NPA | LF | 55.1$\pm$0.9 | 57.3$\pm$1.2 | 28.6$\pm$0.3 | 27.5$\pm$1.0 | 26.4$\pm$0.4 | 25.5$\pm$1.1 | 32.9$\pm$0.4 | 31.8$\pm$0.9 | 61.2$\pm$0.5 | 58.7$\pm$0.8 | 32.1$\pm$0.6 | 28.3$\pm$0.5 | 30.2$\pm$0.7 | 26.0$\pm$0.7 | 36.6$\pm$0.6 | 32.7$\pm$0.7
1-18 | EF | 50.1$\pm$0.0 | 57.1$\pm$1.1 | 33.6$\pm$0.5 | 32.2$\pm$0.7 | 31.6$\pm$0.6 | 30.4$\pm$0.7 | 38.0$\pm$0.5 | 36.8$\pm$0.7 | 50.1$\pm$0.0 | 60.4$\pm$0.6 | 33.2$\pm$0.4 | 34.2$\pm$0.4 | 31.3$\pm$0.5 | 32.5$\pm$0.3 | 37.9$\pm$0.4 | 38.9$\pm$0.3
NAML | LF | 50.0$\pm$0.0 | 62.7$\pm$0.5 | 33.7$\pm$0.8 | 32.0$\pm$0.7 | 31.8$\pm$0.9 | 30.3$\pm$0.7 | 38.1$\pm$0.8 | 36.6$\pm$0.6 | 50.0$\pm$0.0 | 65.4$\pm$0.5 | 32.7$\pm$0.5 | 33.5$\pm$0.5 | 31.0$\pm$0.5 | 31.7$\pm$0.5 | 37.6$\pm$0.4 | 38.2$\pm$0.4
1-18 | EF | 52.6$\pm$1.3 | 59.9$\pm$0.6 | 27.6$\pm$0.8 | 29.2$\pm$0.7 | 25.7$\pm$0.5 | 27.2$\pm$0.9 | 32.3$\pm$0.5 | 33.7$\pm$0.7 | 54.6$\pm$1.4 | 62.8$\pm$0.7 | 31.9$\pm$1.0 | 32.4$\pm$0.5 | 30.0$\pm$1.1 | 30.5$\pm$0.7 | 36.6$\pm$1.0 | 36.9$\pm$0.6
NRMS | LF | 58.9$\pm$1.0 | 60.2$\pm$1.1 | 31.8$\pm$0.7 | 30.7$\pm$0.6 | 29.9$\pm$0.7 | 28.7$\pm$0.6 | 36.3$\pm$0.6 | 35.1$\pm$0.6 | 56.1$\pm$2.1 | 63.6$\pm$1.1 | 32.9$\pm$0.7 | 32.4$\pm$0.6 | 31.7$\pm$1.1 | 30.6$\pm$0.8 | 37.8$\pm$0.4 | 37.1$\pm$0.7
1-18 | EFini | 53.5$\pm$1.2 | 55.4$\pm$0.5 | 29.6$\pm$0.5 | 28.1$\pm$0.7 | 28.0$\pm$0.5 | 26.5$\pm$0.7 | 34.4$\pm$0.4 | 32.9$\pm$0.6 | 50.0$\pm$0.1 | 56.9$\pm$1.5 | 32.5$\pm$2.4 | 31.3$\pm$1.5 | 30.9$\pm$2.4 | 29.8$\pm$1.8 | 37.4$\pm$2.4 | 36.2$\pm$1.6
| EFcon | 50.2$\pm$0.0 | 59.8$\pm$1.4 | 31.8$\pm$0.7 | 31.3$\pm$0.8 | 30.1$\pm$0.8 | 30.3$\pm$1.3 | 36.4$\pm$0.7 | 36.2$\pm$0.7 | 51.4$\pm$0.4 | 54.3$\pm$0.4 | 27.7$\pm$0.4 | 26.5$\pm$0.2 | 25.9$\pm$0.5 | 24.6$\pm$0.2 | 32.3$\pm$0.5 | 31.1$\pm$0.2
LSTUR | LF | 50.0$\pm$0.0 | 50.0$\pm$0.0 | 33.8$\pm$0.6 | 33.9$\pm$0.6 | 31.9$\pm$0.7 | 32.0$\pm$0.7 | 38.0$\pm$0.6 | 38.1$\pm$0.6 | 50.0$\pm$0.0 | 50.0$\pm$0.0 | 34.7$\pm$0.6 | 33.1$\pm$0.2 | 33.2$\pm$0.6 | 31.6$\pm$0.4 | 39.2$\pm$0.5 | 37.7$\pm$0.3
1-18 | EF | 54.0$\pm$0.8 | 60.0$\pm$0.4 | 28.3$\pm$0.5 | 30.6$\pm$0.8 | 26.5$\pm$0.4 | 28.5$\pm$0.8 | 32.9$\pm$0.3 | 34.8$\pm$0.7 | 56.4$\pm$0.8 | 64.5$\pm$0.4 | 33.7$\pm$0.3 | 33.5$\pm$0.4 | 31.9$\pm$0.3 | 31.8$\pm$0.5 | 38.3$\pm$0.2 | 38.1$\pm$0.4
CenNewsRec | LF | 59.3$\pm$0.6 | 61.9$\pm$0.7 | 32.8$\pm$0.8 | 32.2$\pm$0.8 | 30.9$\pm$0.8 | 30.4$\pm$0.8 | 37.1$\pm$0.6 | 36.6$\pm$0.7 | 53.3$\pm$0.7 | 64.2$\pm$0.6 | 33.2$\pm$0.4 | 33.3$\pm$0.4 | 31.4$\pm$0.5 | 31.7$\pm$0.4 | 37.9$\pm$0.4 | 38.1$\pm$0.4
1-18 | EF | 50.6$\pm$0.3 | 62.9$\pm$1.7 | 33.7$\pm$1.0 | 32.4$\pm$0.3 | 31.9$\pm$1.1 | 30.7$\pm$0.4 | 38.3$\pm$0.9 | 37.1$\pm$0.3 | 51.7$\pm$0.2 | 65.8$\pm$0.5 | 34.3$\pm$0.2 | 34.4$\pm$0.5 | 32.5$\pm$0.4 | 32.6$\pm$0.5 | 39.1$\pm$0.4 | 39.1$\pm$0.5
MINS | LF | 59.1$\pm$1.2 | 64.2$\pm$0.7 | 35.0$\pm$0.5 | 34.1$\pm$0.6 | 33.2$\pm$0.6 | 32.3$\pm$0.7 | 39.4$\pm$0.6 | 38.5$\pm$0.6 | 53.8$\pm$0.6 | 66.7$\pm$0.8 | 34.9$\pm$0.2 | 34.8$\pm$0.7 | 33.0$\pm$0.2 | 33.1$\pm$0.7 | 39.5$\pm$0.2 | 39.6$\pm$0.6
| EF | 50.0$\pm$0.0 | 51.0$\pm$2.3 | 26.4$\pm$0.6 | 25.9$\pm$0.9 | 24.4$\pm$0.7 | 23.9$\pm$1.1 | 31.0$\pm$0.6 | 30.5$\pm$0.9 | 50.0$\pm$0.0 | 50.0$\pm$0.0 | 25.2$\pm$0.4 | 24.8$\pm$0.3 | 23.4$\pm$0.7 | 22.6$\pm$0.3 | 30.0$\pm$0.5 | 29.1$\pm$0.3
DKN | LF | 50.0$\pm$0.0 | 50.0$\pm$0.0 | 27.5$\pm$0.6 | 26.4$\pm$0.8 | 25.0$\pm$0.5 | 24.0$\pm$0.8 | 31.7$\pm$0.6 | 30.8$\pm$0.8 | 50.0$\pm$0.0 | 50.0$\pm$0.0 | 29.1$\pm$0.4 | 27.8$\pm$1.1 | 26.3$\pm$0.3 | 25.4$\pm$1.0 | 33.2$\pm$0.4 | 32.1$\pm$1.0
1-18 | EF | 61.4$\pm$1.0 | 63.2$\pm$0.9 | 33.8$\pm$0.6 | 33.7$\pm$0.8 | 32.0$\pm$0.6 | 31.8$\pm$0.9 | 38.4$\pm$0.5 | 38.2$\pm$0.8 | 67.1$\pm$0.8 | 66.4$\pm$0.9 | 35.3$\pm$0.5 | 35.1$\pm$0.5 | 33.6$\pm$0.6 | 33.4$\pm$0.6 | 40.1$\pm$0.5 | 39.9$\pm$0.5
CAUM | LF | 62.4$\pm$0.8 | 63.5$\pm$0.8 | 33.7$\pm$0.6 | 33.7$\pm$0.7 | 31.8$\pm$0.5 | 31.8$\pm$0.8 | 38.2$\pm$0.5 | 38.0$\pm$0.7 | 53.1$\pm$0.3 | 65.9$\pm$0.2 | 34.5$\pm$0.4 | 34.5$\pm$0.1 | 32.6$\pm$0.3 | 32.8$\pm$0.1 | 39.2$\pm$0.3 | 39.3$\pm$0.1

Table 2 shows the performance on MINDsmall and MINDlarge for both C-AG (NPA,
NAML, NRMS, LSTUR, CenNewsRec, and MINS) and C-AW models (DKN, CAUM), under
four different configurations of our comparative evaluation framework: (i)
user modeling with EF vs. LF, combined with (2) training with CE vs. SCL
objective. We next dissect the results along the three axes of our framework
(§2): user modeling, click behavior fusion, and training objectives.

Candidate-Agnostic vs. Candidate-Aware NNRs. We analyze C-AG vs. C-AW models
under their default EF configuration, since with LF all models become
candidate-agnostic. CAUM, with the most complex and candidate-aware UE,
generally outperforms all other models under both training regimes (CE and
SCL) and for most evaluation metrics. The gaps are particularly prominent on
the large training dataset, MINDlarge, w.r.t. the AUC metric. This result
alone could mislead to a conclusion that more complex, candidate-aware user
modeling is necessary for better recommendation. The fact that (1) DKN, as the
other C-AW model in our evaluation – generally performs much worse than C-AG
models, as well as that (2) our LF models variants with trivial, parameterless
UEs match or surpass the performance of CAUM with EF, undermine this
conclusion. With the exception of DKN, all other models exhibit better
performance when trained on the larger MINDlarge dataset. NAML and MINS are,
however, competitive (except w.r.t. AUC metric) on MINDsmall, but fall behind
CAUM on MINDlarge, suggesting that CAUM’s elaborate UE benefits the most from
more training data.

One confounding factor that we do not control for, however, and which warrants
a mindful comparison of the results, is that models differ not just in UE, but
also in NE components, i.e., w.r.t. how they encode news and which features
they use as input. For example, NAML and MINS, with an identical NE, achieve
similar performance on MINDsmall. On MINDlarge, however, the more complex UE
of MINS brings substantial gains over the simpler UE of NAML (but only under
standard EF fusion and CE training).

Early vs. Late Click Behavior Fusion. Replacing complex EF-based UEs with the
simple parameterless LF that we propose brings substantial performance gains
across the board. Averaged across all models and both training objectives, LF
brings massive gains of 5.58 and 4.63 MRR points on MINDsmall and MINDlarge,
respectively. Equally importantly, with LF – i.e., with the same parameterless
UE – models exhibit mutually much more similar performance than under EF, with
other models generally closing the gap to CAUM. This suggests that LF makes
differences in NE architectures across models less consequential, thus not
only simplifying UE with parameterless averaging of clicked news embeddings,
but also allowing for simpler news encoders.

Cross-Entropy vs. Supervised Contrastive Loss. Overall, we find SCL to be a
viable alternative to the common cross-entropy based classification with
negative sampling (compare columns CE and SCL across evaluation metrics in
Table 2). SCL brings large gains over CE in terms of AUC (+8.26 points on
MINDsmall and +12.14 points on MINDlarge, averaged across all models, in both
EF and LF variants). This suggests that, SCL leads to better separation of
clicked and not clicked news in the representation space. In contrast, SCL
falls slightly behind CE according to ranking measures, MRR and nDCG (-1.54
and -1.78 MRR points on MINDsmall and MINDlarge, respectively). We hypothesize
that this is because of hard negatives – news not clicked by the user that
resemble user’s clicked news – for which CE more directly signals irrelevance:
these likely become highly-ranked false positives for SCL-trained models.

Figure 2. Number of model parameters (in millions).

Model Size. Finally, we quantify the reduction in model parameters that LF
brings w.r.t. EF. Figure 2 shows the number of trainable parameters in
original EF configurations, on MINDsmall.333For some models, e.g., LSTUR with
its user embedding matrix, the number of parameters depends on the size of the
training data. While the NE accounts for the majority of parameters in most
models, the plot shows that the proportion of UE parameters is non-negligible
for several models, and largest by a wide margin for LSTUR. With a
parameterless UE, along with performance gains, LF brings a relative reduction
of model size of 14.7%, 18.1%, and massive 82.3% for CenNewsRec, CAUM, and
LSTURini, respectively.

##  5\. Conclusion

Rapid development of personalized neural news recommenders hinders fair
comparative model evaluations and systematic analyses of design choices. In
this work we introduce a unified framework for neural news recommendation
focusing on three crucial design dimensions of NNR: (i) candidate-awareness in
user modeling, (ii) click behavior fusion, and (iii) training objectives.
Extensive evaluation of a wide range of recent state-of-the-art models reveals
that NNR can be drastically simplified: replacing complex user encoders with
parameterless aggregation of clicked news embeddings brings substantial
performance gains across the board, reducing at the same time model
complexity. Further, we show that contrastive learning can be a viable
alternative to standard classification-based (cross-entropy) loss. We hope
that our findings will inspire more transparent NNR evaluation, including
systematic model ablations to uncover components that drive the performance.

###### Acknowledgements.

The authors acknowledge support by the state of Baden-Württemberg through
bwHPC. Andreea Iana was supported by the ReNewRS project grant, which is
funded by the Baden-Württemberg Stiftung in the Responsible Artificial
Intelligence program.

## References

  * (1)
  * An et al. (2019) Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019\.  Neural News Recommendation with Long- and Short-term User Representations. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 336–345.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014\.  Neural Machine Translation by Jointly Learning to Align and Translate.  _ICLR_ (2014).
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013\.  Translating Embeddings for Modeling Multi-relational Data. In _Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2_ , Vol. 26.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014\.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 1724–1734.
  * Gong and Zhu (2022) Shansan Gong and Kenny Q Zhu. 2022.  Positive, Negative and Neutral: Modeling Implicit Feedback in Session-based News Recommendation. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1185–1195.
  * Gulla et al. (2017) Jon Atle Gulla, Lemei Zhang, Peng Liu, Özlem Özgöbek, and Xiaomeng Su. 2017\.  The Adressa Dataset for News Recommendation. In _Proceedings of the International Conference on Web Intelligence_. 1042–1048.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013\.  Learning Deep Structured Semantic Models for Web Search using Clickthrough Data. In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_. 2333–2338.
  * Iana et al. (2022) Andreea Iana, Mehwish Alam, and Heiko Paulheim. 2022\.  A Survey on Knowledge-aware News Recommender Systems.  _Semantic Web_ Preprint (2022), 1–62.
  * Ji et al. (2015) Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015.  Knowledge Graph Embedding via Dynamic Mapping Matrix. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 687–696.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised Contrastive Learning. In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , Vol. 33. 18661–18673.
  * Kim (2014) Yoon Kim. 2014.  Convolutional Neural Networks for Sentence Classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Doha, Qatar, 1746–1751.  https://doi.org/10.3115/v1/D14-1181
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A Method for Stochastic Optimization.  _ICLR_ (2014).
  * Li and Wang (2019) Miaomiao Li and Licheng Wang. 2019.  A Survey on Personalized News Recommendation Technology.  _IEEE Access_ 7 (2019), 145861–145879.
  * Li et al. (2021) Yizhi Li, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2021\.  More Robust Dense Retrieval with Contrastive Dual Learning. In _Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval_. 287–296.
  * Okura et al. (2017) Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017\.  Embedding-based News Recommendation for Millions of Users. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. 1933–1942.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation Learning with Contrastive Predictive Coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global Vectors for Word Representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 1532–1543.
  * Qi et al. (2021a) Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2021a.  Personalized News Recommendation with Knowledge-aware Interactive Matching. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 61–70.
  * Qi et al. (2021b) Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2021b.  PP-Rec: News Recommendation with Personalized User Interest and Time-aware News Popularity. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 5457–5467.
  * Qi et al. (2022) Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2022\.  News Recommendation with Candidate-aware User Modeling. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1917–1921.
  * Qi et al. (2020) Tao Qi, Fangzhao Wu, Chuhan Wu, Yongfeng Huang, and Xing Xie. 2020.  Privacy-Preserving News Recommendation Model Learning. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. 1423–1432.
  * Qi et al. (2021c) Tao Qi, Fangzhao Wu, Chuhan Wu, Peiru Yang, Yang Yu, Xing Xie, and Yongfeng Huang. 2021c.  HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 5446–5456.
  * Raza and Ding (2022) Shaina Raza and Chen Ding. 2022.  News Recommender System: A Review of Recent Progress, Challenges, and Opportunities.  _Artificial Intelligence Review_ (2022), 1–52.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need.  _Proceedings of the 31st International Conference on Neural Information Processing Systems_ 30\.
  * Wang et al. (2020) Heyuan Wang, Fangzhao Wu, Zheng Liu, and Xing Xie. 2020\.  Fine-grained Interest Matching for Neural News Recommendation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 836–845.
  * Wang et al. (2018) Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018\.  DKN: Deep Knowledge-aware Network for News Recommendation. In _Proceedings of the 2018 World Wide Web Conference_. 1835–1844.
  * Wang et al. (2022) Rongyao Wang, Shoujin Wang, Wenpeng Lu, and Xueping Peng. 2022\.  News Recommendation via Multi-interest News Sequence Modelling. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7942–7946.
  * Wei et al. (2021) Yinwei Wei, Xiang Wang, Qi Li, Liqiang Nie, Yan Li, Xuanping Li, and Tat-Seng Chua. 2021\.  Contrastive Learning for Cold-start Recommendation. In _Proceedings of the 29th ACM International Conference on Multimedia_. 5382–5390.
  * Wu et al. (2019b) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b.  Neural News Recommendation with Attentive Multi-view Learning. In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_. 3863–3869.
  * Wu et al. (2019c) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019c.  NPA: Neural News Recommendation with Personalized Attention. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2576–2584.
  * Wu et al. (2019a) Chuhan Wu, Fangzhao Wu, Mingxiao An, Yongfeng Huang, and Xing Xie. 2019a.  Neural News Recommendation with Topic-aware News Representation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 1154–1159.
  * Wu et al. (2019d) Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019d.  Neural News Recommendation with Multi-head Self-attention. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 6389–6394.
  * Wu et al. (2022a) Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2022a.  Rethinking InfoNCE: How Many Negative Samples Do You Need?. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , Lud De Raedt (Ed.). International Joint Conferences on Artificial Intelligence Organization, 2509–2515.
  * Wu et al. (2023) Chuhan Wu, Fangzhao Wu, Yongfeng Huang, and Xing Xie. 2023\.  Personalized News Recommendation: Methods and Challenges.  _ACM Transactions on Information Systems_ 41, 1 (2023), 1–50.
  * Wu et al. (2020b) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020b.  Sentirec: Sentiment Diversity-aware Neural News Recommendation. In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_. 44–53.
  * Wu et al. (2021) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021\.  Empowering News Recommendation with Pre-trained Language Models. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1652–1656.
  * Wu et al. (2022b) Chuhan Wu, Fangzhao Wu, Tao Qi, Chenliang Li, and Yongfeng Huang. 2022b.  Is News Recommendation a Sequential Recommendation Task?. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2382–2386.
  * Wu et al. (2020a) Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020a.  Mind: A Large-scale Dataset for News Recommendation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 3597–3606.
  * Xie et al. (2022) Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Jiandong Zhang, Bolin Ding, and Bin Cui. 2022\.  Contrastive Learning for Sequential Recommendation. In _2022 IEEE 38th International Conference on Data Engineering (ICDE)_. IEEE, 1259–1273.
  * Yang et al. (2022) Chun Yang, Jianxiao Zou, JianHua Wu, Hongbing Xu, and Shicai Fan. 2022.  Supervised Contrastive Learning for Recommendation.  _Knowledge-Based Systems_ 258 (2022), 109973.
  * Yu et al. (2022) Yang Yu, Fangzhao Wu, Chuhan Wu, Jingwei Yi, and Qi Liu. 2022.  Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. 5478–5489.
  * Zhang et al. (2021) Qi Zhang, Qinglin Jia, Chuyuan Wang, Jingjie Li, Zhaowei Wang, and Xiuqiang He. 2021\.  Amm: Attentive Multi-field Matching for News Recommendation. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1588–1592.