#  Quadratic Functional Encryption for Secure Training in Vertical Federated
Learning

Shuangyi Chen1, Anuja Modi2, Shweta Agrawal2, and Ashish Khisti1  1University
of Toronto<EMAIL_ADDRESS><EMAIL_ADDRESS>2IIT
Madras<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Vertical federated learning (VFL) enables the collaborative training of
machine learning (ML) models in settings where the data is distributed amongst
multiple parties who wish to protect the privacy of their individual data.
Notably, in VFL, the labels are available to a single party and the complete
feature set is formed only when data from all parties is combined. Recently,
Xu et al. [1] proposed a new framework called FedV for secure gradient
computation for VFL using multi-input functional encryption. In this work, we
explain how some of the information leakage in Xu et al. can be avoided by
using Quadratic functional encryption when training generalized linear models
for vertical federated learning.

##  I Introduction

In many emerging applications, a machine learning (ML) model must be trained
using private data that is distributed among multiple parties. We study the
setting of vertical federated learning (VFL) where each individual party has
access to a subset of features and labels and must cooperate to train a ML
model that makes use of all the features. When privacy of user data is
required, homomorphic encryption (HE) [2, 3], which enables the computation on
encrypted data, provides a natural solution. In recent years, there has been a
significant interest in HE based VFL systems, see e.g., [5, 6, 7, 8, 9, 10,
11, 12]. Some works such as [7, 8, 11] consider a two-party protocol without
the trusted coordinator, while others [10, 9]. consider a multi-party
settings. Those frameworks require a large amount of peer-to-peer
communications. References [5, 6] propose frameworks comprised of one trusted
coordinator, storing the global weights, and two parties, each with a subset
of vertically partitioned data. However, these frameworks require the trusted
coordinator to share plaintext global weights with parties, which undermines
the model’s confidentiality.

In a recent work, Xu et. al. [1] proposed a generic and efficient privacy-
preserving vertical Federated Learning (VFL) framework known as FedV in the
multiparty setting. FedV makes use of single-input function encryption
($\mathsf{SIFE}$) and multi-input function encryption ($\mathsf{MIFE}$), and
makes the communication between the clients and the aggregator a one-round
interaction. However, FedV still have some key drawbacks. The protocol can
reveal more information to the aggregator than just the final gradient in each
iteration. Moreover, the protocol reveals the respective updated weights in
each iteration to clients, which additionally creates leakage. For more
details, please see Section IV.

###  I-A Our Results.

We observe that the leakage created in FedV is caused by choosing an multi-
input functional encryption ($\mathsf{MIFE}$) scheme that only supports linear
functions. Due to this, the weights are required to be provided to each party
for inclusion in encryption, which creates unnecessary leakage. We observe
that for linear models, this leakage can be prevented by using a more powerful
$\mathsf{MIFE}$ scheme, namely $\mathsf{MIFE}$ for quadratic functions which
can also be constructed using standard assumptions in cryptography [17, 18].
As our main contribution in this work, we demonstrate how such a function
encryption scheme can be applied in VFL training by proposing a novel
construction of function vectors that serve as a basis for generating
decryption keys. Our approach leads to direct computation of the gradients,
without leakage of any intermediate results as is the case with FedV. We
discuss our proposed protocol, SFedV, for linear model training in Section IV
and the extension to logistic regression model in Appendix C. We provide a
thorough analysis of both security and efficiency in Section IV.

##  II System Model

###  II-A System Overview

Our system model involves three types of entities: aggregator, $N$ clients,
and Trusted Third Party (TTP). In the $t$th iteration for $t\in[T]$, each
client holds a subset of features $\mathbold{X}_{i}^{t}\in\mathbb{R}^{S\times
F_{i}}$ where $F_{i}$ is the number of features that client party $i$ holds
and $S$ is the batch size. A complete feature set of the current iteration is
expressed as
$\mathbold{X}^{t}=[\mathbold{X}_{0}^{t}\mathbin{\|}...\mathbin{\|}\mathbold{X}_{N-1}^{t}]\in\mathbb{R}^{S\times
F}$. One of the client parties holds the corresponding labels
$\mathbold{y}^{t}\in\mathbb{R}^{S\times 1}$. The aggregator holds the entire
model weights
$\mathbold{w}^{t}=[\mathbold{w}^{t}_{0}\mathbin{\|}\mathbold{w}^{t}_{1}\mathbin{\|}...\mathbin{\|}\mathbold{w}^{t}_{N-1}]\in\mathbb{R}^{F\times
1}$ where $\mathbold{w}^{t}_{i}\in\mathbb{R}^{F_{i}\times 1}$ is the partial
weights that pertains to $\mathbold{X}_{i}^{t}$. The aggregator is responsible
for computing the model weights and the TTP is responsible for the generation
of keys. In this work we focus on linear models of the form:
$f(\mathbold{X}^{t},\mathbold{w}^{t})=\mathbold{X}^{t}\cdot\mathbold{w}^{t}$
with a squared-error loss function:
$L(\mathbold{w}^{t})=\frac{1}{S}\sum_{s\in[S]}||\mathbold{y}^{t}[s]-(\mathbold{X}^{t}\cdot\mathbold{w}^{t})[s]||^{2}$.
In our discussion, we will define the prediction error as:

$\mathbold{u}^{t}=(\mathbold{y}^{t}-\mathbold{X}_{0}^{t}\cdot\mathbold{w}_{0}^{t}-...-\mathbold{X}_{N-1}^{t}\cdot\mathbold{w}_{N-1}^{t}).$
(1)

The gradient of $L(\mathbold{w}^{t})$ with respect to $\mathbold{w}^{t}$ is
expressed as

$\displaystyle
g(\mathbold{w}^{t})=-\frac{2}{S}\begin{bmatrix}\mathbold{y^{t}}^{\top}\mathbold{X}_{0}^{t}-\sum^{N-1}_{i=0}{\mathbold{w}_{i}^{t}}^{\top}{\mathbold{X}_{i}^{t}}^{\top}\mathbold{X}_{0}^{t}\\\
...\mathbin{\|}\\\
{\mathbold{y}^{t}}^{\top}\mathbold{X}_{N-1}^{t}-\sum^{N-1}_{i=0}{\mathbold{w}_{i}^{t}}^{\top}{\mathbold{X}_{i}^{t}}^{\top}\mathbold{X}_{N-1}^{t}\end{bmatrix}\in\mathbb{R}^{1\times
F}$ (2)

The gradient is used to update the global weights in each iteration according
to $\mathbold{w}^{t+1}=\mathbold{w}^{t}-\alpha g(\mathbold{w}^{t})$ where
$\alpha$ is the learning rate. We also discuss logistic regression model in
Appendix C. In each iteration of the training phase, our protocol takes as
input an encrypted copy of the features $\mathbold{X}_{i}^{t}$ and encrypted
labels $\mathbold{y}^{t}\in\mathbb{R}^{S}$ from clients, and collaboratively
and securely computes the gradients $g(\mathbold{w}^{t})$.

Our threat model is defined as follows: we assume the aggregator is honest-
but-curious meaning it correctly follows the algorithms and protocols but will
try to infer clients’ private data. Additionally, we assume that the
aggregator does not collude with anyone. Similarly, the trusted third party is
assumed not to collude with anyone. With respect to the clients, we assume
that there are at most $N-1$ dishonest clients who may collude together and
share their data to infer honest clients’ information.

The protocol enables all the entities to collaboratively compute the gradient
using vertically partitioned data. During the training process, we aim to
achieve the following privacy requirements: 1) The client $i$ and the
aggregator should learn nothing about data $\mathbold{X}_{j}$ of client $j$
for $i\neq j$. 2) Any client should learn nothing about the trained global
model weights $\mathbold{w}$, intermediate results including the prediction
error as in (1) and the gradient $g(\mathbold{w})$. Moreover, $i$th client
should not learn anything about his/her own corresponding weights
$\mathbold{w}_{i}$.

##  III Preliminaries

###  III-A Functional Encryption

Functional Encryption [14, 15, 16] is a public key encryption scheme that
enables fine-grained access control over the encrypted data. In Single Input
Functional Encryption($\mathsf{SIFE}$), the secret key is associated with a
function $f$, and the ciphertext is associated with the vector $\mathbold{x}$.
The decryption of ciphertext using the secret key outputs $f(\mathbold{x})$.
Intuitively, the security says that the adversary learns nothing about the
input $\mathbold{x}$ beyond what is revealed by
$\\{f_{i}(\mathbold{x})\\}_{i}$ for any set of secret keys corresponding to
the functions $\\{f_{i}\\}_{i}$ that the adversary holds.

###  III-B Multi-Input Functional Encryption

Goldwasser et al. [13] generalized the functional encryption to support
functions with multiple inputs. Multi-Input Functional Encryption, denoted as
$\mathsf{MIFE}$ supports functions with arity greater than one. In
$\mathsf{MIFE}$, the secret key is associated with a function $f$, and the
$i$th ciphertext is associated with the vector $\mathbold{x}_{i}$ for
$i\in[N]$ where $N$ is the arity of the function $f$. The decryption of all
the ciphertexts using the secret key outputs
$f(\mathbold{x}_{1},\ldots,\mathbold{x}_{N})$. We now describe this notion in
more detail.

######  Definition 1 (Multi-Input Functional Encryption ($\mathsf{MIFE}$)
[18]).

Syntax. Let $N$ be the number of encryption slots, and
$\mathcal{F}=\\{\mathcal{F}_{N}\\}_{N\in\mathbb{N}}$ be a function family such
that, for all $f\in\mathcal{F}_{N}$,
$f:\mathcal{X}_{1}\times\cdots\times\mathcal{X}_{N}\to\mathcal{Y}$. Here
$\mathcal{X}_{i}$ and $\mathcal{Y}$ be the input and output spaces
(respectively). A multi-input functional encryption $(\mathsf{MIFE})$ scheme
for function family $\mathcal{F}$ consists of the following algorithms.

$\mathsf{Setup}(1^{\lambda},1^{N})\rightarrow(\mathsf{PP},\\{\mathsf{EK}_{i}\\}_{i},\mathsf{MSK})$.

It takes a security parameter $1^{\lambda}$, number of slots $1^{N}$, and
outputs a public parameter $\mathsf{PP}$, $N$ encryption keys
$\\{\mathsf{EK}_{i}\\}_{i\in[N]}$ and a master secret key $\mathsf{MSK}$. (The
remaining algorithms implicitly take $\mathsf{PP}$ as input.)

$\mathsf{Enc}(\mathsf{EK}_{i},\mathbold{x})\rightarrow\mathsf{CT}_{i}$.

It takes the $i$th encryption key $\mathsf{EK}_{i}$ and an input
$\mathbold{x}\in\mathcal{X}_{i}$, and outputs a ciphertext $\mathsf{CT}_{i}$.

$\mathsf{KeyGen}(\mathsf{MSK},f)\rightarrow\mathsf{SK}$.

It takes the master secret key $\mathsf{MSK}$ and function $f\in\mathcal{F}$
as inputs, and outputs a secret key $\mathsf{SK}$.

$\mathsf{Dec}(\mathsf{CT}_{1},\ldots,\mathsf{CT}_{N},\mathsf{SK})\rightarrow
y$.

It takes $n$ ciphertexts $\mathsf{CT}_{1},\ldots,\mathsf{CT}_{N}$ and secret
key $\mathsf{SK}$, and outputs a decryption value $y\in\mathcal{Y}$ or a
special abort symbol $\bot$.

Correctness. An $\mathsf{MIFE}$ scheme for the function family $\mathcal{F}$
is correct if for all
$\lambda,N\in\mathbb{N},\;(\mathbold{x}_{1},\ldots,\mathbold{x}_{N})\in\mathcal{X}_{1}\times\cdots\times\mathcal{X}_{N},\;f\in\mathcal{F}_{N}$,
we have

$\displaystyle\Pr\left[y=f(x_{1},\ldots,x_{N}):\begin{array}[]{l}(\mathsf{PP},\\{\mathsf{EK}_{i}\\}_{i},\mathsf{MSK})\leftarrow\mathsf{Setup}(1^{\lambda},1^{N})\\\
\\{\mathsf{CT}_{i}\leftarrow\mathsf{Enc}(\mathsf{EK}_{i},\mathbold{x})\\}_{i\in[N]}\\\
\mathsf{SK}\leftarrow\mathsf{KeyGen}(\mathsf{MSK},f)\\\
y=\mathsf{Dec}(\mathsf{CT}_{1},\ldots,\mathsf{CT}_{N},\mathsf{SK})\end{array}\right]=1.$

Security. Intuitively, security says that no information about the messages
can be learned by the adversary except what is revealed by virtue of
functionality – in more detail, an adversary possessing some ciphertexts and
secret keys can perform decryption and learn the output of the functionality,
which itself leaks something about the underlying plaintext. But besides this
necessary leakage, the adversary does not learn anything. We provide the
formal definition of security in Appendix A.

Multi-Input FE for Quadratic Functions. Agrawal, Goyal, and Tomida [18]
constructed a multi-input functional encryption for quadratic functions
($\mathsf{qMIFE}$). Let us define the $N$ input quadratic function $f$ as
$f(\mathbold{x}_{1},\ldots,\mathbold{x}_{N})=\langle\mathbold{c},\mathbold{x}\otimes\mathbold{x}\rangle$
where $\mathbold{x}=(\mathbold{x}_{1}||\ldots||\mathbold{x}_{N})$. Here
$\otimes$ denotes the Kronecker product. A $n$-input $\mathsf{MIFE}$ scheme
for the function class $\mathcal{F}_{m,n}$ is defined as: each $i$th client
encrypts $\mathbold{x}_{i}\in\mathbb{Z}^{m}$ using $i$th encryption key
$\mathsf{EK}_{i}$ to get the $i$th ciphertext $\mathsf{CT}_{i}$ for $i\in[n]$.
The $\mathsf{KeyGen}$ algorithm issues the secret key $\mathsf{SK}$ for
$\mathbold{c}\in\mathbb{Z}^{(mn)^{2}}$ where $\mathbold{c}$ is the vector
representation of the function $f\in\mathcal{F}_{m,n}$. The $\mathsf{Dec}$
algorithm uses the secret key $\mathsf{SK}$ to decrypt
$\mathsf{CT}_{1},\ldots,\mathsf{CT}_{n}$ to get
$\langle\mathbold{c},\mathbold{x}\otimes\mathbold{x}\rangle$ and nothing else.

###  III-C FedV

As the system model of SFedV (Section II-A), FedV involves an aggregator, N
clients, and a Trusted Third Party (TTP). Each client holds a subset of
features $\mathbold{X}_{i}\in\mathbb{R}^{S\times F_{i}}$, and the first client
also has the corresponding labels $\mathbold{y}\in\mathbb{R}^{S}$ along with
its subset of features. The aggregator holds the complete model weights
$\mathbold{w}=[\mathbold{w}_{0}||\mathbold{w}_{1}||...||\mathbold{w}_{N-1}]$
where $\mathbold{w}_{i}\in\mathbb{R}^{F_{i}}$ is the partial weights that
pertains to $\mathbold{X}_{i}$. In each iteration, there are two steps to
compute the gradient

$\displaystyle
g(\mathbold{w})=-\frac{2}{S}\begin{bmatrix}\mathbold{u}^{\top}\mathbold{X}_{0}\mathbin{\|}...\mathbin{\|}\mathbold{u}^{\top}\mathbold{X}_{N-1}\end{bmatrix}$
(3)

where
$(\mathbold{u})^{\top}=(\mathbold{y}-\mathbold{X}_{0}\mathbold{w}_{0}-...-\mathbold{X}_{N-1}\mathbold{w}_{N-1})^{\top}$.

In the first step called Feature Dimension Secure Aggregation, FedV uses a
Multi-Input Functional Encryption $(\mathsf{MIFE})$ scheme for the inner
product functionality [19, 20] to securely compute the prediction error
$\mathbold{u}$. In this step, the aggregator sends each $i$th client the
partial weights $\mathbold{w}_{i}$. Then each $i$th client encrypts each
sample of $(-\mathbold{X}_{i}\mathbold{w}_{i})\in\mathbb{R}^{S}$ and sends the
ciphertext set
$\mathsf{CT}_{-\mathbold{X}_{i}\mathbold{w}_{i}}^{\mathsf{MIFE}}$ to the
aggregator. The first client, holding the label $\mathbold{y}$, encrypts each
sample of $\mathbold{y}-\mathbold{X}_{1}\mathbold{w}_{1}$ and sends the
ciphertext set
$\mathsf{CT}_{\mathbold{y}-\mathbold{X}_{1}\mathbold{w}_{1}}^{\mathsf{MIFE}}$
to the aggregator. The aggregator asks the TTP for the secret key
$\mathsf{SK}_{\mathbold{v}}^{\mathsf{MIFE}}$ corresponding to the fusion
vector $\mathbold{v}$. This vector $\mathbold{v}$ can be a binary vector where
one in $i$th position means that the aggregator has received ciphertext from
client $i$. Using the secret key $\mathsf{SK}_{\mathbold{v}}^{\mathsf{MIFE}}$,
the aggregator decrypts the ciphertexts
$\\{\\{\mathsf{CT}_{-\mathbold{X}_{i}\mathbold{w}_{i}}^{\mathsf{MIFE}}\\}_{i=1}^{N-1},\mathsf{CT}_{\mathbold{y}-\mathbold{X}_{1}\mathbold{w}_{1}}^{\mathsf{MIFE}}\\}$
to get the prediction error $\mathbold{u}$ (Equation (1)), which is the inner
product of the fusion vector $\mathbold{v}$ and the partial predictions from
clients.

In the second step called Sample Dimension Secure Aggregation, FedV uses
Single-Input Functional Encryption $(\mathsf{SIFE})$ scheme to compute the
gradient $g(\mathbold{w})$. In this step, each client $i$ encrypts each
element of $\mathbold{X}_{i}$ and sends the ciphertext set
$\mathsf{CT}_{\mathbold{X}_{i}}^{\mathsf{SIFE}}$ to the aggregator. On
receiving the secret key $\mathsf{SK}_{\mathbold{u}}^{\mathsf{SIFE}}$
corresponding to the prediction error $\mathbold{u}$ from TTP, the aggregator
decrypts the ciphertexts to get
$\\{\mathbold{u}^{\top}\mathbold{X}_{i}\\}_{i\in[N]}$. The aggregator further
processes the decryption results
$\\{\mathbold{u}^{\top}\mathbold{X}_{i}\\}_{i\in[N]}$ according to Equation
(3) to get the gradient $g(\mathbold{w})$. Using the gradients, the model
weights are updated and then the training of the next epoch starts. Note the
transmission of $\mathsf{MIFE}$ ciphertext
($\mathsf{CT}_{\mathbold{X}_{i}\mathbold{w}_{i}}^{\mathsf{MIFE}}$ or
$\mathsf{CT}_{\mathbold{y}-\mathbold{X}_{1}\mathbold{w}_{1}}^{\mathsf{MIFE}}$)
and $\mathsf{SIFE}$ ciphertext
($\mathsf{CT}_{\mathbold{X}_{i}}^{\mathsf{SIFE}}$) can be simultaneous. Thus
the communication between each client and the aggregator is a one-round
interaction in each iteration.

Leakage in FedV. While FedV preserves each client’s data, it reveals the
intermediate result, the prediction error $\mathbold{u}$ to the aggregator.
Moreover, in the Feature Dimension Secure Aggregation step, the $i$th client
is required to know the respective weight $\mathbold{w}_{i}$. Additionally,
the aggregator can use the secret key of $t$th iteration to decrypt the
ciphertext of some other iteration $t^{\prime}$ for $t^{\prime}\neq t$ to
infer more information about client data.

##  IV The Protocol

Now we introduce our protocol with Multi-input Quadratic Functional Encryption
$\mathsf{qMIFE}$ [18] as a privacy enhancement technology to do training in
VFL setting.

At the beginning of the training phase, the aggregator initializes the global
weights $\mathbold{w}^{0}$ and starts training. The training phase is
iterative, where in the $t$th iteration, the TTP runs the
$\mathsf{qMIFE}.\mathsf{Setup}$ algorithm to get the public parameters
$\mathsf{PP}^{t}$, $N$ encryption keys $\\{\mathsf{EK}_{i}\\}_{i\in[N]}^{t}$
and a master secret key $\mathsf{MSK}^{t}$, then delivers the encryption key
$\textsf{EK}_{i}^{t}$ to the corresponding client $i$. After receiving the
encryption key and determining the batch $\mathbold{X}^{t}$ used for training
in this iteration, each client uses $\textsf{EK}_{i}^{t}$ to encrypt
$\mathbold{x}_{i}^{t}$ which is the vectorized ${\mathbold{X}_{i}^{t}}$
(vec$(\cdot)$ stacks the columns of a matrix into a vector) to get ciphertext
$\textsf{CT}_{i}^{t}$. Each client sends $\textsf{CT}_{i}^{t}$ to the
aggregator. The client that holds the labels encrypts $\mathbold{x}_{i}^{t}$
and $\mathbold{y}^{t}$ with $\textsf{EK}_{i}^{t}$ to get ciphertexts
$\textsf{CT}_{i}^{t}$ and $\textsf{CT}_{\mathbold{y}}^{t}$ respectively and
then sends $(\textsf{CT}_{i}^{t},\textsf{CT}_{\mathbold{y}}^{t})$ to the
aggregator. At the same time, the aggregator computes a set of function
vectors $\mathcal{C}^{t}$ according to the model weights $\mathbold{w}^{t}$ of
the current iteration and sends them to TTP to generate a set of decryption
keys. Detailed procedure is described in Section IV-A. Then, the aggregator
decrypts all the ciphertexts
$(\\{\textsf{CT}_{i}^{t}\\}_{i=0}^{N-1},\textsf{CT}_{y}^{t})$ that were
received from clients using the secret keys received from TTP, to get each
element of (2) respectively. By concatenating those elements and further
processing the results, the aggregator gets the gradients. After this, it can
update the global weights and start the training of the next iteration.
Algorithm 1 shows the training procedure for linear models and also supports
the training for logistic regression as discussed in Appendix C.

Algorithm 1 Training Procedure

1:procedure Training-
Aggregator($\mathbold{w}^{t},s,\\{F_{i}\\}_{i=0}^{N-1},N$)

2: $\mathbold{res}=\textbf{0}^{F}$

3: for each $i\in{0,...,N-1}$ do

4: for each $p\in{0,...,F_{i}-1}$ do

5: $\mathbold{c}_{i,p}^{t}:=$ CGEN($\mathbold{w}^{t},S,{F},N,i,p$)

6: end for

7: end for

8: $\mathcal{C}^{t}:=\\{\mathbold{c}^{t}_{i,p},i\in[N],p\in[F_{i}]\\}$

9:
$\\{\\{\mathsf{qMIFE}.\mathsf{SK}^{t}_{\mathbold{c}_{i,p}}\\}_{p=0}^{F_{i}}\\}_{i=0}^{N-1}=\textsf{obtain-
dk-from-TTP}(\mathcal{C}^{t})$

10: for each $i\in{0,...,N-1}$ do

11: if party $i$ has label $\mathbold{y}^{t}$ then

12: ($\mathsf{CT}_{i}^{t},\mathsf{CT}_{y}^{t}$)= obtain-ct-from-client()

13: else

14: $\mathsf{CT}_{i}^{t}$ = obtain-ct-from-client()

15: end if

16: end for

17:
${\mathsf{CT}}^{t}=\\{\\{{\mathsf{CT}}^{t}_{i}\\}_{i=0}^{N-1},{\mathsf{CT}}^{t}_{y}\\}$

18: for each $n\in{0,...,N-1}$ do

19: for each $p\in{0,...,F_{n}-1}$ do

20: idx = $\sum_{i=0}^{n-1}F_{i}+p$

21:
$\mathbold{res}[\textsf{idx}]=\mathsf{qMIFE}.\mathsf{Dec}({\mathsf{CT}}^{t},\mathsf{qMIFE}.\mathsf{SK}^{t}_{\mathbold{c}_{n,p}})$

22: end for

23: end for

24: $\nabla L(\mathbold{w}^{t})=-\frac{2}{S}\mathbold{res}+\lambda\nabla
R(\mathbold{w}^{t})$

25: $\mathbold{w}^{t+1}=\mathbold{w}^{t}-\alpha\nabla L(\mathbold{w}^{t})$

26:end procedure

27:procedure Training-Client($\mathbold{X}_{i}^{t}$)

28: function obtain-ct-from-client()

29: $\mathsf{qMIFE}.\mathsf{EK}_{i}^{t}$= obtain-ek-from-TTP()

30: $\mathbold{x}_{i}^{t}:=\textsf{vec}({\mathbold{X}_{i}^{t}})$

31: if party $i$ has label $\mathbold{y}^{t}$ then

32:
$\mathsf{CT}_{i}^{t}:=\mathsf{qMIFE}.\mathsf{Enc}(\mathsf{qMIFE}.\mathsf{EK}^{t}_{i},\mathbold{x}_{i}^{t})$

33:
$\mathsf{CT}_{y}^{t}:=\mathsf{qMIFE}.\mathsf{Enc}(\mathsf{qMIFE}.\mathsf{EK}_{i}^{t},\mathbold{y}^{t})$

34: Return $(\mathsf{CT}^{t}_{i},\mathsf{CT}_{y}^{t})$ to Aggregator

35: else

36:
$\mathsf{CT}_{i}^{t}:=\mathsf{qMIFE}.\mathsf{Enc}(\mathsf{qMIFE}.\mathsf{EK}_{i}^{t},\mathbold{x}_{i}^{t})$

37: Return $\mathsf{CT}^{t}_{i}$ to Aggregator

38: end if

39: end function

40:end procedure

41:procedure Training-TTP($1^{\lambda},1^{N}$)

42: function obtain-ek-from-TTP()

43:
$(\mathsf{PP}^{t},\\{\mathsf{EK}_{i}\\}^{t}_{i},\mathsf{MSK}^{t})\leftarrow\mathsf{qMIFE}.\mathsf{Setup}(1^{\lambda},1^{N})$

44: Deliver $\mathsf{qMIFE}.\mathsf{EK}^{t}_{i}$ to party $i$, $i\in[N]$

45: end function

46: function obtain-dk-from-TTP($\mathcal{C}^{t}$)

47: for each $i\in{0,...,N-1}$ do

48: for each $p\in{0,...,F_{i}}$ do

49:
$\mathsf{qMIFE}.\mathsf{KeyGen}(\mathsf{qMIFE}.\mathsf{MSK}^{t},\mathbold{c}_{i,p}^{t})\rightarrow\mathsf{qMIFE}.\mathsf{SK}^{t}_{\mathbold{c}_{i,p}}$

50: end for

51: end for

52: Return
$\\{\\{\mathsf{qMIFE}.\mathsf{SK}^{t}_{\mathbold{c}_{i,p}}\\}_{p=0}^{F_{i}}\\}_{i=0}^{N-1}$

53: end function

54:end procedure

###  IV-A Construction of Function Vectors

Our goal is to compute the gradient $g({\mathbold{w})}$ (Equation (2)). For
simplicity, we drop the superscript $t$ in our discussion. We define
$\mathbold{x}=[\mathbold{x}_{0}||...||\mathbold{x}_{N-1}||\mathbold{y}]$ where
$\mathbold{y}$ is the label vector and $\mathbold{x}_{i}$ is the vectorized
$\mathbold{X}_{i}$. Recall $g(\mathbold{w})$ is a vector of length $F$, where
$F$ is the total number of features. The key insight is that for the
$f^{\text{th}},f\in[F]$ element in (2), we construct a function vector
$\mathbold{c}_{f}$ based on weights $\mathbold{w}$ of current iteration, such
that
$g(\mathbold{w})[f]=-\frac{2}{S}\left\langle\mathbold{c}_{f},\mathbold{x}\otimes\mathbold{x}\right\rangle$.
Then the aggregator concatenates $g(\mathbold{w})[f],f\in[F]$ to obtain the
gradients.

For simplicity, we define the following:

$\displaystyle\mathbold{z}_{i}=\mathbold{u}^{\top}\mathbold{X}_{i},~{}~{}\mathbold{b}_{j}^{i}=\mathbold{w}_{j}^{\top}\mathbold{X}_{j}^{\top}\mathbold{X}_{i},~{}~{}\mathbold{b}_{y}^{i}=\mathbold{y}^{\top}\mathbold{X}_{i}.$
(4)

Now we decompose $g(\mathbold{w})$ and $\mathbold{x}\otimes\mathbold{x}$ to
reduce the assignment. Note that to compute $g(\mathbold{w})$ it suffices to
compute $\mathbold{z}_{0},...,\mathbold{z}_{N-1}$ as in (3). Here we define a
set of function vectors $\mathcal{C}=\\{\mathcal{C}_{i}\\}_{i=0}^{N-1}$ and
$\mathcal{C}_{i}=\\{\mathbold{c}_{i,p}\\}_{p=0}^{F_{i}-1}$, where
$\mathcal{C}_{i}$ is a subset of function vectors that are used to compute
elements in $\mathbold{z}_{i}$, $\mathbold{c}_{i,p}$ is the function vector to
compute $p$th element of $\mathbold{z}_{i}$ as in (5).

$\displaystyle\mathbold{z}_{i}[p]=\langle\mathbold{c}_{i,p},\mathbold{x}\otimes\mathbold{x}\rangle$
(5)

We construct $\mathbold{c}_{i,p}$ block by block according to the
decomposition of $\mathbold{x}\otimes\mathbold{x}$. Consider dividing
$\mathbold{x}\otimes\mathbold{x}$ into $N+1$ blocks as in the middle of Figure
1.

Figure 1: Decomposition of
$\langle\mathbold{c}_{i,p},\mathbold{x}\otimes\mathbold{x}\rangle$

Since in the computation of $\mathbold{z}_{i}$, only the component
$\mathbold{x}_{i}\otimes\mathbold{x}$ is required, we set 0 vector of the
corresponding lengths as the coefficients of the blocks
$\\{\mathbold{x}_{j}\otimes\mathbold{x}\\}_{j=0}^{N}$ if $j\neq i$. We design
$\mathbold{a}_{i,p}$ to make
$\mathbold{z}_{i}[p]=\langle\mathbold{a}_{i,p},\mathbold{x}_{i}\otimes\mathbold{x}\rangle$.

Let $\textsf{D}^{f}_{i}$ denotes the $f$th column of $\mathbold{X}_{i}$. Thus
$\mathbold{x}_{i}=[\textsf{D}_{i}^{0};...;\textsf{D}_{i}^{F_{i}-1}]$. Note
that in the computation of $\mathbold{z}_{i}[p]$ only the column
$\textsf{D}_{i}^{p}$ is used, thus we set 0 vector as the coefficients of
$\\{\textsf{D}_{i}^{q}\otimes\mathbold{x}\\}_{q=0}^{F_{i}-1}$ if $q\neq p$.
Hence we can express
$\langle\mathbold{a}_{i,p},\mathbold{x}_{i}\otimes\mathbold{x}\rangle=\langle\mathbold{d},\textsf{D}_{i}^{p}\otimes\mathbold{x}\rangle$.
We design $\mathbold{d}$ to achieve
$\mathbold{z}_{i}[p]=\langle\mathbold{d},\textsf{D}_{i}^{p}\otimes\mathbold{x}\rangle$.

From (2), (3) and (4) note that we can express:

$\displaystyle\mathbold{z}_{i}[p]=\sum_{j=0}^{N-1}-\mathbold{b}_{j}^{i}[p]+\mathbold{b}_{y}^{i}[p]$
(6)

where we have $\mathbold{z}_{i}[p]=(\mathbold{u}^{\top}\mathbold{X}_{i})[p]$,
$\mathbold{b}_{j}^{i}[p]=(\mathbold{w}_{j}^{\top}\mathbold{X}_{j}^{\top}\mathbold{X}_{i})[p]$,$\mathbold{b}_{y}^{i}[p]=(\mathbold{y}^{\top}\mathbold{X}_{i})[p]$.
We expand $\mathbold{b}_{j}^{i}[p]$ by first performing multiplication term-
by-term and then summing products as in (7). Based on this equation, we
determine the method for constructing the coefficients vector $\mathbold{d}$.

$\displaystyle\mathbold{b}_{j}^{i}[p]$
$\displaystyle=(\mathbold{w}_{j}^{\top}\mathbold{X}_{j}^{\top}\mathbold{X}_{i})[p]=(\mathbold{w}_{j}^{\top}\mathbold{X}_{j}^{\top})\textsf{D}_{i}^{p}$
$\displaystyle=\left[\sum_{f=0}^{F_{j}-1}\mathbold{w}_{j}^{\top}[f]\textsf{D}_{i}^{f}[0]\mathbin{\|}...\mathbin{\|}\sum_{f=0}^{F_{j}-1}\mathbold{w}_{j}^{\top}[f]\textsf{D}_{i}^{f}[S-1]\right]\textsf{D}_{i}^{p}$
$\displaystyle=\sum_{s=0}^{S-1}\sum_{f=0}^{F_{j}-1}\mathbold{w}_{j}[f]\textsf{D}_{j}^{f}[s]\textsf{D}_{i}^{p}[s]$
(7)

Now the goal is to construct $\mathbold{d}$ such that
$\mathbold{z}_{i}[p]=\langle\mathbold{d},\textsf{D}_{i}^{p}\otimes\mathbold{x}\rangle$.
We keep decomposing $\mathbold{d}$ and $\textsf{D}_{i}^{p}\otimes\mathbold{x}$
to blocks as in Figure 2.

Figure 2: Decomposition of
$\langle\mathbold{d},\textsf{D}_{i}^{p}\otimes\mathbold{x}\rangle$

In Figure 2, blocks with the same color will be designed to compute the
corresponding term on the right side of Equation (6). Considering
$\mathbold{d}_{s,j},s\in[S]$, we can set the following relations:

$\displaystyle\mathbold{b}_{j}^{i}[p]=\sum_{s=0}^{S-1}\langle\mathbold{d}_{s,j},\textsf{D}_{i}^{p}[s]\mathbold{x}_{j}\rangle$
(8)
$\displaystyle\mathbold{b}_{y}^{i}[p]=\sum_{s=0}^{S-1}\langle\mathbold{d}_{s,N},\textsf{D}_{i}^{p}[s]\mathbold{y}\rangle$
(9)

Next, we introduce the approach to construct $\mathbold{d}_{s,j},j\in[N]$
according to the corresponding weight piece. We remove the outer summation in
(7) and (8) to obtain:

$\displaystyle\sum_{f=0}^{F_{j}-1}\mathbold{w}_{j}[f]\textsf{D}_{j}^{f}[s]\textsf{D}_{i}^{p}[s]=\langle\mathbold{d}_{s,j},\textsf{D}_{i}^{p}[s]\mathbold{x}_{j}\rangle$
(10)

We design $\mathbold{d}_{s,j}$ to achieve (10) as Figure 3 shows. We decompose
$\langle\mathbold{d}_{s,j},\textsf{D}_{i}^{p}[s]\mathbold{x}_{j}\rangle$ into
blocks $\textsf{D}_{i}^{p}[s]\textsf{D}_{j}^{f},f\in[F_{j}]$. In each block
$\textsf{D}_{i}^{p}[s]\textsf{D}_{j}^{f}$, we take one entry
$\textsf{D}_{i}^{p}[s]\textsf{D}_{j}^{f}[s]$ and set its coefficient to
$\mathbold{w}_{j}[f]$ just as the left side of (10). For other unneeded terms,
we set the coefficient to 0.

Figure 3: Decomposition of
$\langle\mathbold{d}_{s,j},\textsf{D}_{i}^{p}[s]\mathbold{x}_{j}\rangle$

The approach to construct $\mathbold{d}_{s,j},j\in[N]$ can be easily extended
to the case for $\mathbold{d}_{s,N}$. By designing the blocks of
$\mathbold{d}$ this way, we can achieve
$\mathbold{z}_{i}[p]=\langle\mathbold{d},\textsf{D}_{i}^{p}\otimes\mathbold{x}\rangle$.
The algorithms to construct the function vectors $\mathbold{c}_{i,p}$ and $d$
are provided in Appendix B.

###  IV-B Privacy Analysis

Recall the aim of our framework. We want the client $i$ and the aggregator to
learn nothing about data $\mathbold{X}_{j}$ of client $j$ for $i\neq j$. We
also want that any client should learn nothing about the trained global model
weights $\mathbold{w}$, intermediate results including error between labels
and feed-forward output as in (1) and the gradient $g(\mathbold{w})$.
Moreover, $i$th client should not learn anything about his/her own
corresponding weights $\mathbold{w}_{i}$. In this section, we prove that we
have achieved the above-stated goal.

######  Theorem IV.1.

If Quadratic MIFE $(\mathsf{qMIFE})$ is secure according to definition 2, then
in each training iteration $t$, $i$th client’s data $\mathbold{X}_{i}^{t}$ for
$i\in[N]$ is hidden from client $j$ and the aggregator, trained global model
weights $\mathbold{w}^{t}$ and intermediate results $\mathbold{u}^{t}$ as in
(1) are hidden from the clients and $i$th client learns nothing about weight
$\mathbold{w}_{i}$.

Let us fix the iteration number to be $t$. In each iteration, the TTP runs the
$\mathsf{qMIFE}.\mathsf{Setup}$ algorithm to get public parameters
$\mathsf{PP}^{t}$, $N$ encryption keys $\\{\mathsf{EK}_{i}\\}_{i\in[N]}^{t}$
and a master secret key $\mathsf{MSK}^{t}$. The clients encrypt their
respective data and send the ciphertexts to the aggregator. The aggregator
asks the TTP for the secret key corresponding to the set of vectors
$\mathcal{C}^{t}$. Each vector $\mathbold{c}_{i,p}^{t}$ is set in such a way
that the $\mathsf{qMIFE}.\mathsf{Dec}$ only reveals the inner product
$\langle\mathbold{c}_{i,p}^{t},\mathbold{x}\otimes\mathbold{x}\rangle=((\mathbold{u}^{t})^{\top}\mathbold{X}_{i}^{t})[p]$.
Quadratic MIFE ensures that nothing about $\mathbold{X}_{i}^{t}$ and
$\mathbold{u}^{t}$ is revealed to the aggregator. Moreover, each client
encrypts their data using different encryption keys. The ciphertexts are
indistinguishable; hence, clients cannot predict other clients’ data.

Unlike FedV, in our framework, the client runs the
$\mathsf{qMIFE}.\mathsf{Enc}$ algorithm which only takes their respective data
and encryption keys as input. Hence, each client $i$ learns nothing about
their respective weight $\mathbold{w}_{i}^{t}$. Moreover, Quadratic MIFE
ensures that client $i$ learns nothing about the global weight
$\mathbold{w}^{t}$. The aggregator does not share the gradients in any form
with the clients. Therefore, the gradient $g(\mathbold{w}^{t})$ is also not
revealed.

Importance of using new $\mathsf{qMIFE}$ instance for each iteration. Let in
the iteration $t$, the ciphertext be $\mathsf{CT}^{t}$ and secret key be
$\mathsf{qMIFE}.\mathsf{SK}^{t}$. Suppose the TTP uses the same $\mathsf{MSK}$
to generate secret keys $\mathsf{qMIFE}.\mathsf{SK}^{t+1}$ for some other
iteration, say $t+1$, then the aggregator may use
$\mathsf{qMIFE}.\mathsf{SK}^{t+1}$ to decrypt the ciphertext $\mathsf{CT}^{t}$
instead of using it to decrypt the ciphertext $\mathsf{CT}^{t+1}$. Using this
"mix-and-match" attack by performing decryptions of secret key and ciphertexts
from different iterations, he will know
$g(\mathbold{w})=-\frac{2}{S}\begin{bmatrix}(\mathbold{u}^{t+1})^{\top}\mathbold{X}_{0}^{t}||...||(\mathbold{u}^{t+1})^{\top}\mathbold{X}_{N-1}^{t}\end{bmatrix}$.

If TTP generates different $\mathsf{qMIFE}$ instance for every iteration, then
decryption of $\mathsf{CT}^{t}$ with secret key
$\mathsf{qMIFE}.\mathsf{SK}^{t+1}$ will give some garbage value which will be
irrelevant for the aggregator. Therefore, it is important for the TTP to
generate a new $\mathsf{qMIFE}$ instance for every iteration.

Comparison of our framework with FedV. Unlike FedV, our framework does not
leak the intermediate result $\mathbold{u}$ to the aggregator. The global
weights $\mathbold{w}$ are kept secret from the clients and each client also
learns nothing about their respective weights. In addition to this, we also
ensure that the aggregator cannot use the mix-and-match attack to learn some
useful information.

###  IV-C Efficiency Analysis

Communication. Regarding communication complexity, SFedV requires one-way
client-aggregator communication, while FedV needs one-round client-aggregator
communication due to the delivery of global weights by the aggregator.
Additionally, SFedV uses a new $\mathsf{qMIFE}$ instance in each iteration to
prevent mix-and-match attacks. Thus, an increase of communication between TTP
and clients becomes necessary. Note that FedV can also prevent mix-and-match
attacks by using new instances of $\mathsf{MIFE}$ and $\mathsf{SIFE}$ in each
iteration. In such a scenario, the client-TTP communication complexity for
each iteration will be the same for both FedV and SFedV.

TABLE I: Comparison of FedV and SFedV regarding the number of encryption
processes on each client and the number of decryption processes on the
aggregator in each iteration.

| FedV | SFedV
---|---|---
| $\mathsf{MIFE}$ | $\mathsf{SIFE}$ | $\mathsf{qMIFE}$
Encryptions on each client | $S$ | $S\cdot F_{i}$ | $1$
Decryptions on the aggregator | $S$ | $F$ | $F$

  * •

$S$: Batch size. $F$: Total number of features. $F_{i}$: Number of features
belonging to client $i$.

Computation. Table I provides a comparison between FedV and SFedV in terms of
the number of encryption and decryption processes in each iteration. The
significant improvement of SFedV is attributed to the advancement of quadratic
$\mathsf{MIFE}$ and the careful design of function vectors.

In terms of the number of the vector corresponding to which the secret keys
are generated, FedV uses two vectors for two steps: $\mathbold{v}$ for feature
dimension secure aggregation and $\mathbold{u}$ for sample dimension secure
aggregation. In contrast, our SFedV framework employs $F$ vectors
$\mathbold{c}$, where $F$ is the total number of features. The increase in
size can be justified by our use of a quadratic $\mathsf{MIFE}$ scheme instead
of inner product $\mathsf{MIFE}$.

##  V Conclusions

Prior $N$-party VFL framework FedV incurs information leakage which seriously
undermines individual data privacy. In this work, to address the privacy
issues, We propose a leak-free protocol, called SFedV, for multiparty VFL
regression model training. Our approach simplifies the VFL pipeline and
preserves the privacy of client data, model weights, and intermediate results,
by designing special function vectors and using a quadratic MIFE scheme to
compute gradients directly.

## References

  * [1] R. Xu, N. Baracaldo, Y. Zhou, A. Anwar, J. Joshi, and H. Ludwig, “Fedv: Privacy-preserving federated learning over vertically partitioned data,” 03 2021\.
  * [2] P. Paillier, “Public-key cryptosystems based on composite degree residuosity classes,” in _Advances in Cryptology—EUROCRYPT’99: International Conference on the Theory and Application of Cryptographic Techniques Prague, Czech Republic, May 2–6, 1999 Proceedings 18_. Springer, 1999, pp. 223–238.
  * [3] A. Acar, H. Aksu, A. S. Uluagac, and M. Conti, “A survey on homomorphic encryption schemes: Theory and implementation,” _ACM Computing Surveys (Csur)_ , vol. 51, no. 4, pp. 1–35, 2018.
  * [4] M. Mohamad, M. Önen, W. Ben Jaballah, and M. Conti, “Sok: Secure aggregation based on cryptographic schemes for federated learning,” in _PETS 2023, 23rd Privacy Enhancing Technologies Symposium, 10-14 July 2023, Lausanne, Switzerland (Hybrid Conference)_ , Lausanne, 2023.
  * [5] S. Hardy, W. Henecka, H. Ivey-Law, R. Nock, G. Patrini, G. Smith, and B. Thorne, “Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption,” 11 2017.
  * [6] K. Yang, T. Fan, T. Chen, Y. Shi, and Q. Yang, “A quasi-newton method based vertical federated learning framework for logistic regression,” _ArXiv_ , vol. abs/1912.00513, 2019.
  * [7] S. Yang, B. Ren, X. Zhou, and L. Liu, “Parallel distributed logistic regression for vertical federated learning without third-party coordinator,” _arXiv preprint arXiv:1911.09824_ , 2019.
  * [8] H. Sun, Z. Wang, Y. Huang, and J. Ye, “Privacy-preserving vertical federated logistic regression without trusted third-party coordinator,” in _2022 The 6th International Conference on Machine Learning and Soft Computing_ , ser. ICMLSC 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 132–138. [Online]. Available: https://doi.org/10.1145/3523150.3523171
  * [9] D. He, R. Du, S. Zhu, M. Zhang, K. Liang, and S. Chan, “Secure logistic regression for vertical federated learning,” _IEEE Internet Computing_ , vol. 26, no. 2, pp. 61–68, 2021.
  * [10] D. Zhao, M. Yao, W. Wang, H. He, and X. Jin, “Ntp-vfl - a new scheme for non-3rd party vertical federated learning,” in _2022 14th International Conference on Machine Learning and Computing (ICMLC)_ , ser. ICMLC 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 134–139. [Online]. Available: https://doi.org/10.1145/3529836.3529841
  * [11] X. Yu, W. Zhao, D. Tang, K. Liang, and J. Du, “Privacy-preserving vertical collaborative logistic regression without trusted third-party coordinator,” _Sec. and Commun. Netw._ , vol. 2022, jan 2022. [Online]. Available: https://doi.org/10.1155/2022/5094830
  * [12] Q. Li, Z. Huang, W.-j. Lu, C. Hong, H. Qu, H. He, and W. Zhang, “Homopai: A secure collaborative machine learning platform based on homomorphic encryption,” in _2020 IEEE 36th International Conference on Data Engineering (ICDE)_ , 2020, pp. 1713–1717.
  * [13] S. Goldwasser, S. D. Gordon, V. Goyal, A. Jain, J. Katz, F.-H. Liu, A. Sahai, E. Shi, and H.-S. Zhou, “Multi-input functional encryption,” in _Annual International Conference on the Theory and Applications of Cryptographic Techniques_. Springer, 2014, pp. 578–602.
  * [14] A. Sahai and B. Waters, “Fuzzy identity-based encryption,” in _Advances in Cryptology – EUROCRYPT 2005_ , R. Cramer, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005, pp. 457–473.
  * [15] D. Boneh and B. Waters, “Conjunctive, subset, and range queries on encrypted data,” in _Proceedings of the 4th Conference on Theory of Cryptography_ , ser. TCC’07. Berlin, Heidelberg: Springer-Verlag, 2007, p. 535–554.
  * [16] D. Boneh, A. Sahai, and B. Waters, “Functional encryption: Definitions and challenges,” in _Theory of Cryptography_ , Y. Ishai, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 253–273.
  * [17] S. Agrawal, R. Goyal, and J. Tomida, “Multi-input quadratic functional encryption from pairings,” in _Crypto_ , 2021, https://ia.cr/2020/1285.
  * [18] ——, “Multi-input quadratic functional encryption: Stronger security, broader functionality,” in _Theory of Cryptography_ , E. Kiltz and V. Vaikuntanathan, Eds. Cham: Springer Nature Switzerland, 2022, pp. 711–740.
  * [19] M. Abdalla, F. Benhamouda, and R. Gay, “From single-input to multi-client inner-product functional encryption,” in _Advances in Cryptology – ASIACRYPT 2019_ , S. D. Galbraith and S. Moriai, Eds. Cham: Springer International Publishing, 2019, pp. 552–582.
  * [20] B. Libert and R. ŢiŢiu, “Multi-client functional encryption for linear functions in the standard model from lwe,” in _Advances in Cryptology – ASIACRYPT 2019_ , S. D. Galbraith and S. Moriai, Eds. Cham: Springer International Publishing, 2019, pp. 520–551.

##  Appendix A Security Definition for MIFE

In an indistinguishability-based security game between a challenger and an
adversary, the challenger runs the $\mathsf{Setup}$ algorithm to generate the
public parameters $\mathsf{PP}$, $N$ encryption keys $\mathsf{EK}_{i}$, and
master secret key $\mathsf{MSK}$. The adversary then chooses the set of
encryption keys that she wants. Then the adversary chooses two messages
$\mathbold{x}^{0}$ and $\mathbold{x}^{1}$ and gives them to the challenger.
The challenger chooses a bit $\beta$ at random and encrypts the message
$\mathbold{x}^{\beta}$ using the $i^{\text{th}}$ encryption key
$\mathsf{EK}_{i}$ to get challenge ciphertext $\mathsf{CT}_{i}$. The adversary
then asks the challenger for the secret keys corresponding to the functions
$f$. At last, the adversary guesses a bit $\beta^{\prime}$ and replies to the
challenger. The admissible adversary wins if $\beta^{\prime}=\beta$. In
security, we want the probability of the adversary winning the security game
to be negligibly close to 1/2.

The adversary is said to be admissible if and only if she sends at least one
element of the form $(i,*,*)$ in the message space and she queries the secret
key for the function $f$ which satisfies the constraint that
$f(\mathbold{x}^{0})=f(\mathbold{x}^{1})$. We formally define MIFE security in
Definition 2.

######  Definition 2 (MIFE Security [18]).

An $\mathsf{MIFE}$ scheme is IND-secure if for any stateful admissible PPT
adversary $\mathcal{A}$, there exists a negligible function
$\mathsf{negl}(\cdot)$ such that for all $\lambda,N\in\mathbb{N}$, the
following probability is negligibly close to 1/2 in $\lambda$:

$\displaystyle\Pr\left[\beta^{\prime}=\beta:\begin{array}[]{l}\beta\leftarrow\\{0,1\\}\\\
(\mathsf{PP},\\{\mathsf{EK}_{i}\\}_{i\in[N]},\mathsf{MSK})\leftarrow\\\
\mathsf{Setup}(1^{\lambda},1^{N})\\\
(\mathcal{CS},\mathcal{MS},\mathcal{FS})\leftarrow\mathcal{A}(1^{\lambda},\mathsf{PP})\text{
s.t. }\\\ \mathcal{CS}\subseteq[N]\\\
\mathcal{MS}=\\{i^{\mu},\mathbold{x}^{\mu,0},\mathbold{x}^{\mu,1}\\}_{\mu\in[q_{c}]}\\\
\mathcal{FS}=\\{f^{v}\\}_{v\in[q_{k}]}\\\
\\{\mathsf{CT}_{\mu}\leftarrow\mathsf{Enc}(\mathsf{EK}_{i^{\mu}},\mathbold{x}^{\mu,\beta})\\}_{\mu}\\\
\\{\mathsf{SK}_{v}\leftarrow\mathsf{KeyGen}(\mathsf{MSK},f^{v})\\}_{v}\\\
\beta^{\prime}\leftarrow\mathcal{A}\left(\\{\mathsf{EK}_{i}\\}_{i\in\mathcal{CS}},\\{\mathsf{CT}_{\mu}\\}_{\mu},\\{\mathsf{SK}_{v}\\}_{v}\right)\end{array}\right]$

where the adversary $\mathcal{A}$ is said to be admissible if and only if

  * •

$q_{c}[i]>0$ for all $i\in[N]$, where $q_{c}[i]$ denotes the number of
elements of the form $(i,*,*)$ in $\mathcal{MS}$.

  * •

$f(\mathbold{x}_{1}^{0},\ldots,\mathbold{x}_{n}^{0})=f(\mathbold{x}_{1}^{1},\ldots,\mathbold{x}_{n}^{1})$
for all sequences
$(\mathbold{x}_{1}^{0},\ldots,\mathbold{x}_{n}^{0},\mathbold{x}_{1}^{1},\ldots,\mathbold{x}_{n}^{1},f)$
such that:

    * –

For all $i\in[n]$,
$[(i,\mathbold{x}_{i}^{0},\mathbold{x}_{i}^{1})\in\mathcal{MS}]$ or
$[i\in\mathcal{CS}\text{ and }\mathbold{x}_{i}^{0}=\mathbold{x}_{i}^{1}]$,

    * –

$f\in\mathcal{FS}$.

##  Appendix B Pseudocode

In this section, we give reference to the algorithms used for constructing the
function vectors. The algorithm 2 shows the construction of
$\mathbold{c}_{i,p}$ and the algorithm 3 shows the construction of vector $d$.

Algorithm 2 Construct c

1:function cgen($\mathbold{w},S,F,N,i,p$)

2: $\mathcal{A}=\\{\mathbold{a}_{i,n}\\}_{n=0}^{N}$

3: for each $n\in{0,...,N}$ do

4: if $n\neq i$ and $i\neq N$ then

5: $\mathbold{a}_{i,n}=\textbf{0}^{S^{2}(F+1)F_{i}}$

6: else if $n==i$ and $i\neq N$ then

7: $\mathcal{D}=\\{\mathbold{d}_{i}\\}_{i=0}^{F_{i}-1}$

8: for each $f\in{0,...,F_{i}-1}$ do

9: if $f==p$ then

10: $\mathbold{d}_{f}=d=$ SUBCGEN($\mathbold{w},S,F_{j},N$)

11: else if $f\neq p$ then

12: $\mathbold{d}_{f}=\textbf{0}^{S^{2}(F+1)}$

13: end if

14: $\mathbold{a}_{i,n}=[\mathbold{d}_{0};\ldots;\mathbold{d}_{F_{i}-1}]$

15: end for

16: else if $i==N$ then

17: $\mathbold{a}_{i,n}=\textbf{0}^{S^{2}(F+1)}$

18: end if

19: end for

20: Set $\mathbold{c}_{i,p}=[\mathbold{a}_{i,0};\ldots;\mathbold{a}_{i,N}]$

21: return $\mathbold{c}_{i,p}$

22:end function

Algorithm 3 Construct subc

1:function subcgen($\mathbold{w},S,F_{j},N$)

2: $d=\\{\mathbold{d}_{s,n}\\},s\in[S],n\in[N+1]$

3: for each $s\in{0,...,S-1}$ do

4: for each $j\in{0,...,N}$ do

5: if $j\neq N$ then

6: $\mathbold{d}_{s,j}=\textbf{0}^{SF_{j}}$ $\triangleright$ Initialization

7: for each $f\in{0,...,F_{j}-1}$ do

8: $\mathbold{d}_{s,j}[fS+s]=\mathbold{w}_{j}[f]$

9: end for

10: else if $j==N$ then

11: $\mathbold{d}_{s,N}=\textbf{0}^{S}$ $\triangleright$ Initialization

12: $\mathbold{d}_{s,N}[s]=1$

13: end if

14: end for

15: $\mathbold{d}_{s}=[\mathbold{d}_{s,0};\ldots;\mathbold{d}_{s,N}]$

16: end for

17: return $\mathbold{d}=[\mathbold{d}_{0};\ldots;\mathbold{d}_{S-1}]$

18:end function

##  Appendix C Extension to Logistic Regression Model

In this section, we extend the protocol to work for logistic regression with
the help of Taylor approximation. The prediction function of logistic models
is as follows:

$\displaystyle
f(\mathbold{x},\mathbold{w})=\frac{1}{1-e^{-\mathbold{x}\mathbold{w}}}$ (11)

We use Cross-Entropy as the loss function for logistic regression. The loss
function in the vectorized form is

$\displaystyle\footnotesize
L(\mathbold{w})=\frac{1}{s}\left[-\mathbold{y}^{\top}\log(f(\mathbold{X},\mathbold{w}))-(\textbf{1}-\mathbold{y})^{\top}\log(\textbf{1}-f(\mathbold{X},\mathbold{w}))\right]$

Here we use Taylor approximation to make the loss function polynomial. In [5],
it takes a Taylor Series expansion of $\log(1+e^{-z})$ around $z=0$.

$\displaystyle\log(1+e^{-z})=\log{2}-\frac{1}{2}z+\frac{1}{8}z^{2}-\frac{1}{192}z^{4}+O(z^{6})$
(13)

We apply (13) to (C) and get the gradients of vector-format expression as
given below.

$\displaystyle
g(\mathbold{w})\approx\frac{1}{S}\left(\frac{1}{4}\mathbold{X}\mathbold{w}-\mathbold{y}+\frac{1}{2}\right)^{\top}\mathbold{X}$
(14)

Then we decompose
$\mathbold{X}\mathbold{w}=\mathbold{X}_{0}\mathbold{w}_{0}+\mathbold{X}_{1}\mathbold{w}_{1}+...+\mathbold{X}_{N-1}\mathbold{w}_{N-1}$
and
$\mathbold{X}=[\mathbold{X}_{0}||\mathbold{X}_{1}||...||\mathbold{X}_{N-1}]$
and substitute the decomposition into (14) to get

$\displaystyle g(\mathbold{w})\approx$
$\displaystyle\frac{1}{S}\begin{bmatrix}-(\mathbold{y}-\frac{1}{2})^{\top}\mathbold{X}_{0}+\frac{1}{4}\sum_{j=0}^{N-1}\mathbold{w}_{j}^{\top}\mathbold{X}_{j}^{\top}\mathbold{X}_{0}||\\\
...||\\\
-(\mathbold{y}-\frac{1}{2})^{\top}\mathbold{X}_{N-1}+\frac{1}{4}\sum_{j=0}^{N-1}\mathbold{w}_{j}^{\top}\mathbold{X}_{j}^{\top}\mathbold{X}_{N-1}\end{bmatrix}$
(15)

Each term in (15) has the similar format to the term in (2) except for
$\mathbold{w}_{j}^{\top}\mathbold{X}_{j}^{\top}\mathbold{X}_{i}$ in (15) has
coefficient $\frac{1}{4}$ and $\mathbold{y}^{\top}\mathbold{X}_{i}$ in (2)
becomes $(\mathbold{y}-\frac{1}{2})^{\top}\mathbold{X}_{i}$ in (15). We can
modify the protocol to compute the gradients for non-linear models without
exposing labels $\mathbold{y}$ and $\mathbold{X}_{i}$. First, when the
aggregator constructs function vector $\mathbold{c}$, instead of using the
original weights, we multiply the weights with $\frac{1}{4}$ element-wise and
use the modified weights to construct $\mathbold{c}$. Moreover, the active
party, instead of sending the ciphertext of $\mathbold{y}$, now sends the
ciphertext of $\mathbold{y}-\frac{1}{2}$. After decrypting and concatenating
all the elements in (15), the aggregator multiplies the concatenated results
with $\frac{1}{S}$ to obtain the gradients. Other procedures remain same as
the procedure for the linear regression models.


# Multimodal Approach for Metadata Extraction from German Scientific
Publications

Azeddine Bouabdallah Jorge Gavilan Jennifer Gerbl Prayuth Patumcharoenpol
bazeddine, jorge, jgerbl<EMAIL_ADDRESS>Institute for Web Science
and Technologies (WeST)
University of Koblenz-LandauKoblenzGermany

###### Abstract.

Nowadays, metadata information is often given by the authors themselves upon
submission. However, a significant part of already existing research papers
have missing or incomplete metadata information. German scientific papers come
in a large variety of layouts which makes the extraction of metadata a non-
trivial task that requires a precise way to classify the metadata extracted
from the documents. In this paper, we propose a multimodal deep learning
approach for metadata extraction from scientific papers in the German
language. We consider multiple types of input data by combining natural
language processing and image vision processing. This model aims to increase
the overall accuracy of metadata extraction compared to other state-of-the-art
approaches. It enables the utilization of both spatial and contextual features
in order to achieve a more reliable extraction. Our model for this approach
was trained on a dataset consisting of around 8800 documents and is able to
obtain an overall F1-score of 0.923.

natural language processing, computer vision, metadata extraction, deep
learning, biLSTM, classification, multimodality

††conference: The 1st Workshop on Bibliographic Data Analysis and Processing;
August 14–18, 2021; ††ccs: Computing methodologies Information
extraction††ccs: Computing methodologies Computer vision representations††ccs:
Computing methodologies Natural language processing††ccs: Applied computing
Document metadata††ccs: Computing methodologies Supervised learning by
classification

##  1\. Introduction

In recent decades, metadata extraction has become an important task in many
fields. This importance arises from the increase in digital content usage, as
more people are now used to obtaining information through the internet.
Consequently, it increased the need of providing easy access and retrieval of
information for users. Ensuring this requires the availability of contents’
metadata to help retrieval systems in finding relevant content. Digital
scholar libraries are one of the major parties needing to keep this easy
access constantly. Recent statistics estimated that there are around 1.8
million scientific publications each year (Ware and Mabe, 2015), most of which
are of a digital format. This high number of publications each year and past
publications introduce a new challenge in acquiring their metadata.

Currently, digital libraries are relying on manual input from authors upon
submission to acquire metadata. However, making this manual task for all past
publications can be time-consuming. In fact, it might be impossible
considering publications of all domains from past decades. As a result,
developing an approach that allows automating this process is highly needed.
Researchers have been constantly trying to solve this by proposing many
approaches, many of which have achieved reliable results. However, these
approaches were designed to tackle English scientific papers. The issue lies
with German scientific publications (Boukhers et al., 2021a). In contrast to
English papers which follow a limited number of templates, German papers have
a variety of different unseen templates. The latter introduces a new level of
challenge for metadata extraction to deal with irregular layouts.

Research efforts that tackled the issue of metadata extraction have always
relied on the natural language processing (NLP) methods to solve it, by either
using rule-based approaches (Romary and Lopez, 2015) or by introducing machine
learning (Alzaidy et al., 2019; An et al., 2017). Even though they were able
to achieve considerable results, the issue persisted with irregular layouts.
In 2021, a research work (Boukhers et al., 2021a) proposed a novel method
named MexPub. While common works used NLP to solve this issue, MexPub
(Boukhers et al., 2021a) treated it as an image-vision problem. Instead of
focusing on contextual features, they concentrate on visual features. This
allowed for more flexibility when dealing with irregular layouts which are
more common with German publications. Despite having improvements, MexPub
lacks the ability to take contextual features into considerations, which can
help in recognizing different classes such as emails, journals, and dates.

Previously, combining multiple types of inputs has proven to be conducive to
improve results for other fields. In this paper, we are proposing a multimodal
deep learning approach that can employ advantages from both worlds of NLP and
image-vision. The primary goal of this work is to emphasize on both contextual
and visual features of publications in an effort to extract metadata reliably
from German publications and increase the overall accuracy compared to
previous state-of-the-art approaches. The ultimate aim of this approach is to
extract segments of text which belong to one of the following classes: title,
abstract, author, journal, address, email, date, DOI, and affiliation.

In the following section, we outline the related works that addressed similar
problems and the solutions they proposed. We introduce our proposed approach
in detail in section 3, followed by a presentation of the experimental setup
as well as a discussion of our obtained evaluation results in section 4.
Finally, we conclude the paper with the conclusion and future work in section
5.

##  2\. Related work

###  2.1. NLP

According to Simões et al. (2009), numerous previous old studies on metadata
extraction relied on text and layout rules, addressing the issue using
context-based classifiers such as Hidden Markov Models (HMMs) and similar
approaches based on HMMs like Maximum Entropy Markov Models (MEMMs). However,
these methods have several issues. First, the evaluation results on English
papers were not reliable as many classes cannot be detected using only the
features HMMs relies on. Second, HMMs struggle when dealing with multiple non-
independent features because the states depend only on their immediate
predecessor. Finally, HMMs and MEMMs are conducive to a weakness known as the
label bias problem (Chaturvedi et al., 2018). One way of dealing with this
issue is to use Conditional Random Fields (CRF) as proposed by GROBID (Romary
and Lopez, 2015), which is a machine learning library for extracting text from
documents into a structured format such as XML. The approach relies on CRF in
order to extract bibliographical header metadata (e.g., title, abstract,
author, etc.). It is able to perform significantly well with the English
language documents but when introduced to newly unseen templates, it faces
challenges recognizing and extracting the metadata.

Usually, with the previous approaches, the document goes through a sequential
pipeline, mostly a two-phase process. Firstly, text segmentation and secondly
the classification into one of the desired classes. The problem is that this
type of approach is more perceptive to accumulated errors within the pipeline.
To overcome such an issue Boukhers et al. (2019) proposed an end-to-end
approach for reference segmentation and extraction from PDF documents. It
learns the different characteristics of references to be used in a coherent
scheme that reduces the error accumulation using a probabilistic approach.

###  2.2. Computer Vision

Computer vision is rarely used in document analysis. However, recent
experiments have shown that it is possible to obtain promising results with
novel approaches in this area. For instance, Stahl et al. (2018) proposed an
approach titled DeepPDF that treats PDF as an image to label its texts to
either ”paragraph” or ”non-paragraph” at a pixel-wise level. It uses the U-Net
and a CNN architecture for semantic segmentation to classify each segment as
“paragraphs“ or “non-paragraphs”.

Only a few approaches in the area of metadata extraction using deep learning
have been proposed. For example, two of which had impressive results, SLLD
from Yang and Hsu (2020) and MexPub from Boukhers et al. (2021a), proposed
approaches which rely on the R-CNN (Girshick et al., 2014) architecture for
object detection. SLLD implements Faster R-CNN (Ren et al., 2017) with
VoVNetV2-39 as a backbone (feature extraction and region selection) to detect
the main region of the layout (e.g., title, authors, abstract of the text)
while MexPub used Mask R-CNN (He et al., 2017) with ResNeXt (Xie et al., 2017)
as a backbone with Feature Pyramid Networks (FPN). One main difference between
SLLD and MexPub is the dataset. While MexPub’s dataset is focusing on German
literature, SLLD focuses more on English language documents. Both approaches
have shown promising results for layout detection and metadata extraction from
scientific literature. However, they still have some flaws. For example, SLLD
fails to detect non-rectangular regions while MexPub has difficulties in
precisely detecting small-sized patterns and detects multiple patterns
mistakenly in some cases which lead to false positives.

Figure 1.  Model Architecture
This figure illustrates the multimodal architecture for metadata extraction

###  2.3. Multimodality

Multimodal deep learning has been used previously in many domains including
audiovisual and image classification as it has shown it can improve the
overall performance by combining multiple types of inputs and making them work
together. Particular applications of multimodality in the field of metadata
extraction have proven better results in comparison to unimodal counterparts
as indicated by the works of Balasubramanian et al. (2016) and Liu et al.
(2018).

In the multimodal metadata extraction system for video lectures presented by
Balasubramanian et al. (2016), both audio and video modalities are considered.
This approach effectively utilizes the correlations between the audio
transcripts and the content in the slides embedded in the video streams, to
extract the defined metadata. A hybrid approach combining a Naive Bayes
classifier and a rule-based refiner is used for the effective retrieval of
metadata in a lecture video. By applying this multimodal methodology, in
comparison to the audio-based approach, an improvement of 114.2% is achieved
when comparing precision, recall, and F-score for several lecture videos.

Liu et al. (2018) was one of the firsts to introduce deep learning technology
to the problem of metadata extraction. The multimodal approach for this
extraction method can process both image data and text data as sources of
information and does not need to design any classification feature.

Employing deep learning networks to model the image and text information of
document headers respectively allowed to perform metadata extraction with
little information loss. This approach presented two types of networks
including Convolutional Neural Networks (CNNs) and Recurrent Neural Networks
(RNNs) to handle both types of data. In the last stage, they combined those
networks into a following Long-Short Term Memory (LSTM) to get the
classifications of each line in paper headers. The overall system performance
of this combined image and text model showed a great improvement after their
integration: The images carry features such as location, font, and layout
information, while the texts carry semantic and character arrangement
information, and the model integration reduces the information loss in the
metadata extraction process in general.

##  3\. Our approach

We can decompose our overall architecture into three submodels: an NLP model,
a computer vision model, and a classifier, as illustrated in Figure 1. The
following subsections describe the preprocessing steps and the detailed
architecture of each submodel.

###  3.1. Preprocessing

Preprocessing is a crucial phase in which the acquired data is transformed to
be the input for the model. For each of the documents, we extract the text of
the first page using CERMINE (Tkaczyk et al., 2015). This tool was chosen
because it is one of the most reliable ways to extract texts even for more
challenging layouts and it enabled us to gather information about the
geometric structure such as text position and font style in an easy manner. We
only consider the first page of a document since the metadata information we
are interested in typically exists within that scope.

After extracting the words of a document, each word runs through a layout and
context feature extraction process. The textual layout feature vector consists
of a set of 16 visual-spatial dependent features such as the horizontal or
vertical space, the relative font size, the relative line number, the count of
occurrences of specific special characters and multiple flags describing
whether the text is in italic, bold or follows a specific common format such
as date or email. These layout features on their own are not enough to
represent the relationship between words and their classes. For instance,
author names have a distinct meaning and usage compared to words that belong
to an abstract. Consequently, we decided that there is a need to encode the
context and the meaning of the words as well. To achieve this, we use the ELMO
model by Che et al. (2018) which encodes them into feature vectors of size
1025 to create a representation for each word. This ELMO model is based on the
work of Peters et al. (2018) and Gardner et al. (2018) and trained on the
German language, therefore it is better suited for our case compared to other
models that have been trained on other languages.

The output of both feature extraction processes is concatenated to form a
single feature vector of size 1041. This serves as input for the natural
language processing submodel.

In contrast to the natural language processing phase, the image-vision does
not rely on text but images. Each of the documents’ first pages will be
converted into an image which will be the input for the image-vision submodel.

###  3.2. NLP Submodel

According to Xu et al. (2019) BiDirectional Long-Short-Term Memory (biLSTM) is
more efficient in learning the context within sentences than Long-Short-Term
Memory (LSTM). Consequently, we have proceeded with the biLSTM as a model of
choice for the NLP submodel. biLSTM essentially is two LSTM layers stacked
together, as shown in Figure 2. The first layer is a forward LSTM and the
second is a backward LSTM. This allows the preservation of past and future
information for a better contextual understanding.

The biLSTM model takes as input a word embedding vector of length 1041 at each
time step. The embedding vector is composed of two parts, the first part of
length 16 contains layout features (font size of the word, font style, spacing
between the word and the line above/under, etc.) and the second part comprises
the ELMO (Che et al., 2018) embedding results.

Our implementation of the biLSTM model comprises 2 hidden layers: forward and
backward LSTM with 256 hidden dimensions each, then a fully connected layer of
512 input nodes, and 10 output nodes with a softmax activation function to
obtain probability scores for the word belonging to each of the 10 classes
(abstract, author, email, address, date, journal, affiliation, DOI, title,
unclassified).

Figure 2.  The NLP BiLSTM submodel architecture

The output of the biLSTM is of length 512 (each LSTM layer’s hidden dimension
is 256) is fed to a classifier (fully connected neural network) with a softmax
activation function that would result in an output of length 10, each output
node $i$ represents the probability of a word being class $i$.

###  3.3. Image vision Submodel

In the image vision submodel we use the pre-trained model from MexPub
(Boukhers et al., 2021a), which is implemented with Mask R-CNN-FPN. Mask R-CNN
is used for object instance segmentation and extended from Faster R-CNN. The
improvements from its predecessor are a faster trainability as well as better
overall results. By changing Region of Interest (RoI)-Pooling to RoIAlign, the
model receives more precise region proposals for the objects to process. In
addition, Mask R-CNN adds another two convolution layers that generate a mask
for each RoI and segment at a pixel level.

The MexPub model is suited to our task since it aims to extract the same
metadata and is already trained on German scientific papers. In addition,
MexPub uses transfer learning by re-training the model from PubLayNet (Zhong
et al., 2019), which already has been trained on a large scientific literature
dataset for classifying the following classes of title, text, list, table, and
figure and is later fine-tuned to extract metadata classes instead.

The submodel is built on Detectron2 (Wu et al., 2019), a state-of-the-art
detection and segmentation algorithm implemented in PyTorch. The library comes
with pre-trained Mask R-CNN using ResNeXt-101 as a backbone with Feature
Pyramid Network (FPN), and the process architecture is following Figure 3.
Firstly, the PDF is converted into an image before feeding it into FPN to
generate feature maps at a different scale. The output from FPN is fed into
Region Proposal Network (RPN) to generate a bounding box that possibly
contains the desired object. Then, a fully convolutional mask prediction
branch is added to the head. Afterwards, ROI-Head obtains the output from FPN
and RPN and use two fully convolution networks to generate bounding box and
classification results. Finally, the output is filtered out by non-maximum
suppression (NMS) to limit the results to the top hundred scoring detection
boxes.

In this submodel, we collect the bounding box and extract the text inside it,
then aggregates the probability for all possible classes in that box from the
model before feeding it into the classifier.

Figure 3.  The image-vision submodel (MASK R-CNN FPN) architecture

###  3.4. Classifier

At the end of our architecture pipeline is our last submodel (classifier) that
takes the output of the biLSTM and the processed output of Mask R-CNN model as
input. The model of choice for this task is biLSTM because of the ability to
learn patterns within sequences. The model takes a vector of length 20
(concatenation of both outputs of length 10 each) as input, and returns a
probability distribution of length 10. The model consists of two stacked LSTM
layers (forward and backward LSTMs) with 256 hidden dimensions as shown in
Figure 4. Finally, a fully connected layer containing 512 input nodes and 10
output nodes with a softmax activation function. The output gives a
probability distribution of a word belonging to the 10 classes.

Figure 4.  The Last BiLSTM submodel architecture

##  4\. Experiment

###  4.1. Data

Due to the nature of our study that is focused on extracting metadata from
German scientific papers only, acquiring labeled and suitable data is a
challenging task. As a computer-vision approach does not necessarily rely on
the language used, we assumed that a natural language processing model will be
affected by the language in the training process. Therefore, it was important
for us to filter documents that mainly used the German language. However,
crawling a lot of German-only documents with all required metadata was
difficult. Some of the metadata was often missing and many repositories of
scientific publications do not offer an option to filter out by language.

In this paper, we used a collection of 300 manually annotated and 8518
generated documents. The dataset includes a variety of different non-standard
layouts that is required to prevent our model on overfitting by learning exact
specifics of the layouts and their text order.

The 300 documents are randomly selected from the
SSOAR111https://www.gesis.org/ssoar/home repository consisting of German
scientific publications from the field of Social Science. We extracted and
preprocessed the text of the first page for each document and labeled it on a
word-by-word level based on our desired metadata classes. In case a word does
not belong to any of these nine classes, we label them as unclassified (i.e.
in the case of words of the first chapter such as the introductory section).
The annotations for each document are saved in a CSV format.

However, manually annotating documents on a word-by-word level is complex and
time-consuming. Therefore, we created a data generation process to enlarge our
dataset. Due to this we randomly selected and extracted metadata records of
German publications from SSOAR11footnotemark: 1,
media/rep/222https://www.mediarep.org/ and
DBLP333https://dblp.org/xml/release/. We used 13 commonly used layouts that we
identified in our manually annotated dataset. Those layouts, among others,
have been identified and used by MexPub (Boukhers et al., 2021a) as well. We
generated 670 synthetic documents for each layout template by inserting the
collected metadata at their respective positions in the template. Two of those
layouts included an English abstract alongside the German abstract for which
we did not have enough data crawled. Thus, we only generated 516 synthetic
documents for each of those.

For each generated document we considered all components of the metadata as a
whole, using all the extracted metadata from a particular record to generate a
single document containing the same data. This approach was chosen in order to
preserve the context of the words in a document such that e.g. the meaning of
the title and abstract relate to each other. We assumed that the context of
words might be important for the model to learn. However, we did not try to
fully randomize each field to generate different combinations of metadata,
which would have enabled us to generate much more documents with the same
amount of data, to test whether this affects the model’s performance. An
example of a generated document using this process is shown in Figure 5.

###  4.2. Training

To train and evaluate our model, we randomly split our training data into 70%
training, 15% validation, and 15% testing. Because of the high complexity of
the architecture proposed, each of the submodels is trained independently
using the processed data except for the image-vision model. The image-vision
submodel (Boukhers et al., 2021a) is pre-trained on a similar dataset. Having
the submodels trained separately enables us to identify and evaluate the
issues of the individual submodels, making debugging easier as well as
achieving a faster process overall. Figure 5 demonstrates the final model’s
output of two randomly selected documents from the testing set. This shows the
approach’s capability to enhance the performance of metadata extraction by
using both contextual and visual features. Tables 1, 2. 3 summarize the
results obtained by this process for all submodels. We trained each of the
biLSTM models with 300 iterations and a batch size of 2000.

Figure 5.  Example outputs of two randomly selected documents Table 1. Performance of the NLP submodel using the testing set | Precision | Recall | F1-score
---|---|---|---
Overall | 0.835 | 0.819 | 0.827
Abstract | 0.934 | 0.922 | 0.928
Author | 0.860 | 0.960 | 0.907
Email | 0.938 | 0.949 | 0.943
Address | 0.631 | 0.857 | 0.727
Date | 0.980 | 0.800 | 0.881
Journal | 0.900 | 0.737 | 0.810
Affiliation | 0.820 | 0.727 | 0.771
DOI | 0.503 | 0.500 | 0.501
Title | 0.956 | 0.919 | 0.937
Table 2. Performance of the image-vision submodel using the testing set | Precision | Recall | F1-score
---|---|---|---
Overall | 0.908 | 0.898 | 0.904
Abstract | 0.961 | 0.913 | 0.936
Author | 0.932 | 0.940 | 0.936
Email | 0.918 | 0.891 | 0.904
Address | 0.897 | 0.906 | 0.901
Date | 0.934 | 0.931 | 0.932
Journal | 0.960 | 0.895 | 0.926
Affiliation | 0.691 | 0.742 | 0.716
DOI | 0.931 | 0.952 | 0.941
Title | 0.950 | 0.920 | 0.935
Table 3. Performance of final model using the testing set | Precision | Recall | F1-score
---|---|---|---
Overall | 0.944 | 0.902 | 0.923
Abstract | 0.989 | 0.962 | 0.975
Author | 0.962 | 0.984 | 0.973
Email | 0.985 | 0.953 | 0.969
Address | 0.922 | 0.940 | 0.931
Date | 0.968 | 0.972 | 0.970
Journal | 0.971 | 0.937 | 0.954
Affiliation | 0.750 | 0.429 | 0.546
DOI | 0.986 | 0.967 | 0.976
Title | 0.963 | 0.961 | 0.962

The proposed approach is able to achieve an overall F1-score of 92.18% in the
testing set when training all the components independently with 300
iterations. Achieving a high overall F1-score validates the model’s
proficiency in utilizing contextual and visual features to accurately extract
metadata from German scientific publications. In fact, it was able to surpass
the performance of the components (NLP and Image-vision model). The results
show that the approach was able to perform better on extracting abstracts,
authors, dates, and DOIs. The high F1-score for the abstract class could be
explained by abstracts usually containing patterns that are unique across all
layouts and typically containing 150-300 words that results in more space
allocation within the paper, since both models are better at detecting larger
patterns than smaller ones. Furthermore, in some cases we identified commonly
used words such as ”journal” that could act as marker words indicating a
specific class. Similarly, the abstract is often preceded by words such as the
German word ”Zusammenfassung”. As for the author class, using the contextual
features helps to achieve this high performance because usually names are
unique within the document which allows the model to better detect such names
within a document page. For the email, DOI, and date classes the high F1-score
is believed to be caused by classes following a unique pattern across all
papers, for example, the emails are usually within the format of
<EMAIL_ADDRESS>This unique pattern allows the model to learn much better
in extracting emails; the same logic applies for dates and DOIs.

However, along with the great evaluation results the approach has had, there
are some weaknesses revealed by these results. The affiliation class appears
to have a very low F1-score compared to other classes. We hypothesize that
this low result is caused by the low occurrences and variety of the
affiliation within the used dataset, therefore, adding more training data is
advised in order to increase the performance of the model with such a class.

###  4.3. Comparison with other approaches

To have a better understanding of the performance of our model, we compared
the approach against other state-of-the-art approaches GROBID (Romary and
Lopez, 2015) and MexPub (Boukhers et al., 2021a). To allow for a fair
comparison, we randomly selected 300 documents from our testing set that our
model has not seen during training. These documents will serve as input for
all the approaches. For all methods, we extracted the metadata from the given
300 documents and calculated the cosine similarity between the extracted
metadata and the ground truth metadata (labeled metadata) of the document.
Note that the given 300 documents are labeled beforehand. If the cosine
similarity was higher than 0.85, the extracted metadata is matching the
correct one. The reason for allowing a dissimilarity of 0.15 is because there
are cases where the extracted metadata misses a part of the text, but this
does not necessarily mean the extraction is false. After extracting the
metadata and calculating the cosine similarity for checking the F1-score is
calculated to evaluate the approaches. Note that we have not included the
evaluation for the classes date and DOI because some of the implemented
approaches are not designed to extract them. Table 4 shows the obtained
comparison results.

Table 4. F1-score comparison between our approach, MexPub and GROBID | Our approach | MexPub | GROBID
---|---|---|---
Overall | 0.846 | 0.823 | 0.618
Abstract | 0.923 | 0.910 | 0.821
Author | 0.807 | 0.824 | 0.770
Email | 0.844 | 0.901 | 0.624
Address | 0.870 | 0.821 | 0.324
Journal | 0.835 | 0.828 | 0.741
Affiliation | 0.679 | 0.535 | 0.240
Title | 0.964 | 0.942 | 0.812

Even though the proposed approach has slightly outperformed MexPub and GROBID
in most classes, there are still two classes in which the approach did not
perform better than MexPub. We hypothesize that the low results in the author
and email classes compared to the ones from MexPub were due to the influence
by the two submodels. If only one of the models outputs a bad result, then the
last model is going to be influenced badly by this. As a result, we presume
that implementing a way to add a confidence level to each of the submodels
outputs would enhance the performance of the approach. By doing so, the last
model will be able to weight each submodel output depending on its confidence
level to prevent the influence of wrong results to the overall performance.
Moreover, decreasing the model complexity could also help to solve the issue
of error aggregation.

##  5\. Conclusion

In this paper, we introduce an approach that automatically extracts metadata
from German scientific papers using a deep learning multimodal approach. It is
able to take advantage of layout and contextual features from a document by
using both image-vision and natural language processing (NLP) models.
Documents will be input into two different submodels (biLSTM, and pre-trained
Mask R-CNN), the output of each of these models will be combined together into
a single feature vector and fed into a final biLSTM model to classify words
within a document into the appropriate metadata class (e.g., title, author,
email, etc.). The overall performance of the approach was able to surpass
other works by averaging an F1-score of 0.923 for all the classes. These
results validate the assumption that including contextual and visual features
increase the overall performance of metadata extraction for German scientific
papers.

In future work, it may be worth exploring more possibilities to decrease the
complexity of the architecture to avoid the problem of error aggregation and
allow for more generalization. Adding confidence level to each of the
submodels could reduce the problem of error aggregation in the final submodel.
In addition, using an additional conditional random field (CRF) layer at the
end of the biLSTM classifier submodel could boost classification results as
shown by (Prasad et al., 2018). Moreover, due to the complexity of the
architecture, increasing the training data with a larger amount of different
layouts can further increase the performance especially with classes that got
low results in the current approach. Finally, by collecting a larger dataset
we could make a more in depth comparison between a pure NLP approach and a
multimodal approach to better understand the strength and weaknesses of each
one.

## Venue

This paper has been accepted and presented at the non-archival workshop:
BiblioDAP@KDD2021444https://bibliodap.uni-koblenz.de/ (Boukhers et al., 2021b)

## References

  * (1)
  * Alzaidy et al. (2019) Rabah Alzaidy, Cornelia Caragea, and C Lee Giles. 2019\.  Bi-LSTM-CRF sequence labeling for keyphrase extraction from scholarly documents. In _The world wide web conference_. 2551–2557.
  * An et al. (2017) Dong An, Liangcai Gao, Zhuoren Jiang, Runtao Liu, and Zhi Tang. 2017.  Citation metadata extraction via deep neural network-based segment sequence labeling. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 1967–1970.
  * Balasubramanian et al. (2016) Vidhya Balasubramanian, Sooryanarayan Gobu Doraisamy, and Navaneeth Kumar Kanakarajan. 2016\.  A Multimodal Approach for Extracting Content Descriptive Metadata from Lecture Videos.  _J. Intell. Inf. Syst._ 46, 1 (2016), 121–145.  https://doi.org/10.1007/s10844-015-0356-5
  * Boukhers et al. (2019) Zeyd Boukhers, Shriharsh Ambhore, and Steffen Staab. 2019\.  An End-to-End Approach for Extracting and Segmenting High-Variance References from PDF Documents. In _2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL)_. 186–195.  https://doi.org/10.1109/JCDL.2019.00035
  * Boukhers et al. (2021a) Zeyd Boukhers, Nada Beili, Timo Hartmann, Prantik Goswami, and Muhammad Arslan Zafar. 2021a.  MexPub: Deep Transfer Learning for Metadata Extraction from German Publications. In _2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL)_. IEEE.
  * Boukhers et al. (2021b) Zeyd Boukhers, Philipp Mayr, and Silvio Peroni. 2021b.  BiblioDAP: The 1st Workshop on Bibliographic Data Analysis and Processing. In _Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_.
  * Chaturvedi et al. (2018) Richa Chaturvedi, Deepak Arora, and Pawan Singh. 2018\.  Conditional random fields and regularization for efficient label prediction.  _ARPN Journal of Engineering and Applied Sciences_ 13, 20 (2018), 8332–8336.
  * Che et al. (2018) Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018.  Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation. In _Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_. Association for Computational Linguistics, Brussels, Belgium, 55–64.  http://www.aclweb.org/anthology/K18-2005
  * Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018.  AllenNLP: A Deep Semantic Natural Language Processing Platform. In _Proceedings of Workshop for NLP Open Source Software (NLP-OSS)_. Association for Computational Linguistics, Melbourne, Australia, 1–6.  https://doi.org/10.18653/v1/W18-2501
  * Girshick et al. (2014) Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation.  _2014 IEEE Conference on Computer Vision and Pattern Recognition_ (2014), 580–587.  arXiv:arXiv:1311.2524
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017.  Mask R-CNN. In _2017 IEEE International Conference on Computer Vision (ICCV)_. 2980–2988.  https://doi.org/10.1109/ICCV.2017.322
  * Liu et al. (2018) Runtao Liu, Liangcai Gao, Dong An, Zhuoren Jiang, and Zhi Tang. 2018.  Automatic Document Metadata Extraction Based on Deep Networks. In _Natural Language Processing and Chinese Computing_. Springer International Publishing, 305–317.  https://doi.org/10.1007/978-3-319-73618-1_26
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep Contextualized Word Representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. Association for Computational Linguistics, New Orleans, Louisiana, 2227–2237.  https://doi.org/10.18653/v1/N18-1202
  * Prasad et al. (2018) Animesh Prasad, Manpreet Kaur, and Min-Yen Kan. 2018.  Neural ParsCit: A Deep Learning-Based Reference String Parser.  _Int. J. Digit. Libr._ 19, 4 (2018), 323–337.  https://doi.org/10.1007/s00799-018-0242-1
  * Ren et al. (2017) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2017\.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 39, 6 (2017), 1137–1149.  https://doi.org/10.1109/TPAMI.2016.2577031
  * Romary and Lopez (2015) Laurent Romary and Patrice Lopez. 2015.  Grobid-information extraction from scientific publications.  _ERCIM News_ 100 (2015).
  * Simões et al. (2009) Gonçalo Simões, H. Galhardas, and Luísa Coheur. 2009\.  Information Extraction tasks : a survey. In _Proceedings of INForum’09_.
  * Stahl et al. (2018) Christopher G. Stahl, Steven R. Young, Drahomira Herrmannova, Robert M. Patton, and Jack C. Wells. 2018\.  DeepPDF: A Deep Learning Approach to Extracting Text from PDFs.  (2018).  https://www.osti.gov/biblio/1460210
  * Tkaczyk et al. (2015) Dominika Tkaczyk, Paweł Szostek, Mateusz Fedoryszak, Piotr Jan Dendek, and Lukasz Bolikowski. 2015\.  CERMINE: Automatic Extraction of Structured Metadata from Scientific Literature.  _Int. J. Doc. Anal. Recognit._ 18, 4 (2015), 317–335.  https://doi.org/10.1007/s10032-015-0249-8
  * Ware and Mabe (2015) Mark Ware and Michael Mabe. 2015.  The STM Report: An overview of scientific and scholarly journal publishing.  (03 2015).
  * Wu et al. (2019) Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. 2019.  Detectron2.  https://github.com/facebookresearch/detectron2.
  * Xie et al. (2017) Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017.  Aggregated Residual Transformations for Deep Neural Networks. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 5987–5995.  https://doi.org/10.1109/CVPR.2017.634
  * Xu et al. (2019) Guixian Xu, Yueting Meng, Xiaoyu Qiu, Ziheng Yu, and Xu Wu. 2019.  Sentiment Analysis of Comment Texts Based on BiLSTM.  _IEEE Access_ 7 (2019), 51522–51532.  https://doi.org/10.1109/ACCESS.2019.2909919
  * Yang and Hsu (2020) Huichen Yang and William H. Hsu. 2020.  Vision-Based Layout Detection from Scientific Literature using Recurrent Convolutional Neural Networks.  25th International Conference on Pattern Recognition (ICPR2020).  (2020).  arXiv:arXiv:2010.11727
  * Zhong et al. (2019) Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. 2019\.  PubLayNet: largest dataset ever for document layout analysis. In _2019 International Conference on Document Analysis and Recognition (ICDAR)_. IEEE, 1015–1022.  https://doi.org/10.1109/ICDAR.2019.00166