# Adapter-based Selective Knowledge Distillation for Federated Multi-domain
Meeting Summarization

Xiachong Feng, Xiaocheng Feng, Xiyuan Du, Min-Yen Kan, Bing Qin Corresponding
author: Xiaocheng FengXiachong Feng, Xiaocheng Feng, Xiyuan Du and Bing Qin
are with the Research Center for Social Computing and Information Retrieval,
Harbin Institute of Technology, 150001, Harbin, Heilongjiang, China (e-mail:
xiachongfeng,xcfeng,xydu,qinb@ir.hit.edu.cn)Xiaocheng Feng and Bing Qin are
with the Peng Cheng Laboratory, 518000, Shenzhen, Guangdong, China (e-mail:
xcfeng,qinb@ir.hit.edu.cn)Min-Yen Kan is with the School of Computing,
National University of Singapore, 117417, Singapore (e-mail:
<EMAIL_ADDRESS>
###### Abstract

Meeting summarization has emerged as a promising technique for providing users
with condensed summaries. However, existing work has focused on training
models on centralized data, neglecting real-world scenarios where meeting data
are infeasible to collect centrally, due to their sensitive nature. This gap
motivates us to explore federated learning for meeting summarization. Two
critical challenges impede progress. First, state-of-the-art summarizers are
based on parameter-heavy pre-trained models. Exchanging such a model’s
parameters across clients imposes large bandwidth costs. Second, as real-world
meeting data belong to various domains and are distributed across clients,
they are instances of non-identically and independently distributed (non-IID).
IID assumptions do not hold, which changes which forms of learning algorithms
best apply. To address this, we propose Adapter-based Federated Selective
Knowledge Distillation (AdaFedSelecKD) for training performant client models.
Specifically, we develop an adapter-based summarization model where two
adapters cooperatively facilitate learning using fewer parameters to reduce
communication costs. Then, we devise a selective knowledge distillation
strategy, assisting clients in robustly handling domain-focused modelling on
their own data, while leveraging global parameters based on non-IID data.
Extensive experiments on the QMSum benchmark demonstrate AdaFedSelecKD can
achieve comparable performance with powerful centralized training methods, and
shows its generalizability and robustness.

###### Index Terms:

Meeting Summarization, Federated Learning, Knowledge Distillation, Parameter-
efficient Fine-tuning.

##  I Introduction

Meeting summarization aims to produce concise meeting summaries given lengthy
meeting transcripts, efficiently facilitating readers to grasp essential
meeting information [1]. With the advancement of meeting technologies, many
meetings are now also recorded regularly and automatically transcribed with AI
tools, facilitating offline meeting reviews. Meeting summarization can
leverage these inputs, further building capabilities to mitigate meeting
overload.

Meeting summarization has attracted extensive research attention as of late
[2, 3]. Existing endeavours focus on developing performant summarization
models, utilizing data resources located in a single location (also known as
centralized meeting summarization). [4, 5, 6, 7, 8, 9, 10].

While meaningful in theory, in practice real-world meeting summarization has
additional privacy challenges that substantially change the problem framing.
Concretely speaking, real-world meetings inextricably contain highly private
and sensitive information; e.g., confidential company contents and personal
information that are private[11]. When extended to multi-modal data, video and
audio meeting recordings often also meet with facial representation and
voiceprint issues since both are likewise highly sensitive [12]. For these
reasons, meeting data is highly sensitive and unable to be shared for model
training purposes and is typically siloed. This makes the collection of
meeting data in a central location infeasible.

As such, despite the encouraging research achievements reported in the current
literature, we find such solutions do not meet the requirements of real-world
scenarios. They neglect the investigation towards developing solutions where
meeting data are necessarily siloed and are distributed across different
client sites.

Figure 1: The overall federated learning framework of multi-domain meeting
summarization. In the concrete setting for this paper, there is one central
server and three clients covering distinct domains: Academic, Committee and
Product. Each client uniquely maintains its own domain-specific data.

To close the above gap, we take the first step to study the meeting
summarization task by leveraging a federated learning framework, a widely-
adopted approach for decentralized machine learning. It enables model training
across multiple distributed clients [13, 14]. Figure 1 depicts the entire
learning framework that aims to effectively train performant client-side
summarization models by deriving global knowledge from other clients, without
needing to access their private data. However, there are two critical
challenges that need to be carefully addressed in order to learn high-
performance summarization models under federated learning. First, current
state-of-the-art meeting summarization models are based on pre-trained
language models that maintain a very large number of parameters. Updating all
model parameters represents an infeasible communication cost. Instead, limited
scale client–server communication is more realistic. This restricts the
exchange of parameter updates between the server and its clients to a budget.
Second, meetings distributed across multiple clients often belong to different
domains. Figure 1 illustrate this scenario, in which there exists three
meeting domains: academic, committee, and product, respectively. A single,
central model would not serve to support the distinct needs of the different
domains. This challenging non-identically and independently distributed (non-
IID) data learning setting often causes the client model to deviate from its
own domain as it learns global knowledge based on non-IID data.

To mitigate the above two challenges, we propose a unified method, dubbed
Adapter-based Federated Selective Knowledge Distillation (AdaFedSelectKD). To
address the first challenge, we draw support from parameter-efficient fine-
tuning techniques and design an adapter-based summarization model to reduce
communication costs. Specifically, we introduce a few lightweight trainable
adapters [15] to pre-trained language models [16, 17] while keeping the pre-
trained language models frozen. We meticulously design two types of adapters —
global adapter and local adapter — tailored for the federated learning
framework to facilitate information exchange between the server and clients.
In particular, the global adapter is responsible for providing global
knowledge while local adapters are optimized towards the local meeting
summarization task. To address the second challenge, we devise a federated
selective knowledge distillation strategy to not only effectively derive
global knowledge for the client summarization model, but also train the model
to favour its own local domain performance. Specifically, the client model
adopts knowledge distillation [18] as the optimization algorithm to both learn
from its local data and distill global knowledge from the global adapter.
Moreover, we propose an entropy-based selective strategy based on the
assumption that the higher the entropy of global knowledge, the more uncertain
the knowledge. This adaptively distills knowledge from the global adapter.

We conduct experiments on the QMSum benchmark [6], which comprises meeting
summarization data across three distinct domains: academic, committee and
product. The automatic evaluation results based on three model variants across
three clients consistently demonstrate the efficacy of our proposed method.
Our results achieve comparable results to centralized training methods.
Moreover, human evaluation results validate the substantial improvements
attained by our method over baseline approaches. We further conduct downstream
analyses of our model’s various settings that allow us to conclude that our
method is both generalizable and robust.

TABLE I: Mathematical notations utilized in this paper. Notation | Description
---|---
Dataset-related
$\mathbb{D}$ | meeting summarization dataset
$\mathbb{X}$ | a set of input documents
$\mathbb{Y}$ | a set of meeting summaries
$\mathcal{X}$ | one input document $\mathcal{X}\in\mathbb{X}$
$\mathcal{Y}$ | one meeting summary $\mathcal{Y}\in\mathbb{Y}$
$V$ | vocabulary
Framework-related
$\mathcal{S}$ | server
$\mathcal{C}$ | client
$i$ | index of the client
$\mathbb{C}$ | a set of clients, $\mathcal{C}_{i}\in\mathbb{C}$
${\mathcal{A}}_{c}$ | client-side optimization algorithm
${\mathcal{A}}_{s}$ | server-side aggregation algorithm
Model-related
${\mathcal{M}}$ | client-side task model (meeting summarizer)
$\boldsymbol{\mathit{W}}$ | learnable parameters of the model ${\mathcal{M}}$
$\boldsymbol{\mathit{X}}$ | output representation of the encoder layer
$\boldsymbol{\mathit{Y}}$ | output representation of the decoder layer
${\hat{\boldsymbol{\mathit{Y}}}}$ | output representation of the adapter
$l$ | index of the transformer layer
$L$ | the number of transformer layers
$n$ | the dimension of the model ${\mathcal{M}}$
$m$ | adapter bottleneck dimension
$q$ | normalized output distribution (after softmax)
Learning-related
$\mathcal{L}$ | loss function
$\lambda$ | weight for the knowledge distillation loss $\mathcal{L}_{KD}$
$r$ | index of the federated learning round
$H({q})$ | entropy of the distribution ${q}$
$\tau$ | threshold for the entropy

##  II Preliminaries

We first introduce the multi-domain meeting summarization dataset and the task
definition, then provide an overview of federated learning. We define all of
the mathematical notation employed in this work in Table I.

###  II-A Multi-domain Meeting Summarization Dataset

TABLE II: Statistics for the QMSum dataset encompass three domains: Academic, Committee and Product. “#” indicates the quantity of document–summary pairs. The average number of turns during meetings is denoted by “Avg. Turns”. “Avg. Speakers” represents the average number of speakers participating in the meetings. “Avg. Tokens” refers to the average number of words spoken during the meetings, and “Avg. Sum” indicates the average number of words in the summaries. | Academic | Committee | Product
---|---|---|---
| Train | Valid | Test | Train | Valid | Test | Train | Valid | Test
# | 218 | 45 | 49 | 284 | 67 | 66 | 593 | 125 | 129
Avg. Turns | 54.64 | 59.33 | 46.45 | 9.51 | 9.13 | 10.85 | 68.61 | 79.30 | 77.86
Avg. Speakers | 4.35 | 4.09 | 4.22 | 3.26 | 3.10 | 4.14 | 3.76 | 3.90 | 3.93
Avg. Tokens | 1049.8 | 1156.53 | 912.00 | 667.14 | 607.9 | 713.26 | 898.04 | 903.53 | 933.67
Avg. Sum | 46.22 | 50.67 | 43.82 | 77.95 | 73.67 | 69.94 | 65.09 | 65.38 | 57.38

In this paper, we leverage the QMSum dataset [6] to conduct experiments under
the federated learning setting. QMSum consists of query–summary pairs over 232
meeting transcripts from three distinct domains: namely academic, committee
and product meetings. This dataset is thus well-suited for the multi-domain
meeting summarization task. Under our federated scenario, we posit that three
clients hold meetings from each of the three distinct domains, respectively.
Notably, QMSum is a query-based meeting summarization dataset, in which each
instance is composed of a specific query, the relevant meeting transcripts and
the corresponding summary. Following Lee and Sogaard [8], we concatenate the
query and the meeting transcripts to construct the input document
$\mathcal{X}$, resulting in a parallel corpus
${\mathbb{D}}:({\mathcal{X}},{\mathcal{Y}})\in({\mathbb{X}},{\mathbb{Y}})$,
where $\mathcal{Y}$ is the corresponding summary with respect to
${\mathcal{X}}$. We give detailed statistics for the QMSum dataset in Table
II.

###  II-B Task Definition

Given the document $\mathcal{X}$, the client-side meeting summarization model
aims to produce a concise meeting summary $\mathcal{Y}$, where $\mathcal{X}$
is the concatenation of one query’s words and relevant meeting transcripts
$[\underbrace{x_{1},x_{2},...,x_{i}}_{\texttt{query}}\texttt{\\#SEP\\#}\underbrace{x_{i+1},x_{i+2},...,x_{|\mathcal{X}|}}_{\texttt{transcripts}}]$,
#SEP# denotes one specific token between the query and transcripts. Note that
speaker roles, such as “marketing” and “project manager”, are treated as
ordinary tokens and included in transcripts. $\mathcal{Y}$ consists of
$|\mathcal{Y}|$ words $[y_{1},y_{2},...,y_{|\mathcal{Y}|}]$. A brief example
is shown as follows:

  * •

Query $\mathcal{X}_{[1:i]}$: Summarize the discussion about the trends of
current remote controls.

  * •

Meeting Transcripts $\mathcal{X}_{[i+1:|\mathcal{X}|]}$: Marketing: This is
just a presentation on the trends that we’re gonna use to make the product
stand out from …… Project Manager: What do you think of adding an LCD? ……

  * •

Corresponding Summary $\mathcal{Y}$: The group discussed different trends
based on different ages of people, …, finally they decide to add an LCD
screen.

###  II-C Federated Learning Framework

We investigate the multi-domain meeting summarization task under the federated
learning framework. Federated learning adopts a client–server paradigm and
enables the collaborative training of models across multiple decentralized
data sources without harvesting the sensitive raw data [13].

Roughly speaking, the federated learning methodology progresses in a
synchronous, iterative fashion. During each learning round, every client
locally optimizes its own client-side model based on its private data via the
client-side optimization algorithm and then transmits the updated parameters
to the central server. Subsequently, the server gathers the updates from
clients and aggregates them into new server parameters by means of the server-
side aggregation algorithm. Finally, the new global parameters are broadcast
from the server to all clients for the next round. It should be noted that
three crucial components constitute this learning process:

  * •

Client-side model ${\mathcal{M}}$ housing the client-specific model parameters
$\boldsymbol{\mathit{W}}$. It is in charge of generating meeting summaries.

  * •

Client-side optimization algorithm ${\mathcal{A}}_{c}$ endeavors to optimize
the client-side model ${\mathcal{M}}$ based on the local private data
${\mathbb{D}}$.

  * •

Server-side aggregation algorithm ${\mathcal{A}}_{s}$ is responsible for
aggregating parameters furnished by clients.

Formally speaking, there exists a set $\mathbb{C}$ of $|\mathbb{C}|$ clients,
where each individual client is denoted $\mathcal{C}\in\mathbb{C}$. As
illustrated in our concrete scenario in Figure 1, there are three
($|\mathbb{C}|=3$) clients in our set; namely, the academic client, the
committee client, and the product client. Each client $\mathcal{C}_{i}$
possesses its own private domain-specific corpus of meeting summaries
${\mathbb{D}}_{i}$, as well as a client-specific meeting summarization model
${\mathcal{M}}_{i}$. The learnable parameters ${\boldsymbol{\mathit{W}}}_{i}$
of model ${\mathcal{M}}_{i}$ are optimized using the client-side optimization
algorithm ${\mathcal{A}}_{c}$ based on the local dataset ${\mathbb{D}}_{i}$ in
the $r_{th}$ round of optimization.

${\boldsymbol{\mathit{W}}}_{i}^{r+1}\leftarrow{\mathcal{A}}_{c}({\mathcal{M}}_{i}({\boldsymbol{\mathit{W}}}_{i}^{r}),{\mathbb{D}}_{i}).$
(1)

Subsequently, the central server $\mathcal{S}$ aggregates all updated
parameters ${\boldsymbol{\mathit{W}}}^{r+1}_{i}$ from the clients and adopts a
server-side aggregation algorithm (${\mathcal{A}}_{s}$) to consolidate the
information. In particular, we utilize the Federated Averaging (FedAvg)
algorithm [19] as our aggregation algorithm ${\mathcal{A}}_{s}$.

$\begin{split}{\boldsymbol{\mathit{W}}}^{r+1}\leftarrow\sum_{i=1}^{|\mathbb{C}|}\frac{|{\mathbb{D}}_{i}|}{|{\mathbb{D}}|}{\boldsymbol{\mathit{W}}}_{i}^{r+1},\;\text{where}\;|{\mathbb{D}}|=\sum_{i=1}^{|\mathbb{C}|}|{\mathbb{D}}_{i}|,\end{split}$
(2)

where $|\mathbb{C}|$ denotes the number of clients, $|{\mathbb{D}}_{i}|$
represents the number of instances in the local dataset ${\mathbb{D}}_{i}$ and
$|{\mathbb{D}}|$ gives the total number of instances among all clients.

Afterwards, the newly-gathered server-side parameters
${\boldsymbol{\mathit{W}}}^{r+1}$ are distributed to all clients
$\mathcal{C}_{i}$ to offer enriched global knowledge. In the forthcoming
methodology section (§III), we demonstrate our contribution towards a more
communication-efficient client-side model ${\mathcal{M}}_{i}$ and robust
client-side optimization algorithm ${\mathcal{A}}_{c}$.

Figure 2: Illustration of our proposed AdaFedSelectKD learning framework. The
overall framework adheres to a client–server learning paradigm. At the bottom,
three clients are depicted, where each client adopts the selective knowledge
distillation algorithm to optimize its own adapter-based meeting summarizer
using its domain-specific private data. Two types of adapters are tailored for
the information exchange between the server and clients, including the global
adapter and the local adapter. The optimized parameters from three clients are
then conveyed to the central server. At the top, the central server employs
the federated averaging algorithm to aggregate client information. The
resulting new parameters are distributed to the clients for the subsequent
learning round.

##  III Methodology

We invent an integrated method, adapter-based federated selective knowledge
distillation (AdaFedSelectKD), to achieve the efficient and robust federated
multi-domain meeting summarization task. It comprises (1) A client-side model
${\mathcal{M}}$, which is an adapter-based meeting summarizer, and (2) A
client-side optimization algorithm ${\mathcal{A}}_{c}$: which is a selective
knowledge distillation algorithm. The overall AdaFedSelectKD learning
procedure is illustrated in Figure 2.

###  III-A Overview

Our proposed AdaFedSelectKD method significantly enhances two dimensions of
the overall learning process.

  * •

At the model level, we introduce a client-side model called the adapter-based
meeting summarizer (${\mathcal{M}}$), which employs a frozen pre-trained
language model as its backbone model (refer to §III-B2) and integrates few
learnable lightweight adapters to facilitate communication-efficient learning
(refer to §III-B3). Specifically, we develop two kinds of adapters tailored
for the federated learning setting, namely the global adapter and the local
adapter. The global adapter functions as an intermediary for exchanging
information between the server and clients, while the local adapter not only
distils global knowledge from the global adapter but also is optimized towards
the local domain.

  * •

At the algorithm level, we devise a client-side optimization algorithm termed
the selective knowledge distillation strategy (${\mathcal{A}}_{c}$), which
adaptively and robustly optimizes local learnable adapters. Concretely, the
knowledge distillation method permits deriving global knowledge from the
server while ensuring the summarizer is prone to the local domain (refer to
§III-C2). Furthermore, our meticulously designed selective strategy draws
support from entropy as a measure of uncertainty and shows great promise in
transferring credible global knowledge to clients (refer to §III-C3).

###  III-B Adapter-based Meeting Summarizer

In this section, we elucidate our motivation for incorporating adapters and
subsequently delineate the precise model architecture for both the backbone
model as well as two varieties of adapters, namely the global adapter and the
local adapter.

####  III-B1 Motivation

In this section, we elaborate on the motivation underlying the design of our
adapter-based meeting summarization system by addressing the following two
questions:

  * •

Why do we employ the adapter? In recent years, pre-trained language models
have dominated the natural language processing field and have achieved
remarkable success. Therefore, it is ideal to leverage such models as potent
meeting summarization systems. However, there are two key challenges. Firstly,
exchanging the parameters of these pre-trained language models incurs high
client–server communication costs due to a large number of model parameters.
Secondly, the lack of sufficient hardware capabilities in real-world scenarios
means some clients may struggle to handle such compute-intensive tasks. On
this account, we apply the parameter-efficient fine-tuning strategy to the
pre-trained language model by fine-tuning only a few lightweight adapters,
thereby addressing the above challenges.

  * •

Why do we design two types of adapters? The most primitive and widely adopted
federated learning algorithm is the Federated Averaging, which directly
broadcasts newly aggregated global parameters to clients to initialize their
models for the next round of training [19]. However, such a method notoriously
performs poorly when clients hold non-IID data since aggregating divergent
model parameters leads to model distraction [20], thereby leading to the
client model cannot focus on its own domain. To combat this issue, we design
two types of adapters. One is the global adapter that receives server
parameters and provides global knowledge via output distribution. The other is
the local adapter that is optimized towards the local summarization task. By
bifurcating the parameters in this fashion, we overcome the difficulties that
arise from aggregating disparate model parameters across clients with non-IID
data.

####  III-B2 Backbone Model

We employ two types of pre-trained language models, one is BART [16] and the
other is LED [17], as the backbone model. Both of them adopt the Transformer
architecture [21] and have been pre-trained on a huge volume of data. They
inherit a sequence-to-sequence framework, whereby the encoder first encodes
the source sequence into distributed representations, which are then decoded
by the decoder to generate the target summary.

Formally speaking, the input to the encoder is $\boldsymbol{\mathit{X}}^{0}$,
which denotes the sum of the word embeddings
$\boldsymbol{\mathit{X}}_{\mathrm{emb}}$ and position embeddings
$\boldsymbol{\mathit{X}}_{\mathrm{pos}}$ of the input document $\mathcal{X}$.
$\stackrel{{\scriptstyle L}}{{\underset{l=1}{:=}}}$ symbolizes $L$ identical
layers and $\boldsymbol{\mathit{X}}^{l-1}$ signifies the output representation
of the ${l-1}_{th}$ encoder layer. Besides, $\textsc{Ffn}(\cdot)$ represents a
position-wise feed-forward network, and $\textsc{Self-Att}(\cdot)$ denotes a
multi-head self-attention.

$\begin{split}\boldsymbol{\mathit{X}}^{L}&=\textsc{Encoder}(\boldsymbol{\mathit{X}}^{0})\stackrel{{\scriptstyle
L}}{{\underset{l=1}{:=}}}\textsc{Ffn}\left(\textsc{Self-
Att}(\boldsymbol{\mathit{X}}^{l-1})\right)\end{split}$ (3)

The decoder takes the output $\boldsymbol{\mathit{X}}^{L}$ of the encoder and
the shifted right representation $\boldsymbol{\mathit{Y}}^{0}$ of
$\mathcal{Y}$ as the input to produce the final representation
$\boldsymbol{\mathit{Y}}^{L}$, which will be projected into the vocabulary
space in order to predict the summary.

$\begin{split}\boldsymbol{\mathit{Y}}^{L}&=\textsc{Decoder}(\boldsymbol{\mathit{Y}}^{0},\boldsymbol{\mathit{X}}^{L})\\\
&\stackrel{{\scriptstyle
L}}{{\underset{l=1}{:=}}}\textsc{Ffn}\left(\textsc{Cross-
Att}\left(\textsc{Self-
Att}(\boldsymbol{\mathit{Y}}^{l-1}),\boldsymbol{\mathit{X}}^{L}\right)\right)\end{split}$
(4)

where Cross-Att represents multi-head cross-attention. Additionally, each
encoder and decoder layer is surrounded by residual connection [22] and layer
normalization [23].

####  III-B3 Global-Local Adapters

Adapters are additional modules interpolated between layers of a pre-trained
model111We conduct preliminary experiments and find that it is more effective
to only add adapters between decoder layers of the pre-trained language model
for the meeting summarization task. Similar conclusions are corroborated by
Dai et al. [24]..

Note that the core attribute of adapters is the exceedingly small number of
parameters compared with the entire pre-trained language model, which paves
the way for efficient fine-tuning and communication cost reduction.
Specifically, we craft two types of adapters tailored for the federated
learning framework.

  * •

Global adapter plays the role of parameter container, which receives
aggregated parameters from the server and generates the output distribution
that provides global knowledge to the local client. Note that the global
adapter is only responsible for passing parameters and will not be optimized.

  * •

Local adapter combined with the pre-trained language model servers as the
final client meeting summarization model, which is core to mitigate the non-
IID data learning challenge. Instead of directly adopting server parameters as
local adapter parameters, the local adapter is optimized towards its local
domain by training on the local private dataset while deriving global
knowledge from the global adapter.

Figure 3: Illustration of the adapter architecture. Two types of adapters are
added between transformer layers, including the global adapter and the local
adapter. Both adapters share the same architecture, comprising a down-
projection feed-forward layer, a non-linear activation function, an up-
projection feed-forward layer and a residual connection module equipped with
layer normalization. The global adapter receives parameters from the server
and provides global knowledge, whereas the local adapter is co-optimized
through training on the local data and distilling knowledge from the global
adapter. The updated parameters are then transmitted to the server for the
next round of learning.

Despite their distinct functions, the two types of adapters share an identical
architecture. Specifically, we have adopted the bottleneck adapter
architecture exemplified in Houlsby et al. [15] for our adapters. Precisely,
each adapter consists of two feed-forward layers, one non-linear activation
function, and a residual connection module with layer normalization. The
overall architecture is illustrated in Figure 3.

Formally speaking, given the adapter bottleneck dimension $m$, each adapter
first utilizes a down-projection feed-forward module with learnable parameter
${\boldsymbol{\mathit{W}}}_{\text{down}}\in{\mathbb{R}}^{n\times m}$ to
project the input ${\boldsymbol{\mathit{Y}}}^{l}\in{\mathbb{R}}^{n}$ into the
$m$-dimensional representation222We use the notation $\boldsymbol{\mathit{Y}}$
since our adapters are added between transformer decoder layers., where
${\boldsymbol{\mathit{Y}}}^{l}$ is produced by the $l_{th}$ transformer
decoder layer of the pre-trained language model and $m$ is smaller than $n$.
Subsequently, a non-linear activation function ReLu and an up-projection feed-
forward module with learnable parameter
${\boldsymbol{\mathit{W}}}_{\text{up}}\in{\mathbb{R}}^{m\times n}$ are
employed to project the vector back into $n$-dimensional representation.
Finally, a residual connection and layer normalization is applied to produce
final ${\hat{\boldsymbol{\mathit{Y}}}}^{l}$.

$\begin{split}{\hat{\boldsymbol{\mathit{Y}}}}^{l}\leftarrow\texttt{LayerNorm}({\boldsymbol{\mathit{Y}}}^{l}+\texttt{ReLu}({{\boldsymbol{\mathit{Y}}}^{l}}{{\boldsymbol{\mathit{W}}}_{\text{down}}}){\boldsymbol{\mathit{W}}}_{\text{up}})\end{split}$
(5)

Concretely, upon processing by the global adapter,
${\boldsymbol{\mathit{Y}}}^{l}$ and ${\hat{\boldsymbol{\mathit{Y}}}}^{l}$ are
instantiated as ${\boldsymbol{\mathit{Y}}}^{l}_{g}$ and
${\hat{\boldsymbol{\mathit{Y}}}}^{l}_{g}$, respectively. Likewise, when
processed by the local adapter, ${\boldsymbol{\mathit{Y}}}^{l}$ and
${\hat{\boldsymbol{\mathit{Y}}}}^{l}$ are instantiated as
${\boldsymbol{\mathit{Y}}}^{l}_{l}$ and
${\hat{\boldsymbol{\mathit{Y}}}}^{l}_{l}$, respectively.

###  III-C Selective Knowledge Distillation Strategy

We first introduce our motivation to leverage knowledge distillation as the
client-side optimization algorithm, and then present our selective strategy
that further boosts the performance.

####  III-C1 Motivation

Despite the apparent benefits of federated learning, the non-IID data learning
setting leads to the domain drift problem of the client model. In other words,
directly using the global parameters derived by aggregating updates from
distinct different domains makes it impossible for local models to focus on
their own domain. To remedy this issue, previous efforts have discovered
knowledge distillation as one performant method [20]. Building on this
foundation, we first put forward our optimization method by dexterously
unifying both the adapter-based parameter-efficient fine-tuning strategy and
the knowledge distillation method, which not only reduces communication costs
but also facilitates the robust learning of non-IID data across clients.
Additionally, global knowledge from the server is not always informative and
beneficial [25]. To address this concern, we devise a selective strategy,
culminating in our final client-side optimization algorithm $\mathcal{A}_{c}$,
AdaFedSelectKD, which adaptively and robustly distils credible global
knowledge to the client model. Algorithm 1 shows the entire AdaFedSelectKD
algorithm.

Algorithm 1 Adapter-based Federated Selective Knowledge Distillation
Algorithm.
$\mathbb{C}$ is the client set, $\boldsymbol{\mathit{W}}$ is the learnable
parameters, $\mathbb{D}$ is the meeting summarization dataset, $E$ is the
number of local epochs and $\eta$ is the learning rate.

1:procedure Server executes:

2: initialize ${\boldsymbol{\mathit{W}}}^{1}$

3: for each round $r=1,2,\dots$ do

4: for each client $\mathcal{C}_{i}\in\mathbb{C}$ in parallel do

5:
${\boldsymbol{\mathit{W}}}_{i}^{r+1}\leftarrow\textsc{AdaFedSelectKD}(i,{\boldsymbol{\mathit{W}}}^{r})$

6: end for

7: $|{\mathbb{D}}|=\sum_{i=1}^{|\mathbb{C}|}|{\mathbb{D}}_{i}|$

8:
${\boldsymbol{\mathit{W}}}^{r+1}\leftarrow\sum_{i=1}^{|\mathbb{C}|}\frac{|{\mathbb{D}}_{i}|}{|{\mathbb{D}}|}{\boldsymbol{\mathit{W}}}_{i}^{r+1}$

9: end for

10:end procedure

11:

12:procedure AdaFedSelectKD:($i,{\boldsymbol{\mathit{W}}}$)

13: for each local epoch from $1$ to $E$ do

14: for each training instance $(\mathcal{X},\mathcal{Y})\in\mathbb{D}_{i}$ do

15: for each training target word $y_{t}\in\mathcal{Y}$ do

16:
$q_{g},q_{l}={\mathcal{M}_{i}}(\boldsymbol{\mathit{W}},(\mathcal{X},\mathcal{Y}_{[1:y_{t-1}]}))$

17: if $H({q_{g}})<\tau$ then

18:
$\mathcal{L}=(1-\lambda)\mathcal{L}_{CE}\left(q_{l},y_{t}\right)+\lambda\mathcal{L}_{KL}\left(q_{l},q_{g}\right)$

19: else

20: $\mathcal{L}=\mathcal{L}_{CE}\left(q_{l},y_{t}\right)$

21: end if

22:
$\boldsymbol{\mathit{W}}\leftarrow\boldsymbol{\mathit{W}}-\eta\triangledown{\mathcal{L}}(\boldsymbol{\mathit{W}};(\mathcal{X},\mathcal{Y}_{[1:y_{t-1}]}))$

23: end for

24: end for

25: end for

26: return ${\boldsymbol{\mathit{W}}}$ to server

27:end procedure

####  III-C2 Knowledge Distillation

Knowledge distillation is a solid method for transferring knowledge of the
teacher model to the student model by minimizing the discrepancy between the
outputs from two models with a proxy dataset [26].

Formally speaking, given the training set ${\mathbb{D}}$, for each training
instance $({\mathcal{X}},{\mathcal{Y}})\in({\mathbb{X}},{\mathbb{Y}})$, we
obtain the final-layer representations,
${\hat{\boldsymbol{\mathit{Y}}}}^{L}_{g}$ and
${\hat{\boldsymbol{\mathit{Y}}}}^{L}_{l}$, produced by the global adapter and
the local adapter, respectively. After being transformed by the language head,
which projects the representation into $|V|$-dimensional probability
distributions (after softmax operation), we obtain the outputs $q_{g}$ and
$q_{l}$, respectively. Within our framework, we regard $q_{g}$ as the output
of the teacher model and $q_{l}$ as the output of the student model.
Consequently, the local adapter can be trained utilizing a linear combination
of two loss functions.

$\begin{split}\mathcal{L}=(1-\lambda)\mathcal{L}_{CE}\left(q_{l},y\right)+\lambda\mathcal{L}_{KL}\left(q_{l},q_{g}\right)\end{split}$
(6)

where $\mathcal{L}_{CE}$ represents the cross-entropy loss between the
predicted distribution $q_{l}$ and the one-hot true label $y$.
$\mathcal{L}_{KL}$ denotes the Kullback-Leibler divergence between $q_{g}$ and
$q_{l}$. The scalar $\lambda$ serves to determine the weight between the two
loss terms in the overall objective function.

####  III-C3 Selective Strategy

The immaturity parameters provided by the server inevitably introduce useless
information to local model learning [27]. To alleviate this problem, we draw
inspiration from previous works in classification [28] and summarization [29],
which employ the entropy as a measure of uncertainty, and devise a selective
strategy to adaptively distill the knowledge provided by the server.

In detail, when training the $t_{th}$ target word $y_{t}$ of the instance
$(\mathcal{X},\mathcal{Y})$, we have a normalized $|V|$-dimensional
probability distribution
${q_{g}}=[{q_{g}^{1}},{q_{g}^{2}},...,{q_{g}^{|V|}}]$, where $|V|$ is the
vocabulary size. Given this, the entropy of ${q_{g}}$ is defined as:

$\begin{split}H({q_{g}})=-\sum_{i=1}^{|V|}P\left({q_{g}^{i}}\right)\log
P\left({q_{g}^{i}}\right)\end{split}$ (7)

We assume that the higher the entropy, the more uncertain the knowledge
provided by the global adapter, which means global knowledge with high entropy
has no confidence in handling the current learning situation, thereby needing
to be ignored. Based on this assumption, we finally propose our selective
knowledge distillation strategy whereby the knowledge distillation loss is
only accounted for when the entropy falls below a pre-defined entropy
threshold $\tau$.

$\begin{split}\mathcal{L}=\begin{cases}(1-\lambda)\mathcal{L}_{CE}\left(q_{l},y_{t}\right)+\lambda\mathcal{L}_{KL}\left(q_{l},q_{g}\right)&\text{
if }H({q_{g}})<\tau\\\ \mathcal{L}_{CE}\left(q_{l},y_{t}\right)&\text{
otherwise. }\end{cases}\end{split}$ (8)

##  IV Experiments

In this section, we first introduce our research questions and then present
baseline methods including both non-federated and federated learning settings,
and finally describe evaluation metrics and implementation details.

###  IV-A Research Questions

Our experiments are intended to address the following research questions:

  * •

Research Question 1: How does the proposed AdaFedSelecKD perform, and is it
comparable to powerful centralized training methods?

  * •

Research Question 2: How well does the proposed AdaFedSelecKD generalize? Can
it achieve good performance under a variety of settings, particularly under
more severe non-IID data situations?

  * •

Research Question 3: How does the proposed selective knowledge distillation
strategy work specifically and what are the underlying mechanisms?

TABLE III: Main results of adopting BART-large as the backbone model. The AdaCentralized is one super strong method that explicitly trains the model on data from all three domains. Client models obtained via different methods are tested on the domain-specific test set of each client. $\dagger$ and $\dagger\dagger$ indicate the first-ranked and second-ranked results respectively. Results are averaged over three random runs. Client | Setting | Method | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---|---|---
Academic | Single | AdaSingle | 24.83 | 5.70 | 17.24
Centralized | AdaCentralized | 26.74†† | 7.95† | 20.73†
2-6 | Federated | AdaFedAvg | 25.76 | 6.18 | 18.70
| Federated | AdaFedKD | 26.66 | 7.02 | 19.26
| Federated | AdaFedSelectKD | 27.09† | 7.62†† | 19.79††
Committee | Single | AdaSingle | 32.38 | 13.08 | 23.24
Centralized | AdaCentralized | 34.59†† | 15.79† | 25.32††
2-6 | Federated | AdaFedAvg | 33.80 | 14.04 | 24.19
| Federated | AdaFedKD | 34.12 | 14.65 | 24.83
| Federated | AdaFedSelectKD | 34.70† | 15.19†† | 25.37†
Product | Single | AdaSingle | 31.83 | 11.11 | 21.27
Centralized | AdaCentralized | 34.53† | 12.78†† | 23.71†
2-6 | Federated | AdaFedAvg | 32.50 | 12.19 | 22.45
| Federated | AdaFedKD | 33.09 | 12.50 | 23.11
| Federated | AdaFedSelectKD | 33.32†† | 12.86† | 23.53††

###  IV-B Baseline Methods

Our baseline methods can be divided into two categories: non-federated
learning and federated learning. All adopt the adapter-based pre-trained
language model as the backbone model.

  * •

AdaSingle.
Setting: Non-federated learning setting.
Model: The model only has the local adapter that will be optimized.
Method: Training and testing the model with data from a single domain.

  * •

AdaCentralized.
Setting: Non-federated learning setting.
Model: The model only has the local adapter that will be optimized.
Method: Training the model using the whole QMSum that covers all three domains
and testing the model using only one single-domain data. Centralized training
methods are always viewed as one super strong baseline for federated methods.

  * •

AdaFedAvg.
Setting: Federated learning setting.
Model: The model has one type of adapter that will be optimized during
training.
Method: The fundamental federated learning algorithm, where clients hold
client-specific parameters and perform local updates based on their private
data via maximum likelihood estimation. Afterwards, the server gathers the
weighted average of all client updates as the new global parameters, which
will be distributed to all clients as new client-specific parameters.

  * •

AdaFedKD.
Setting: Federated learning setting.
Model: The model has two types of adapters including both the global adapter
and the local adapter. The local adapter is optimized during training.
Method: Each client performs local updates via knowledge distillation while
the server employs federated averaging as the parameter-aggregation algorithm.

  * •

AdaFedSelectKD.
Setting: Federated learning setting.
Model: The model has two types of adapters including both the global adapter
and the local adapter. The local adapter is optimized during training.
Method: Based on the AdaFedKD, the selective strategy is introduced to filter
out global knowledge.

TABLE IV: Main results of adopting BART-base as the backbone model. The AdaCentralized is one super strong method that explicitly trains the model on data from all three domains. Client models obtained via different methods are tested on the domain-specific test set of each client. $\dagger$ and $\dagger\dagger$ indicate the first-ranked and second-ranked results respectively. Results are averaged over three random runs. Client | Setting | Method | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---|---|---
Academic | Single | AdaSingle | 23.97 | 5.70 | 17.81
Centralized | AdaCentralized | 25.66† | 6.72† | 18.88††
2-6 | Federated | AdaFedAvg | 24.19 | 6.19 | 18.20
| Federated | AdaFedKD | 24.84 | 6.25 | 18.32
| Federated | AdaFedSelectKD | 25.25†† | 6.48†† | 18.98†
Committee | Single | AdaSingle | 27.68 | 9.41 | 19.30
Centralized | AdaCentralized | 28.61†† | 11.23† | 21.21†
2-6 | Federated | AdaFedAvg | 28.09 | 9.87 | 20.31
| Federated | AdaFedKD | 28.21 | 10.40 | 20.49
| Federated | AdaFedSelectKD | 28.88† | 11.14†† | 21.07††
Product | Single | AdaSingle | 28.47 | 9.86 | 19.85
Centralized | AdaCentralized | 30.83† | 11.15†† | 20.92††
2-6 | Federated | AdaFedAvg | 29.34 | 10.47 | 20.42
| Federated | AdaFedKD | 29.89 | 10.56 | 20.74
| Federated | AdaFedSelectKD | 30.41†† | 11.25† | 21.15†

###  IV-C Evaluation Metrics

We adopt the standard metrics ROUGE [30] for evaluation and obtain the $F_{1}$
scores for ROUGE-1, ROUGE-2, and ROUGE-L that measures the word-overlap,
bigram-overlap and longest common sequence between the ground-truth and the
generated summary respectively. We use the implementation provided by
HuggingFace333https://github.com/huggingface/evaluate.

###  IV-D Implementation Details

We use the Flower framework to simulate the federated learning
environment444https://github.com/adap/flower. Specifically, we establish one
central server and three distributed clients for the academic, committee, and
product domains, respectively. During each round of federated learning, all
three clients are engaged in the training process, indicating a client
participation rate of 100%. The server employs the federated averaging
algorithm to aggregate the gathered information. On the client side, we employ
both BART [16] and LED [17] as the backbone model to conduct experiments. We
initially conduct hyperparameter search experiments via grid search to
determine the final hyperparameters. For the BART-large model and LED-large
model, adapters are added to the top six transformer decoder layers with an
adapter bottleneck dimension of 2048. For the BART-base model, adapters are
added to the top three transformer decoder layers with an adapter bottleneck
dimension of 1536. For each client, we used the AdamW optimizer with a
learning rate of 2e-4 and a batch size of 16. The weight decay is set to 0.01.
The loss weight $\lambda$ is set to 0.2 and the entropy threshold $\tau$ for
the selective strategy is set to 5 across all model variants555Our codes and
models will be made public..

TABLE V: Main results of adopting LED-large as the backbone model. The AdaCentralized is one super strong method that explicitly trains the model on data from all three domains. Client models obtained via different methods are tested on the domain-specific test set of each client. $\dagger$ and $\dagger\dagger$ indicate the first-ranked and second-ranked results respectively. Results are averaged over three random runs. Client | Setting | Method | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---|---|---
Academic | Single | AdaSingle | 24.28 | 6.03 | 17.58
Centralized | AdaCentralized | 26.30† | 6.94†† | 18.64††
2-6 | Federated | AdaFedAvg | 25.13 | 6.34 | 18.19
| Federated | AdaFedKD | 25.49 | 6.58 | 18.30
| Federated | AdaFedSelectKD | 25.92†† | 7.09† | 18.67†
Committee | Single | AdaSingle | 32.69 | 13.03 | 22.58
Centralized | AdaCentralized | 33.71† | 14.94†† | 23.87†
2-6 | Federated | AdaFedAvg | 32.72 | 13.88 | 23.22
| Federated | AdaFedKD | 32.98 | 14.39 | 23.46
| Federated | AdaFedSelectKD | 33.53†† | 14.95† | 23.85††
Product | Single | AdaSingle | 30.12 | 9.85 | 19.70
Centralized | AdaCentralized | 32.72† | 10.75† | 22.17††
2-6 | Federated | AdaFedAvg | 30.43 | 9.93 | 21.66
| Federated | AdaFedKD | 31.69 | 10.12 | 21.93
| Federated | AdaFedSelectKD | 32.07†† | 10.67†† | 22.64†

##  V Results

###  V-A Research Question 1

To answer the first research question “How does the proposed AdaFedSelecKD
perform, and is it comparable to powerful centralized training methods?”, we
conduct both automatic evaluations by comparing various methods and human
evaluations to comprehensively access the performance.

####  V-A1 Automatic Evaluation

The results illustrated in Tables III, IV and V correspond to the BART-large,
BART-based and LED-large backbone models, respectively. To sum up, the
following conclusions can be drawn. Firstly, the outcomes confirm that our
proposed AdaFedSelectKD outperforms the baseline method AdaFedAvg, improving
the ROUGE score by approximately 1.2 points. Secondly, compared with AdaFedKD,
our optimized AdaFedSelectKD demonstrates superior performance, which confirms
that the selective strategy constitutes the vital component for robust and
efficacious federated knowledge distillation. Thirdly, the results also
validate that our AdaFedSelectKD can achieve comparable even superior
performance relative to AdaCentralized, which is a piece of solid evidence to
verify the effectiveness of our method. Fourthly, the improvements achieved
across the three model variants indicate the stability and generalizability of
our method. Due to the better results based on the BART-large backbone model,
the following experiments are all based on the BART-large model.

Figure 4: Generated meeting summary comparison of AdaFedSelectKD with other
methods on 60 randomly-chosen meetings. For example, compared with
AdaFedSelectKD, AdaFedAvg performs better on 4 of the 60 summaries and worse
on 52.

####  V-A2 Human Evaluation

We employ three evaluators to undertake our human evaluation. All three
evaluators are researchers in natural language processing who are well-versed
in the task of meeting summarization. Each evaluator is remunerated $30 for
this evaluation task.

First, 20 meetings are randomly selected per domain, amounting to 60 meetings
in total. For each meeting, we generate its meeting summary based on four
methods: AdaCentralized, AdaFedAvg, AdaFedKD and AdaFedSelectKD. Each
evaluator is provided with the meeting and a pair of summaries generated based
on AdaFedSelectKD and another method respectively, in random order. Ehe
evaluator determines which summary is better (wins) or decides a tie between
the two summaries according to domain expertise, informativeness and factual
correctness of summaries.

We count the number of wins, ties and losses for each method, with the average
results across the three evaluators (Figure 4). These observations point to
the conclusion that our method exhibits an impressive win rate of up to 87%
vis-à-vis the baseline AdaFedAvg method. It is noteworthy that AdaFedSelectKD
achieves competitive results compared with the strong AdaCentralized, with a
43% win rate. Additionally, the comparison with AdaFedKD also proves the
necessity of our designed selective strategy.

###  V-B Research Question 2

To answer the second research question “How well does the proposed
AdaFedSelecKD generalize? Can it achieve good performance under a variety of
settings, particularly under more severe non-IID data situations?”, we set up
various experimental settings to provide more comparisons.

####  V-B1 IID and Balanced Data Setting

Figure 5: Average ROUGE results based on the IID and balanced data setting,
where each client maintains meeting summarization data of the same
distribution (IID) and holds the same amount of data instances (balanced).

Under this setting, each client maintains meeting summarization data of the
same distribution (IID), with equal quantities distributed across the three
clients (balanced). First, we evenly divide the data of the three domains into
three parts respectively. Then, for the data of each domain, we distribute the
three divided parts into the three clients respectively, resulting in our IID
and balanced data setting, wherein each client holds one-third of the data in
each of the three domains. Subsequently, we conduct experiments leveraging
AdaFedAvg, AdaFedKD and AdaFedSelectKD based on this newly curated data. The
results (shown in Figure 5) are averaged over three random runs, with the data
being randomly re-divided for each run. We find that given the IID and
balanced data, all three clients demonstrate similar performance, with our
AdaFedSelectKD being more effective compared with other federated baselines.
Despite the effectiveness of both AdaFedKD and AdaFedSelectKD, we find they
contribute marginally under this setting. Our evidence reinforces the findings
of previous works that federated knowledge distillation methods excel at
overcoming the challenge of non-IID data learning but contribute little under
the IID data setting.

####  V-B2 Non-IID and Balanced Data Setting

Figure 6: Average ROUGE results based on the non-IID and balanced data
setting, where each client maintains domain-specific meeting summarization
data (non-IID) and holds the same amount of data instances (balanced).

Under this setting, each client maintains domain-specific meeting
summarization data (non-IID), with equal quantities distributed across the
three clients (balanced). Specifically, for each domain, we randomly select
200 training instances, 40 validation instances and 40 testing instances from
the corresponding QMSum portion, resulting in balanced data quantities across
the three clients. Subsequently, we conduct experiments using AdaFedAvg,
AdaFedKD and AdaFedSelectKD based on this newly curated balanced data. The
results (depicted in Figure 6) are averaged over three random runs, with data
randomly re-selected for each run. Firstly, it is evident that the ROUGE
results show varying degrees of decline due to the reduction of data quantity
relative to the full amount of data. Secondly, under this well-formed setting,
the utility of our AdaFedSelectKD is more fully exploited, with over 1 point
ROUGE improvement directly compared with AdaFedKD. We attribute this to the
fact that the balanced data setting, which facilitates a consistent parameter
optimization process on the client side, thus generating stable global
parameters that allow our entropy-based selective strategy to make reliable
distillation decisions. Thirdly, the results reveal that despite having the
same amount of data across all three clients, the committee client attains
superior performance. As indicated in Table II, this can be attributed to the
committee meeting’s fewer turns and reduced input tokens, making it easier to
train an effective summarizer.

####  V-B3 Extreme Non-IID and Unbalanced Data Setting

Figure 7: Average ROUGE results based on the extreme non-IID and unbalanced
data setting. “Learning without SAMSum client” means the original three
clients participate in the federated learning process while “Learning with
SAMSum client” means there are four clients in total with the newly-added
SAMSum client joining the learning process.

To further verify the effectiveness and robustness of our method, we set up an
extreme non-IID and unbalanced data distribution setting to assess the
performance of different methods. To this end, we employ the SAMSum dialogue
summarization dataset [31] and establish the fourth client, which will
participate in the federated learning process along with the previous three
clients. Specifically, SAMSum is a widely-used dataset for the dialogue
summarization task, which is vastly different from QMSum. The number of
instances (more than 16000 instances), topics (in various scenes of real
life), the length of the dialogue (120 tokens on average), the length of the
summary (23 tokens on average) and the number of turns (11 turns on average)
all differ greatly from QMSum. We conduct experiments leveraging AdaFedAvg,
AdaFedKD and AdaFedSelectKD. The results are shown in Figure 7. Firstly,
according to Figure 7(a), we find the previous three clients do not benefit
from the newly introduced SAMSum client and actually perform worse. This is in
line with the previous conclusion that the federated averaging algorithm has
severe limitations in the presence of non-IID data. Secondly, on the contrary,
knowledge distillation-based federated learning algorithms exhibit their
advantages under this challenging setting, with improvements regarding the
ROUGE score, as shown in Figure 7(a) and (b). Thirdly, it is worth noting that
AdaFedSelectKD achieves the best results, demonstrating its robustness and
efficacy when dealing with extreme non-IID data.

####  V-B4 Client Sampling Setting

Figure 8: Average ROUGE score based on the client sampling setting. “Sample 2
clients” means that during each round of learning, 2 clients are randomly
selected to participate in the learning process.

In this scenario, we simulate a more pragmatic federated learning environment,
in which only a subset of clients participate in each round of the learning
process. Specifically, we set the participation rate to 70%, meaning that in
our setup, two clients are randomly chosen to participate in the learning
procedure during each round. The results are illustrated in Figure 8. Firstly,
the experiment shows that utilizing all three clients — academic, committee
and product — leads to better performance than learning from only two clients.
Secondly, our proposed method AdaFedSelectKD consistently and stably
outperforms the other baseline methods.

Figure 9: Part-of-speech tag distribution for academic target summary words
learned (a) using knowledge distillation loss and (b) without using knowledge
distillation loss.

###  V-C Research Question 3

To answer the third research question “How does the proposed selective
knowledge distillation strategy work specifically and what are the underlying
mechanisms?”, we assess the training process by examining whether appropriate
target summary words are learned using knowledge distillation and their part-
of-speech (POS) tag distribution.

For each client, we extract all target summary words trained from the first
round until the final, optimally-performing round to determine whether each
summary word adopts knowledge distillation loss and its POS tag information.
For example, at the first client, suppose there is only one training instance
with the corresponding summary “project manager decides to use lcd”, and after
two rounds of training, the model achieves its best performance with three
target summary words: manager (once), decides (twice). These are optimized
using the knowledge distillation loss. Therefore, the proportion of words
optimized via knowledge distillation is $3/(2\times 6)=25\%$, where 2 is the
number of learning rounds and 6 is the number of words in the sentence.

Accordingly, our statistics show that knowledge distillation loss is 86%, 88%,
and 89% utilized in generating the target summary words during the training
process for the academic, committee, and product clients, respectively.
Furthermore, we calculate the part-of-speech tag distribution of target
summary words learned with and without knowledge distillation. Figure 9
illustrates the outcomes for the academic client, while the other two clients
exhibit similar distributions. The observations suggest that nouns and verbs
constitute nearly all of the words optimized through knowledge distillation
loss, aligning with the intuition that both nouns and verbs are essential for
articulating the core and domain-specific ideas of the meetings. In contrast,
determiners and pronouns make up 74% of all words learned without using
knowledge distillation loss.

TABLE VI: Meeting summaries from our three domains generated by different
methods.

Method | Meeting Summaries
---|---
Academic
AdaFedAvg |  PhD C explained that there were various delays with different components along the processing chain.
1-2 AdaFedSelectKD |  PhD C said that there were delays of 100ms for silence, 40ms at the input, and 10ms from LDA filters.
1-2 Gold Standard |  PhD C explained that the silence probabilities had a 100ms delay, the delta at the input had a 40ms delay, and a 10ms delay was created by LDA filters.
Committee
AdaFedAvg |  The governing organization developed the framework and simplified the options for resolving issues outside of legal proceedings.
1-2 AdaFedSelectKD |  The National Police Chiefs’ Council streamlined out-of-court disposals by developing the police approach.
1-2 Gold Standard |  The National Police Chiefs’ Council was responsible for developing the police approach to out-of-court disposals and simplifying the range of out-of-court disposals.
Product
AdaFedAvg |  Industrial Designer proposed an eco-friendly option but Project Manager agreed more with the commercially-appealing proposal.
1-2 AdaFedSelectKD |  The Industrial Designer suggested solar panel and rechargeable batteries but the Project Manager preferred Project Manager’s cradle idea.
1-2 Gold Standard |  Industrial Designer proposed to incorporate a solar panel and rechargeable batteries, but Project Manager agreed more with Marketing’s proposal to include a cradle.

Figure 10: Averaged ROUGE results based on the noun–verb-based hard knowledge
distillation and our proposed selective knowledge distillation strategy.

This distribution insight on the target summary words elicits a natural
follow-up: “How effective would it be to use knowledge distillation by
absolute means only when learning target nouns and verbs?”. To address this
question, we conduct experiments by exclusively adopting knowledge
distillation rigorously upon learning nouns and verbs. Figure 10 shows the
results. We can clearly find that our AdaFedSelectKD, which adaptively makes
the distillation decision exhibits superior performance compared to the hard
distillation method, demonstrating the necessity of our designed selective
strategy.

###  V-D Case Study

Table V-C illustrates summaries of meetings from three domains generated using
various methodologies. We observe that the baseline method, AdaFedAvg,
consistently produces generic meeting summaries lacking in detailed
information. In contrast, our proposed method, AdaFedSelectKD, yields
summaries that are more informative and tailored to the domain. Moreover, the
gold standard meeting summaries continue to demonstrate advantages in
conciseness and informativeness, highlighting the challenges intrinsic to
meeting summarization.

##  VI Related Work

###  VI-A Meeting Summarization

Meeting summarization [32, 2, 3] aims to pack crucial information of a given
meeting into a concise yet comprehensive summary highlighting the most salient
points. In addition to the challenges inherent in traditional summarization
tasks, meeting summarization must address unique difficulties arising from its
multi-participant nature. To facilitate progress in this domain, various
datasets have been curated [33, 34, 6, 9, 10], enabling the development of
state-of-the-art models that incorporate versatile knowledge [4, 5, 35, 36,
37, 7, 38] and achieve the best results. However, privacy concerns, which are
inextricably intertwined with meeting content, have received little attention
in the literature, hampering real-world application. Lee and Sogaard [8] take
the initiative to address this issue by exploring differential privacy (DP)
[39] for meeting summarization, focusing primarily on a single domain. In this
work, we conduct the first systematic study of meeting summarization under the
federated learning framework, accounting for the heterogeneity and unbalance
of data across multiple domains.

###  VI-B Federated Learning

Federated learning enables collaborative machine learning without the
centralized collection of potentially sensitive raw data, thereby paving the
way for stronger privacy guarantees when building predictive models [19]. With
mounting concerns regarding privacy issues, this paradigm has garnered
significant research interest including diverse research directions [13]. In
particular, owing to the inevitable inclusion of private information in texts,
a variety of studies have explored diverse natural language processing tasks
within the federated learning framework [14]. The predominant efforts in this
realm have focused on natural language understanding tasks, such as spoken
language understanding [40] and text classification [41]. Recent years have
witnessed a trend toward applying the federated learning framework to natural
language generation tasks [42]. Our work follows this line of work and is the
first to explore the federated multi-domain meeting summarization task.

###  VI-C Parameter-efficient Fine-tuning

The field of natural language processing is currently dominated by large
language models [43]. Despite their superiority, fine-tuning all the
parameters of these immense models on various downstream tasks becomes
prohibitively complicated as both model size and number of tasks increase
[44]. To alleviate this problem, parameter-efficient fine-tuning is coming to
the rescue by updating only a small number of extra parameters while keeping
most pre-trained parameters frozen [15, 45]. Opportunely, such lightweight
alternatives are well-suited for reducing communication costs in the federated
learning framework. Based on this foundation, the amalgamation of federated
learning and parameter-efficient fine-tuning unveils vast potential for
diverse applications [46]. In this paper, we craft two types of adapter
modules, a global adapter and a local adapter, which collaboratively and
efficiently facilitate federated client–server communication.

###  VI-D Knowledge Distillation

Knowledge distillation refers to the process of transferring knowledge from a
teacher model to a student model without significant performance degradation.
It has proven to be an effective method for improving model performance [26].
In recent years, knowledge distillation has been applied in the federated
learning framework and has demonstrated its ability to mitigate the effects of
data heterogeneity [20]. Our proposed framework builds upon this and takes one
step further to explore the combination of both knowledge distillation and
parameter-efficient fine-tuning while introducing one carefully designed
selective strategy to enable an adaptive learning process.

##  VII Limitations and Potential Advancements

The aforementioned experiments are conducted using the dataset provided by the
research community and based on a simulated federated environment. This may
not fully reflect the complexities of real-world scenarios and could
potentially lead to two limitations:

1\. Well-curated dataset exhibits less variability. In reality, meetings, even
those within the same domain, vary with respect to their participants,
discussion topics, and duration. Moreover, meeting transcripts are typically
generated via automatic speech recognition (ASR) systems, resulting in noisy,
imperfect textual data. Useful forms of meeting summaries also depend on the
target user needs, spanning full-text summaries, highlighted extracts,
identification of action items, and more. While our research attempts to
address real-world limitations, the dataset employed in our experiment likely
does not adequately capture the complexities of real-world meeting settings.
We envision that future advancements can develop more appropriate benchmarks,
design more comprehensive experimental paradigms, and possibly even
collaborate with corporations to narrow the divide between research
explorations and practical real-world applications.

2\. Simulated federated environment lacks uncertainty. In practical federated
learning deployments, addressing the challenges of non-IID and unbalanced data
is still insufficient. It is also imperative to overcome various issues
stemming from communication uncertainties, such as client–server latency,
asynchronous client learning updates, and client dropout. These factors are
unaddressed in the current research, and present challenges to any federated
learning techniques. As such, future advancements could conduct larger-scale
decentralized experiments in which the federated learning procedure is
intentionally perturbed to stress-test such a framework’s robustness to
uncertainty, thereby approximating real-world conditions more closely.

##  VIII Conclusion and Future Work

We examine the multi-domain meeting summarization task under a federated
learning paradigm. We show that this represents a more pragmatic and realistic
configuration than prior work on learning meeting summary models over
centralized meeting data. Moreover, to mitigate two challenges, namely limited
server–client communication and the non-IID data learning situation, we
propose a unified method, AdaFedSelectKD, which succeeds in reducing
communication costs and addressing the domain drift problem of the client
model. Through comprehensive empirical studies, our method demonstrates its
effectiveness and robustness that can achieve comparable results with
centralized training methods while exhibiting its superiority in handling the
intricacies of non-IID data.

We believe that future work will strive to apply the proposed method to real
scenarios. In our own work, we aim to craft data resources and design
experimental settings to adequately simulate real-world federated learning
circumstances. We plan to collaborate with organizations to implement the
federated learning framework and evaluate our proposed method in addressing a
variety of exigencies. Further investigation will build from this foundation,
to incorporate differential privacy techniques to further augment our model’s
privacy preservation characteristics. These advances promise to make
AdaFedSelectKD a practical solution for meeting summarization.

## Acknowledgments

This work was supported by National Key R&D Program of China via grant
2020AAA0106502, National Natural Science Foundation of China (NSFC) via grant
62276078, Key R&D Program of Heilongjiang via grant 2022ZX01A32 and the
International Cooperation Project of PCL, PCL2022D01.

## References

  * [1] S. Banerjee, P. Mitra, and K. Sugiyama, “Generating abstractive summaries from meeting transcripts,” _Proceedings of the 2015 ACM Symposium on Document Engineering_ , 2015.
  * [2] L. P. Kumar and A. Kabiri, “Meeting summarization: A survey of the state of the art,” _ArXiv preprint_ , 2022.
  * [3] V. Rennard, G. Shang, J. Hunter, and M. Vazirgiannis, “Abstractive meeting summarization: A survey,” _ArXiv preprint_ , 2022.
  * [4] X. Feng, X. Feng, B. Qin, and X. Geng, “Dialogue discourse-aware graph model and data augmentation for meeting summarization,” in _Proc. of IJCAI_ , 2020\.
  * [5] C. Zhu, R. Xu, M. Zeng, and X. Huang, “A hierarchical network for abstractive meeting summarization with cross-domain pretraining,” in _Proc. of EMNLP Findings_ , 2020.
  * [6] M. Zhong, D. Yin, T. Yu, A. Zaidi, M. Mutuma, R. Jha, A. H. Awadallah, A. Celikyilmaz, Y. Liu, X. Qiu, and D. Radev, “QMSum: A new benchmark for query-based multi-domain meeting summarization,” in _Proc. of NAACL_ , 2021\.
  * [7] Z. Liu and N. F. Chen, “Dynamic sliding window modeling for abstractive meeting summarization,” in _Proc. of Interspeech_ , 2022.
  * [8] S. Lee and A. Sogaard, “Private meeting summarization without performance loss,” 2023.
  * [9] Y. Hu, T. J. Ganter, H. Deilamsalehy, F. Dernoncourt, H. Foroosh, and F. Liu, “Meetingbank: A benchmark dataset for meeting summarization,” _ArXiv preprint_ , 2023.
  * [10] H. Wu, M. Zhan, H. Tan, Z. Hou, D. Liang, and L. Song, “Vcsum: A versatile chinese meeting summarization dataset,” _ArXiv preprint_ , 2023.
  * [11] H. McCosker, A. Barnard, R. Gerber _et al._ , “Undertaking sensitive research: Issues and strategies for meeting the safety needs of all participants,” in _Forum Qualitative Sozialforschung/Forum: Qualitative Social Research_ , 2001.
  * [12] A. Jain, L. Hong, and S. Pankanti, “Biometric identification,” _Communications of the ACM_ , 2000.
  * [13] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” _arXiv: Artificial Intelligence_ , 2019.
  * [14] M. Liu, S. Ho, M. Wang, L. Gao, Y. Jin, and H. Zhang, “Federated learning meets natural language processing: A survey,” _ArXiv preprint_ , 2021.
  * [15] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” in _Proc. of ICML_ , 2019.
  * [16] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in _Proc. of ACL_ , 2020.
  * [17] I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” _ArXiv preprint_ , 2020.
  * [18] J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” _International Journal of Computer Vision_ , 2020.
  * [19] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Proc. of AISTATS_ , 2017.
  * [20] A. Mora, I. Tenison, P. Bellavista, and I. Rish, “Knowledge distillation for federated learning: a practical guide,” _ArXiv preprint_ , 2022.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proc. of NeurIPS_ , 2017.
  * [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. of CVPR_ , 2016.
  * [23] J. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _ArXiv preprint_ , 2016.
  * [24] D. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei, “StableMoE: Stable routing strategy for mixture of experts,” in _Proc. of ACL_ , 2022.
  * [25] Y. He, Y. Chen, X. Yang, H. Yu, Y.-H. Huang, and Y. Gu, “Learning critically: Selective self-distillation in federated learning on non-iid data,” _IEEE Transactions on Big Data_ , 2022.
  * [26] G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _ArXiv preprint_ , 2015.
  * [27] D. Qi, H. Zhao, and S. Li, “Better generative replay for continual federated learning,” _ArXiv preprint_ , 2023.
  * [28] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _Proc. of ICML_ , 2016.
  * [29] L. van der Poel, R. Cotterell, and C. Meister, “Mutual information alleviates hallucinations in abstractive summarization,” in _Proc. of EMNLP_ , 2022\.
  * [30] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in _Text Summarization Branches Out_ , 2004.
  * [31] B. Gliwa, I. Mochol, M. Biesek, and A. Wawer, “SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization,” in _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , 2019\.
  * [32] X. Feng, X. Feng, and B. Qin, “A survey on dialogue summarization: Recent advances and new frontiers,” _ArXiv preprint_ , 2021.
  * [33] J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal _et al._ , “The ami meeting corpus: A pre-announcement,” in _International workshop on machine learning for multimodal interaction_ , 2005.
  * [34] A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke _et al._ , “The icsi meeting corpus,” in _Proc. of ICASSP_ , 2003.
  * [35] K. Riedhammer, B. Favre, and D. Z. Hakkani-Tür, “A keyphrase based approach to interactive meeting summarization,” _2008 IEEE Spoken Language Technology Workshop_ , 2008.
  * [36] M. Zhong, Y. Liu, Y. Xu, C. Zhu, and M. Zeng, “Dialoglm: Pre-trained model for long dialogue understanding and summarization,” in _Proc. of AAAI_ , 2022\.
  * [37] Y. Zhang, A. Ni, T. Yu, R. Zhang, C. Zhu, B. Deb, A. Celikyilmaz, A. H. Awadallah, and D. Radev, “An exploratory study on long dialogue summarization: What works and what’s next,” in _Proc. of EMNLP Findings_ , 2021.
  * [38] Y. Zhang, A. Ni, Z. Mao, C. H. Wu, C. Zhu, B. Deb, A. Awadallah, D. Radev, and R. Zhang, “Summn: A multi-stage summarization framework for long input dialogues and documents,” in _Proc. of ACL_ , 2022.
  * [39] C. Dwork and A. Roth, “The algorithmic foundations of differential privacy,” _Found. Trends Theor. Comput. Sci._ , 2014.
  * [40] Z. Huang, F. Liu, and Y. Zou, “Federated learning for spoken language understanding,” in _Proc. of COLING_ , 2020.
  * [41] Z. Li, S. Si, J. Wang, and J. Xiao, “Federated split bert for heterogeneous text classification,” _2022 International Joint Conference on Neural Networks (IJCNN)_ , 2022.
  * [42] B. Y. Lin, C. He, Z. Ze, H. Wang, Y. Hua, C. Dupuy, R. Gupta, M. Soltanolkotabi, X. Ren, and S. Avestimehr, “FedNLP: Benchmarking federated learning methods for natural language processing tasks,” in _Proc. of ACL Findings_ , 2022.
  * [43] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in _Proc. of NeurIPS_ , 2020.
  * [44] J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards a unified view of parameter-efficient transfer learning,” in _Proc. of ICLR_ , 2022.
  * [45] X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” in _Proc. of ACL_ , 2021.
  * [46] H. Zhao, W. Du, F. Li, P. Li, and G. Liu, “Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning,” _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022.