# Improved optimization strategies for deep Multi-Task Networks

Lucas Pascal, Pietro Michiardi, Xavier Bost, Benoit Huet, and Maria A. Zuluaga

###### Abstract

In Multi-Task Learning (MTL), it is a common practice to train multi-task
networks by optimizing an objective function, which is a weighted average of
the task-specific objective functions. Although the computational advantages
of this strategy are clear, the complexity of the resulting loss landscape has
not been studied in the literature. Arguably, its optimization may be more
difficult than a separate optimization of the constituting task-specific
objectives. In this work, we investigate the benefits of such an alternative,
by alternating independent gradient descent steps on the different task-
specific objective functions and we formulate a novel way to combine this
approach with state-of-the-art optimizers. As the separation of task-specific
objectives comes at the cost of increased computational time, we propose a
random task grouping as a trade-off between better optimization and
computational efficiency. Experimental results over three well-known visual
MTL datasets show better overall absolute performance on losses and standard
metrics compared to an averaged objective function and other state-of-the-art
MTL methods. In particular, our method shows the most benefits when dealing
with tasks of different nature and it enables a wider exploration of the
shared parameter space. We also show that our random grouping strategy allows
to trade-off between these benefits and computational efficiency.

###### Index Terms:

Multi-Task Learning, Optimization, Deep neural networks.

##  1 Introduction

Multi-Task Learning (MTL) consists in jointly, rather than individually,
learning multiple tasks with the goal of improving generalization performance.
This is done by training a shared model for all tasks [2]. In deep MTL, the
shared model consists in the parameters of a deep network.

The optimization objective in MTL is formulated as the combination into a
single aggregated objective function, in the form of a weighted sum of all the
task-specific objective functions [2], which the shared model is trained to
minimize. This formulation is particularly convenient for deep MTL. First, it
is computationally efficient, since a single gradient descent step (i.e. one
forward and one backward propagation) optimizes the shared parameters w.r.t.
every task, with the only extra computational cost coming from the task-
specific parts of the network. Second, in the case of convex task-specific
objective functions, it guarantees convergence to a minima, since the average
objective function remains convex.

Despite its simplicity, computational efficiency and wide adoption [9, 11, 15,
20, 22, 27, 28], the use of such an objective function can be problematic due
to the complex loss landscapes of deep networks. In fact, the landscape
complexity increases exponentially with the depth of the network [17]. As a
result, the network parameter update can be sub-optimal when the task
gradients conflict, or when one task dominates because of a much higher
gradient magnitude w.r.t. the other tasks [35]. This has led researchers to
investigate alternative methods to balance the task-specific objectives using
adaptable task-specific weights [3, 5, 10, 13, 20] or by altering the gradient
of the objective function [4, 36] in cases of conflicts between different
tasks.

Partitioning methods [1, 26, 29, 34] are a recent set of works that propose to
circumvent the problems associated to the optimization of MTL’s aggregated
objective function by using task-specific partitioning of the parameters. They
aim to reduce the occurrence of conflicts between tasks by letting each task
adapt the shared representation of the network with a few task-specific
weights. By construction, these methods optimize each task-specific objective
function alternately and independently, which is not equivalent to optimizing
the average sum of the different task-specific objectives, i.e. the aggregated
loss. Despite promising results, the difference between the two schemes has
never been studied, nor are there theoretical guarantees that this strategy
can lead to similar results as the aggregated loss one. This is in part can be
explained by the fact that the optimization scheme of partitioning methods is
always coupled with parameter partitioning, making it difficult to evaluate
which part of the contribution is due to the partitioning, and which one is
due to the optimization scheme.

Motivated by the promising results from recent works [1, 26, 29, 34], in this
paper we propose to study the task-specific alternate and independent
optimization used in partitioning methods and we formulate improvements to it
by making the following contributions. 1) We prove that this optimization
strategy provides convergence guarantees similar to the aggregated loss
optimization for stochastic gradient descent in the convex case, and we
provide the associated convergence bound. 2) We show that current alternated
optimization schemes do not operate truly independent task-specific update
steps, due to the momentum mechanisms of most of the existing optimizers,
which memorize previous updates, thus bringing them into the current learning
steps. We therefore propose a novel alternated optimization scheme that
performs truly independent task-specific update steps with momentum-based
state-of-the-art optimizers. 3) To account for the losses in computational
efficiency that alternate independent updates incur into, we introduce a task
grouping strategy to reduce training time when dealing with a high number of
tasks. Using three well-known benchmark datasets, we validate our work and
highlight our contributions by demonstrating that the proposed optimization
strategies consistently lead to substantial improvements in terms of both
generalization performance and benchmark results over existing state-of-the-
art methods.

##  2 Related work

Our work focuses on the study of the alternate and independent optimization of
task-specific objective functions used in partitioning methods, which we
review in the following. As most MTL methods rely on the use of an aggregated
objective function, we provide a brief overview of such approaches. We refer
the reader to [7, 35] for a broader review of this family of methods.

Aggregated loss. Most MTL works optimize an aggregated objective function
containing all the task-specific objective functions [2] and they develop
different strategies to mitigate the problems associated to the complexity of
the loss. A first family of methods uses dynamic loss weighting strategies to
control the influence of tasks in the main objective function and to account
for the learning dynamics of the different tasks [3, 10, 13, 20]. Other works
formulate the problem as a multi-objective optimization task [8], which
converges to a Pareto optimal solution, from which no task can be improved
without hurting another one [19, 31]. This greedy strategy can be particularly
sensitive to strongly interfering tasks, quickly converging into poor minima
[29]. A more recent line of works proposes to modify the task-specific
gradients before averaging them, when there are conflicting gradient
directions, under the hypothesis that they can be destructive for some tasks
[4, 36]. Although some of these works provide a better convergence in specific
conflicting regions of the loss landscape, the guarantees are very local and
become of relative importance in the case of complex non-convex loss
landscapes, for which a wider exploration of the parameter space might be
preferred over greedy strategies. From this perspective, in this work we favor
the optimization strategies adopted in partitioning methods, as they do not
make any assumptions that are detrimental to learning and they avoid greedy
improvements.

Partitioning methods. These works define task-specific parameter partitioning,
to allow tasks specify their own usage of the shared parameters, while
potentially reducing cases of task interference [1, 26, 29, 34]. They
alternately compute independent forward and backward passes for each task
according to the corresponding task-specific partitioning, and alternate these
optimization steps among the different tasks. The partitioning itself can then
be of different kinds. In [26], the authors use task-specific Squeeze and
Excitation blocks to create a real-valued partitioning. Other works apply a
binary partitioning to the parameters, either by training the partition [1]
with the Gumbel Softmax trick [12, 23], by fixing static partitions [34] or by
making the partitioning randomly evolve [29]. Similar strategies are used in
[24, 25], without involving any shared weights optimization, since the
partitioning is applied to a frozen pre-trained network. Regarding the
optimization strategy, only [26] discusses it explicitly as a way to exploit
the benefits of partitioning. Therefore, it has not been studied in isolation
in any previous work. Moreover, these works do not consider that moving
average mechanisms (e.g. momentum) included in state-of-the-art optimizers can
mix previous gradient descent directions from different tasks. This means that
the individual objective functions are only partially separated. Our work
proposes a full separation of the individual objective functions, by defining
individual moving average mechanisms for every task. To account for the
incurred computational overhead of such methods, we propose a random task
grouping as a trade-off between the benefits of alternate and independent
task-specific optimization and computational efficiency.

##  3 Alternate and independent optimization of task-specific objective
functions

We first formalize the standard MTL optimization setup using an aggregated
loss before our contributions.

###  3.1 Standard MTL optimization with aggregated loss

Let $\xi_{t}$ be an input data instance randomly sampled during optimization
step $t$, and $k$ the index of the $N$ tasks. We define the aggregated multi-
task objective function to minimize as

$\displaystyle F(w_{t},\xi_{t})=\sum_{k=1}^{N}c^{(k)}\cdot
F^{(k)}(w_{t},\xi_{t}),$ (1)

where $F^{(k)}$ is the objective function associated to task $k$ using shared
parameters $w_{t}$, and $c^{(k)}$ are the task-specific weighting
coefficients. For the sake of simplicity in the notation, but without loss of
generality, we consider here a uniform weighting of the different task
objective functions, i.e. $c^{(k)}=1$. Hereinafter, we use superscripts to
index elements associated to different tasks, and subscripts to index the
steps of the optimization process.

When using stochastic gradient descent (SGD) to optimize Eq. 1, the shared
parameters $w$ at step $t+1$ are updated according to the following rule:

$\displaystyle w_{t+1}=w_{t}-\eta_{t}\sum_{k=1}^{N}\frac{\partial}{\partial
w_{t}}F^{(k)}\left(w_{t},\xi_{t}\right),$ (2)

where $\eta_{t}$ is the learning rate. For the ease in notation, let us denote
$g^{(k)}(w_{t},\xi_{t})$ the derivative of $F^{(k)}$ w.r.t. the parameters
$w_{t}$. Eq. 2 can be rewritten as

$\displaystyle
w_{t+1}=w_{t}-\eta_{t}\sum_{k=1}^{N}g^{(k)}\left(w_{t},\xi_{t}\right).$ (3)

In the remaining of the document, we will refer to this optimization strategy
with an aggregated loss as MTL with Shared Update Steps (MTL-SUS) to reflect
the characteristics of its update rule (Eq. 3).

###  3.2 Alternate and independent optimization of task-specific objective
functions for SGD

We now formalize the SGD update step for the alternate and independent
optimization scheme used in state-of-the-art MTL partitioning works [1, 26,
29, 34] and we provide convergence bounds in the special case of convex
objective functions.

We define a multi-task update step as the alternate Individual Update Steps
(IUS) of each of the $N$ task-specific objective functions, and denote
$\smash{w_{t}^{(k)}}$ the parameters optimized w.r.t. task $k$, during the
$t$-th update step. The individual update step $t+1$ of task $k$ is:

$w_{t+1}^{(k)}=\begin{cases}w_{t}^{(N)}-\eta_{t}\cdot
g^{(k)}\left(w_{t}^{(N)},\xi_{t}\right),\quad k=1\\\\[10.0pt]
w_{t+1}^{(k-1)}-\eta_{t}\cdot
g^{(k)}\left(w_{t+1}^{(k-1)},\xi_{t}\right),\quad\forall\quad k>1.\end{cases}$
(4)

To reflect the properties of the update rule (Eq. 4), we denote this
optimization strategy MTL-IUS.

MTL holds analogies with the federated learning problem [16, 18, 32], where
each involved device accounts for one task objective function, with
communication after each update step. MTL-SUS corresponds to a full-device
participation, and MTL-IUS to a single device participation. Using this
analogy, we take inspiration from [18] to prove that alternate and independent
optimization schemes using the MTL-IUS rule converge to the optimum in the
case of convex objective functions.

######  Assumption 1.

$F^{(1)},...,F^{(N)}$ are all $L$-smooth: for all $v$, $w$, $F^{(k)}(v)\leq
F^{(k)}(w)+(v-w)^{T}\nabla F^{(k)}(w)+\frac{L}{2}||v-w||_{2}^{2}$.

######  Assumption 2.

$F^{(1)},...,F^{(N)}$ are all $\mu$-strongly convex: for all $v$, $w$,
$F^{(k)}(v)\geq F^{(k)}(w)+(v-w)^{T}\nabla
F^{(k)}(w)+\frac{\mu}{2}||v-w||_{2}^{2}$.

######  Assumption 3.

Let $\xi_{t}$ be uniformly sampled from the data. The variance of stochastic
gradients is uniformly bounded: $\mathbb{E}||\nabla
F^{(k)}(w_{t},\xi_{t})-\nabla F^{(k)}(w_{t})||^{2}\leq{\sigma^{(k)}}^{2}$.

######  Assumption 4.

The expected squared norm of stochastic gradients is uniformly bounded:
$\mathbb{E}||\nabla F^{(k)}(w_{t},\xi_{t})||^{2}\leq G^{2}$.

Let us denote $F^{*}$ and ${F^{(k)}}^{*}$ as the minima values of $F$ and
$F^{(k)}$ respectively, and $w^{*}$ as the minimizer of $F$. We thus define

$\Gamma=F^{*}-\frac{1}{N}\sum_{k=1}^{N}{F^{(k)}}^{*},$

which can be considered as the heterogeneity of the different task-specific
objective functions.

######  Theorem 1.

Let $\gamma\geq 2\frac{L}{\mu}-1$ and $\eta_{t}=\frac{2}{\mu(\gamma+t)}$ the
learning rate. Under assumptions 1, 2, 3 and 4, the optimization scheme MTL-
IUS presented in Eq. (4) satisfies

$\mathbb{E}[F(w_{T})]-F^{*}\leq\frac{L}{\gamma+T}\left(\frac{2B}{\mu^{2}}+\frac{(\gamma+1)}{2}\mathbb{E}||w_{1}-w^{*}||^{2}\right),$
(5)

with $B=\sum_{k=1}^{N}p_{k}^{2}\sigma_{k}^{2}+2L\Gamma+G^{2}$.

Proof: See appendix 6 ∎

Theorem 1 guarantees the convergence of MTL-IUS to the optimum in
$\mathcal{O}(\frac{1}{T})$ with a decreasing learning rate for stochastic
gradient descent. However, it should be noted that the bound does not increase
directly with the number of tasks, but with the heterogeneity term $\Gamma$,
which corresponds to the average deviation between the optimum $F^{*}$ and the
values of the task-specific objective functions $\smash{{F^{(k)}}^{*}}$ at
that point.

###  3.3 Alternate and independent optimization of task-specific objective
functions for moving-average based optimizers

In most of deep learning’s literature and in current practice, more complex
optimizers than a regular SGD are preferred. State-of-the-art optimizers, such
as Adam [14], generally use exponential moving average mechanisms, e.g.
momentum, to smooth and regulate the optimization trajectory. These mechanisms
can partially compromise the independence of the task-specific update steps,
i.e. MTL-IUS, since every learning step includes descent directions of
previous tasks. In this section, we propose a new update rule that allows to
use Individual Optimizers (IO), providing improved optimization independence
to the different tasks.

MTL-IUS with momentum. We first reformulate the update step of MTL-IUS with a
momentum-based optimizer, as it is done in partitioning MTL works [1, 26, 29].

Figure 1: NYUv2 best averaged and task-specific validation losses for the
different tasks with respect to the learning rate $\eta$.

Let us define $\hat{m}$ a generic optimizer rule, which adjusts the gradient
descent direction based on exponential moving average mechanisms [14]. The
update step $t+1$ of task $k$ (Eq. (4)) can be then reformulated as:

$w_{t+1}^{(k)}=\begin{cases}w_{t}^{(N)}-\eta_{t}\cdot\hat{m}\left(g^{(k)}\left(w_{t}^{(N)},\xi_{t}\right)\right),\quad
k=1\\\\[10.0pt]
w_{t+1}^{(k-1)}-\eta_{t}\cdot\hat{m}\left(g^{(k)}\left(w_{t+1}^{(k-1)},\xi_{t}\right)\right),\quad\forall\quad
k>1,\par\end{cases}$ (6)

where $\hat{m}$ mixes $g^{(k)}$ with descent directions $g^{({j\neq i})}$ of
previous tasks update steps. This implies that using this rule does not allow
for fully independent task-specific update steps thanks to $\hat{m}$.

Individual Optimizers (MTL-IO). We propose to use individual exponential
moving averages for each task to allow for fully independent task-specific
update steps. We formulate this update rule, using Individual Optimizers
(IOs), as:

$w_{t+1}^{(k)}=\begin{cases}w_{t}^{(N)}-\eta_{t}\cdot\hat{m}^{(k)}\left(g^{(k)}\left(w_{t}^{(N)},\xi_{t}\right)\right),\quad
k=1\\\\[10.0pt]
w_{t+1}^{(k-1)}-\eta_{t}\cdot\hat{m}^{(k)}\left(g^{(k)}\left(w_{t+1}^{(k-1)},\xi_{t}\right)\right),\quad\forall\,\,k>1\end{cases}$
(7)

with $\hat{m}^{(k)}$ the exponential moving average mechanism of task $k$. We
denote this alternate and independent optimization strategy as MTL-IO to
reflect the properties of its update rule. With MTL-IO, the memory term
introduced by $\hat{m}^{(k)}$ only involves previous updates of task $k$. This
is equivalent to using one IO per task, which avoids the memory leakage that
MTL-IUS incurs into.

###  3.4 Mitigating computational costs through task grouping

Alternate and independent optimization schemes using MTL-IUS or MTL-IO require
a training step per task per input sample, i.e $N$ training steps in total.
This makes the inference and training time proportional to the number of
tasks. To compensate for the increased computational burden, we propose to
equally and randomly distribute the $N$ tasks among a set
$\smash{\mathcal{\hat{T}}=\\{\mathcal{\hat{T}}_{l}\\}_{l\in\\{1..\hat{N}\\}}}$
of $\smash{\hat{N}}$ super-tasks, with $\smash{\hat{N}\in\\{1..N\\}}$. The set
of super-tasks can be optimized with MTL-IUS or MTL-IO, whereas the individual
super-task gradients are expressed as the weighted sum of the gradients of
their constituting tasks:

$w_{t+1}^{(l)}=\begin{cases}w_{t}^{(\hat{N})}-\eta_{t}\cdot\hat{m}\left(\sum_{k\in\mathcal{\hat{T}}_{l}}g^{(k)}(w_{t}^{(\hat{N})},\xi_{t})\right),\quad
l=1\\\\[10.0pt]
w_{t+1}^{(l-1)}-\eta_{t}\cdot\hat{m}\left(\sum_{k\in\mathcal{\hat{T}}_{l}}g^{(k)}\left(w_{t+1}^{(l-1)}.\xi_{t}\right)\right),\quad\forall\,\,l>1\end{cases}$
(8)

For the sake of brevity, we only present the formulation adapted for MTL-IUS.
Derivation with MTL-IO is straightforward. One should note that this
formulation is equivalent to MTL-IUS (respectively, MTL-IO) when $\hat{N}=T$,
and to MTL-SUS when $\hat{N}=1$. Such task grouping thus defines an adjustable
compromise between tasks independence and training speed.

##  4 Experiments and results

We report four different experiments performed on three widely used datasets
for MTL. We first study the generalization performance of MTL-SUS, MTL-IUS and
MTL-IO on NYUv2 [33], with three heterogeneous tasks, and on the seven
homogeneous segmentation tasks of Cityscapes [6]. We then study how our
grouping strategy behaves on the $40$ tasks of Celeb-A [21]. We finally
provide insights about the shared parameter space exploration operated by the
studied optimization schemes.

In this study, our models are compared with state-of-the-art multi-task
methods, including three aggregated loss optimization methods, GradNorm [3],
MGDA [31], PCGrad [4], and one partitioning method, Maximum Roaming (MR) [29].
To ensure fairness of comparison, all baselines have been implemented in the
same pipeline, under Pytorch $1.2$, and run on Nvidia Titan-XP GPUs. For every
baseline compared on a dataset, we perform a grid search on the learning rate.
All the scripts we used to perform our experiments are available on
GitHub.111https://github.com/lucaspascal/AITSO_MTL.. The data used comes from
official releases for the three datasets.

###  4.1 Scene understanding on NYUv2

The NYUv2 dataset [33] provides close to $1500$ annotated indoor images
extracted from videos captured with the Microsoft Kinect in different
buildings. The small number of images and the complexity of the scenes makes
it particularly difficult and interesting for multi-task problems. We use it
here for one $13$-class semantic segmentation task, one depth estimation task,
and one normal estimation task. We judge this multi-task setting as
particularly challenging, since it is made of three tasks of different
natures.

All baselines use a U-Net [30], which is known to be particularly efficient
for dense labelling tasks. To maximize the proportion of shared weights in the
network, all encoders and decoders are shared by all the tasks, while the
task-specific prediction heads are made of a single $1\times 1$ convolutional
layer per task.

The image size is set to $288\times 384$ with a batch size of $8$. All models
are trained for $500$ epochs, and all performances are reported over $3$
random seeds. Reported metrics are mean Intersection over Union (mIoU) and
pixel accuracy for segmentation tasks, absolute and relative error for depth
estimation, and mean and median angle error for normal estimation.

To understand the general behavior of the compared optimization strategies
MTL-SUS, MTL-IUS and MTL-IO, we first study their generalization performance
w.r.t. the learning rate $\eta$, which has a direct influence on the
sensitivity of deep networks to minima in the loss landscape, and therefore on
their convergence. We report in Fig. 1 the best validation losses achieved by
MTL-SUS, MTL-IUS and MTL-IO over a full training, w.r.t. different values of
$\eta$. We made sure for every model that the validation loss reached its
minimum before the end of the training.

Figure 2: Cityscapes best averaged and task-specific validation losses w.r.t
the learning rate $\eta$.

We first observe that MTL-IO is by far the best overall performing method.
Above $\eta=5e^{-5}$, it substantially improves the losses of each of the
three tasks, which suggests that it is able to consistently reach better loss
regions in the shared parameter space. We also observe a smaller difference
between MTL-IUS and MTL-SUS. Specifically, MTL-IUS performs better for smaller
learning rates, and worse beyond $\eta=2e^{-4}$. It is also interesting to
note the overall distorted aspect of these curves, with high standard
deviations, while more "convex" shapes could be expected. This suggests that
the loss landscape is particularly complex, compared to that one of other
datasets (see Fig. 2,3).

We compare the models with the best validation losses for each method with the
state-of-the-art baselines (Table I). While MTL-IUS and MTL-SUS perform
similarly, MTL-IO achieves the best results overall on segmentation and depth
estimation metrics, and falls close behind MR for normals estimation, making
it the best performing multi-task method on this dataset.

| Segmentation | Depth estimation | Normals estimation |
---|---|---|---|---
|  mIoU
$(\uparrow)$ |  Pix. Acc.
$(\uparrow)$ |  Abs. Err.
$(\downarrow)$ |  Rel. Err.
$(\downarrow)$ |  Mean Err.
$(\downarrow)$ |  Med. Err.
$(\downarrow)$ |  Rank
$(\downarrow)$
MTL-SUS |  $24.29\newline (\pm 0.38)$ |  $48.79\newline (\pm 0.35)$ |  $68.48\newline (\pm 0.28)$ |  $52.93\newline (\pm 0.59)$ |  $27.28\newline (\pm 0.02)$ |  $33.70\newline (\pm 0.03)$ |  $4.33$
MTL-IUS |  $24.93\newline (\pm 0.33)$ |  $48.91\newline (\pm 0.17)$ |  $67.76\newline (\pm 0.61)$ |  $53.25\newline (\pm 0.77)$ |  $27.28\newline (\pm 0.05)$ |  $33.70\newline (\pm 0.05)$ |  $3.88$
MTL-IO |  $\mathbf{29.48}\newline \mathbf{(\pm 0.30)}$ |  $\mathbf{54.06}\newline \mathbf{(\pm 0.21)}$ |  $\mathbf{66.07}\newline \mathbf{(\pm 0.68)}$ |  $\mathbf{50.27}\newline \mathbf{(\pm 0.28)}$ |  $\mathbf{27.08}\newline \mathbf{(\pm 0.03)}$ |  $\mathbf{33.35}\newline \mathbf{(\pm 0.05)}$ |  $\mathbf{1.00}$
GradNorm |  $28.09\newline (\pm 0.85)$ |  $52.91\newline (\pm 0.74)$ |  $70.26\newline (\pm 1.4)$ |  $52.40\newline (\pm 1.35)$ |  $27.18\newline (\pm 0.05)$ |  $33.44\newline (\pm 0.07)$ |  $3.16$
PCGrad |  $24.51\newline (\pm 0.32)$ |  $48.87\newline (\pm 0.41)$ |  $67.89\newline (\pm 0.67)$ |  $52.29\newline (\pm 0.75)$ |  $27.32\newline (\pm 0.04)$ |  $33.65\newline (\pm 0.08)$ |  $3.50$
MR |  $28.82\newline (\pm 0.18)$ |  $\mathbf{53.77}\newline \mathbf{(\pm 0.61)}$ |  $71.30\newline (\pm 0.97)$ |  $53.37\newline (\pm 0.97)$ |  $\mathbf{27.03}\newline \mathbf{(\pm 0.02)}$ |  $\mathbf{33.27}\newline \mathbf{(\pm 0.03)}$ |  $2.66$
TABLE I: NYUv2 results with standard deviations. The best results (and
statistically equivalent) per column are highlighted in each category. Methods
ranks are computed by averaging the rank of the methods on each metric.

###  4.2 Multi-class segmentation on Cityscapes

The Cityscapes dataset [6] provides $5$K fine-grained annotated street-view
images captured from a car point of view, in many different German cities. We
use here the $7$ semantic segmentation classes as $7$ independent tasks, to
observe how the different optimization methods behave with more homogeneous
tasks. We use the same network as for NYUv2, and images are resized to
$128\times 256$ with a batch size of $24$. We conduct a study similar to
NYUv2, reported in Fig. 2, and a benchmark in Table II.

| Segmentation |
---|---|---
| mIoU $(\uparrow)$ | Pix. Acc. $(\uparrow)$ | Rank $(\downarrow)$
MTL-SUS | $\mathbf{69.79\pm 0.40}$ | $\mathbf{90.40\pm 0.08}$ | $\mathbf{1.0}$
MTL-IUS | $\mathbf{69.74\pm 0.05}$ | $90.22\pm 0.13$ | $2.5$
MTL-IO | $69.52\pm 0.26$ | $\mathbf{90.28\pm 0.14}$ | $2.5$
GradNorm | $69.00\pm 0.12$ | $90.17\pm 0.10$ | $5.0$
PCGrad | $\mathbf{69.73\pm 0.05}$ | $\mathbf{90.42\pm 0.06}$ | $\mathbf{1.0}$
MR | $69.15\pm 0.08$ | $89.79\pm 0.08$ | $5.5$
TABLE II: Cityscape results with standard deviations. The best results (and
statistically equivalent) per column are highlighted. Methods ranks are
computed by averaging the rank of the methods on each metric.

Here we observe smoother curves with smaller variations, which confirms our
intuition that this setting is easier to optimize. The three optimization
methods report very similar results. While the best overall validation loss is
reached by MTL-SUS, MTL-IUS and MTL-IO are consistently better in the lower
learning rates, suggesting that they are more robust to local minima in the
loss landscape. As one could expect given Fig. 2, all the methods, including
the state-of-the art benchmarks, present a very close performance. It seems
like in a simple setting with a well calibrated learning rate, the influence
of a particular optimization method is greatly reduced.

###  4.3 Multi-attribute segmentation on Celeb-A

The CelebA dataset [21] contains $200$K face images of $10$K different
celebrities. Annotations are provided for $40$ different facial attributes. In
this experiment, we study the proposed random grouping strategy with a large
number of tasks, to establish to which extent it can reduce the computational
overhead of MTL-IUS and MTL-IO without affecting their performance. We
consider each of the $40$ provided facial attributes as independent facial
attribute detection task, and operate different groupings on them,
$\hat{N}=\\{2,4,8,20,40\\}$.

We use a fully shallow $9$-layer convolutional network similar to [4], with a
task-specific fully-connected prediction layer for each task (or task group).
The image size is set to $64\times 64$, with a batch size of $256$. All models
are trained over $15$ epochs. We report the average classification error,
along with F1-score, since the different attributes are not balanced. Reported
results are averaged over $3$ seeds, with new task groups are sampled for
every seed. Figure 3 shows the best achieved validation losses over training
w.r.t. the learning rate and for different number of super-tasks
$\smash{\hat{N}}$. Table III summarizes the benchmark result.

For a certain number of super-tasks, i.e. $\smash{\hat{N}\leq 8}$, MTL-IUS and
MTL-IO are able to reach an improvement compared to MTL-SUS. Then, the
performance decreases up to $\smash{\hat{N}=40}$ (i.e. no grouping). This
suggests that the use of a grouping strategy is beneficial both in terms of
performance and computational efficiency when dealing with a large number of
tasks. The latter is further confirmed in Table III, where the best grouping
models of MTL-IUS and MTL-IO also outperform other state-of-the art methods on
both average error and F-score metrics.

| Avg. error $(\downarrow)$ | F1-score $(\uparrow)$ | Rank $(\downarrow)$
---|---|---|---
MTL-SUS | $9.20\pm 0.02$ | $68.66\pm 0.24$ | $6.0$
MTL-IUS ($2$ groups) | $9.16\pm 0.03$ | $68.69\pm 0.21$ | $-$
MTL-IUS ($4$ groups) | $9.13\pm 0.02$ | $69.01\pm 0.27$ | $-$
MTL-IUS ($8$ groups) | $\mathbf{9.10\pm 0.03}$ | $68.93\pm 0.38$ | $3.0$
MTL-IUS ($20$ groups) | $9.20\pm 0.05$ | $68.82\pm 0.11$ | $-$
MTL-IUS ($40$ groups) | $9.27\pm 0.05$ | $68.97\pm 0.17$ | $-$
MTL-IO ($2$ groups) | $9.15\pm 0.02$ | $68.85\pm 0.33$ | $-$
MTL-IO ($4$ groups) | $\mathbf{9.11\pm 0.02}$ | $\mathbf{69.48\pm 0.15}$ | $\mathbf{1.0}$
MTL-IO ($8$ groups) | $9.15\pm 0.01$ | $68.55\pm 0.09$ | $-$
MTL-IO ($20$ groups) | $9.23\pm 0.02$ | $68.61\pm 0.31$ | $-$
MTL-IO ($40$ groups) | $9.24\pm 0.00$ | $68.57\pm 0.04$ | $-$
GradNorm | $9.16\pm 0.03$ | $69.16\pm 0.19$ | $4.0$
PCGrad | $9.16\pm 0.04$ | $\mathbf{69.26\pm 0.30}$ | $2.5$
MGDA-UB | $9.42\pm 0.03$ | $67.68\pm 0.12$ | $7.0$
MR | $\mathbf{9.12\pm 0.00}$ | $\mathbf{69.21\pm 0.31}$ | $\mathbf{1.0}$
TABLE III: Celeb-A results with standard deviations. The best results (and
statistically equivalent) per column are highlighted. In parenthesis are
reported the different number of task groups for MTL-IUS and IO. The best ($4$
and $8$) and worst ($40$) performing numbers of task groups are reported for
both the strategies.

More generally, we observe a progressive evolution of the curves w.r.t. the
different values of $\hat{N}$: as with Cityscapes, the less we group tasks
together, the better is the optimization of IUS and IO in the low learning
rates, while it is the opposite in the high learning rates. This behavior
suggests that these methods introduce more noise in the optimization, making
them less sensitive to local minima, and pushing for a larger exploration of
the shared parameter space.

Figure 3: Celeb-A best averaged validation losses w.r.t the learning rate
$\eta$ and different numbers of task groups. (Left) MTL-IUS models compared to
MTL-SUS. (Right) MTL-IO models compared to MTL-SUS.

###  4.4 Covered distance

We provide insights about how the studied optimization strategies explore the
shared parameter space, by measuring the distances covered in this space
during training. Using the Frobenius norm, we measure the shortest path from
the network’s initialization to the loss minimum (shortest) and the total
covered distance (total), which is the sum of the distances covered at each
update step. We also report the total-to-shortest-distance ratio, which gives
an insight on how much the surrounding space has been explored for a given
shortest path.

Table IV summarizes the obtained measures during the optimization of MTL-SUS,
MTL-IUS and MTL-IO over all datasets. All three baselines start from the same
parameter initialization (i.e. same point in the parameter space) and use the
same learning rate (the one providing the best performances overall). The
measurements are performed at the same point in time (the closest from the
validation loss minimum). The results are averaged over 3 different seeds. As
a reminder, the aggregated multi-task loss uses uniform weights of $1$ for
every task, so that the gradient scaling is the same for the three
optimization strategies.

Dataset | Model | Total covered dist. | Shortest path | Ratio
---|---|---|---|---
NYUv2 | SUS | $879.33\pm 18.22$ | $38.77\pm 0.53$ | $22.68\pm 0.17$
IUS | $2597.74\pm 49.59$ | $67.44\pm 1.02$ | $38.52\pm 0.24$
IO | $2769.52\pm 40.20$ | $71.06\pm 0.80$ | $38.97\pm 0.22$
Cityscapes | SUS | $909.38\pm 10.02$ | $74.38\pm 0.80$ | $12.23\pm 0.01$
IUS | $8102.39\pm 13.16$ | $262.68\pm 0.48$ | $30.85\pm 0.02$
IO | $8438.78\pm 105.47$ | $277.29\pm 5.69$ | $30.44\pm 0.35$
Celeb-A | SUS | $1398.01\pm 3.54$ | $99.66\pm 0.22$ | $14.03\pm 0.00$
IUS (4 groups) | $5636.18\pm 19.59$ | $223.83\pm 0.69$ | $25.18\pm 0.02$
IO (4 groups) | $6560.56\pm 45.21$ | $242.55\pm 1.80$ | $27.05\pm 0.12$
IUS (8 groups) | $14955.87\pm 15.55$ | $387.13\pm 0.48$ | $38.63\pm 0.01$
IO (8 groups) | $13992.50\pm 268.21$ | $369.67\pm 5.39$ | $37.85\pm 0.17$
IUS (40 groups) | $65376.36\pm 164.54$ | $837.32\pm 1.91$ | $78.08\pm 0.08$
IO (40 groups) | $59627.39\pm 746.35$ | $764.42\pm 9.01$ | $78.00\pm 0.07$
TABLE IV: Total covered and shortest path distances in the shared parameter
space from a same initialization point.

We observe that both MTL-IUS and MTL-IO travel more, both in terms of total
covered and shortest path distances. The distances increase globally with the
number of tasks (or task groups). The increase in the shortest path distance
indicates that these methods are able to discover more distant loss regions
compared to MTL-SUS. These observations correlate with the findings from our
previous experiments, suggesting that some of these more distant regions can
provide better generalization performance.

Most interestingly, MTL-IUS and MTL-IO present a higher total-to-shortest path
distance ratio compared to MTL-SUS, which means that between two locations in
the shared parameter space, MTL-IUS and MTL-IO follow a much more oscillating
trajectory. This observation suggests that optimizing tasks-specific objective
functions, instead of an aggregated multi-task objective one, introduces more
stochasticity to the optimization, which is usually beneficial in deep
learning.

##  5 Conclusion

We have studied the alternate and independent optimization of the task-
specific objective functions (MTL-IUS) used in partitioning methods, and
compared it to the optimization of the more standard aggregated objective
function (MTL-SUS), which is widely adopted in the literature. Taking
inspiration from Federated Learning works, we first demonstrated that the
former has similar convergence properties as the latter when dealing with
convex objective functions. We then showed that the existing partitioning
methods do not operate truly independent update steps due to the momentum
mechanisms included in state-of-the-art optimizers. We thus have formulate a
novel strategy, using an independent optimizer per task (MTL-IO), that favors
task independence. To account for the computational overhead of these
strategies, we have also proposed a random grouping strategy, that strikes a
balance between computational efficiency, which has the merits of potentially
reducing the carbon footprint of the proposed methods, and the benefits of a
task-independent optimization strategy.

Our experimental results over three datasets show both MTL-IUS and MTL-IO
achieve an overall better generalization performance w.r.t standard aggregated
objective optimization (MTL-SUS) and state-of-the-art MTL baselines using an
aggregated objective function. In particular, MTL-IO shows important
improvements in settings where tasks are of a very different nature. Our
results also confirm that the proposed random grouping strategy applied to
MTL-IUS and MTL-IO is beneficial both in terms of performance and efficiency,
when dealing with a great number of tasks. Finally, we showed that MTL-IUS and
IO allow parameters updates to travel a longer distance, and ensure a more
thorough exploration of the shared parameter space.

While the optimization of an aggregated multi-task objective function is
widely adopted in the literature, our investigations suggest that its usage
should be questioned, despite the computational benefits it provides. We hope
this work will inspire further studies in this direction, as many existing
optimization methods of the aggregated multi-task objective function could be
adapted to task-specific update steps.

## References

  * [1] Felix J.S. Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel C. Alexander, and Jorge Cardoso.  Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels.  In The IEEE International Conference on Computer Vision (ICCV), pages 1385–1394, 2019.
  * [2] Rich Caruana.  Multitask Learning.  Machine Learning, 28(1):41–75, 1997.
  * [3] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich.  GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks.  In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 794–803, 2018.
  * [4] Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov.  Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout.  Advances in Neural Information Processing Systems, 33:2039–2050, 2020.
  * [5] S. Chennupati, G. Sistu, S. Yogamani, and S. A. Rawashdeh.  MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1200–1210, June 2019.  ISSN: 2160-7516.
  * [6] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The Cityscapes Dataset for Semantic Urban Scene Understanding.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3213–3223, 2016.
  * [7] Michael Crawshaw.  Multi-Task Learning with Deep Neural Networks: A Survey.  arXiv:2009.09796 [cs, stat], 2020.
  * [8] Jean-Antoine Désidéri.  Multiple-gradient Descent Algorithm (MGDA) for Multiobjective Optimization.  Comptes Rendus Mathematique, 350(5):313–318, 2012.
  * [9] Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L. Yuille.  NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3200–3209, 2019.
  * [10] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei.  Dynamic Task Prioritization for Multitask Learning.  pages 270–287, 2018.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [12] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical Reparameterization with Gumbel-Softmax.  arXiv:1611.01144 [cs, stat], 2017.
  * [13] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7482–7491, 2018.
  * [14] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  arXiv:1412.6980 [cs], 2017.
  * [15] Iasonas Kokkinos.  Ubernet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6129–6138, 2017.
  * [16] Jakub Konečný, Brendan McMahan, and Daniel Ramage.  Federated optimization:distributed optimization beyond the datacenter.  arXiv:1511.03575 [cs.LG], 2015.
  * [17] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.  Visualizing the Loss Landscape of Neural Nets.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [18] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang.  On the convergence of fedavg on non-iid data.  arXiv:1907.02189 [stat.ML], 2020.
  * [19] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong.  Pareto Multi-Task Learning.  In Advances in Neural Information Processing Systems 32, pages 12060–12070, 2019.
  * [20] Shikun Liu, Edward Johns, and Andrew J. Davison.  End-To-End Multi-Task Learning With Attention.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1871–1880, 2019.
  * [21] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In The IEEE International Conference on Computer Vision (ICCV), pages 3730–3738, 2015.
  * [22] Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris.  Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5334–5343, 2017.
  * [23] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh.  The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables.  arXiv:1611.00712 [cs, stat], 2017.
  * [24] Arun Mallya, Dillon Davis, and Svetlana Lazebnik.  Piggyback: Adapting a single network to multiple tasks by learning to mask weights.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 67–82, September 2018.
  * [25] Massimiliano Mancini, Elisa Ricci, Barbara Caputo, and Samuel Rota Bulo.  Adding new tasks to a single network with weight transformations using binary masks.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, September 2018.
  * [26] Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos.  Attentive Single-Tasking of Multiple Tasks.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1851–1860, 2019.
  * [27] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert.  Cross-stitch networks for multi-task learning.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3994–4003, 2016.
  * [28] Taylor Mordan, Nicolas Thome, Gilles Henaff, and Matthieu Cord.  Revisiting Multi-Task Learning with ROCK: a Deep Residual Auxiliary Block for Visual Detection.  In Advances in Neural Information Processing Systems 31, pages 1310–1322, 2018.
  * [29] Lucas Pascal, Pietro Michiardi, Xavier Bost, Benoit Huet, and Maria A. Zuluaga.  Maximum Roaming Multi-Task Learning.  arXiv:2006.09762 [cs.CV], 2020.
  * [30] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Lecture Notes in Computer Science, pages 234–241, Cham, 2015. Springer International Publishing.
  * [31] Ozan Sener and Vladlen Koltun.  Multi-Task Learning as Multi-Objective Optimization.  In Advances in Neural Information Processing Systems 31, pages 527–538, 2018.
  * [32] Reza Shokri and Vitaly Shmatikov.  Privacy-preserving deep learning.  In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, page 1310–1321, 2015.
  * [33] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor Segmentation and Support Inference from RGBD Images.  In European Conference on Computer Vision (ECCV) 2012, Lecture Notes in Computer Science, pages 746–760, 2012.
  * [34] Gjorgji Strezoski, Nanne van Noord, and Marcel Worring.  Many Task Learning With Task Routing.  In The IEEE International Conference on Computer Vision (ICCV), pages 1375–1384, 2019.
  * [35] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool.  Multi-task learning for dense prediction tasks: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021.
  * [36] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.  Gradient Surgery for Multi-Task Learning.  Advances in Neural Information Processing Systems, 33:5824–5836, 2020.

##  6 Proof of Theorem 1

In this section we prove the convergence of MTL-IUS for stochastic gradient
descent when dealing with convex objective functions (Theorem 1). We take
inspiration from the proof of convergence of the FedAvg algorithm [18] for
partial device participation in federated learning.

###  6.1 Problem setting and notations

For the ease of notation, in the following we drop the double indexing of $w$
(tasks and iterations) used in equation 4. We thus rewrite the IUS update rule
(i.e. equation 4) of the weights at update step $t+1$ as:

$\begin{cases}v_{t+1}^{k}=w_{t}-\eta_{t}\nabla
F^{(k)}(w_{t},\xi_{t}^{k}),\\\\[10.0pt] w_{t+1}=v_{t+1}^{s_{t}},\end{cases}$

where $v_{t+1}^{k}$ is the SGD from $w_{t}$ with respect to task $k$ objective
function, and $w_{t+1}$ only follows the SGD of one task $s_{t}$ (while others
are not considered), with $s_{t}$ selected in $\\{1,...,N\\}$, for every step
$t$. We prove here the theorem for $s_{t}$ randomly selected at uniform at
each iteration $t$, which also proves it for our case, since the proof only
assumes a sampling probability of $p_{k}=\frac{1}{N}$ for any task $k$.

We define $\bar{v}_{t}=\frac{1}{N}\sum_{k=1}^{N}v_{t}^{k}$ the averaged SGD at
step $t+1$, $\bar{g}_{t}=\frac{1}{N}\sum_{k=1}^{N}\nabla F^{(k)}(w_{t})$ the
averaged gradient and $g_{t}=\frac{1}{N}\sum_{k=1}^{N}\nabla
F^{(k)}(w_{t},\xi_{t}^{k})$ the averaged stochastic gradient. We have
$\bar{v}_{t+1}=w_{t}-\eta_{t}g_{t}$ and $\mathbb{E}g_{t}=\bar{g}_{t}$.

###  6.2 Key lemmas

We first state two lemmas used in the theorem’s proof. We leave their proof
for Appendix 6.4.

######  Lemma 1.

Under assumptions 1 and 2, if $\eta_{t}\leq\frac{1}{L}$, we have:

$\displaystyle\mathbb{E}||\overline{v}_{t+1}-w^{*}||^{2}\leq$
$\displaystyle(1-\mu\eta_{t})\mathbb{E}||w_{t}-w^{*}||^{2}$
$\displaystyle+2L\eta_{t}^{2}\Gamma+\frac{\eta_{t}^{2}}{N^{2}}\sum_{k=1}^{N}\sigma_{k}^{2}$
(9)

######  Lemma 2.

For $\eta_{t}$ non-increasing such as $\eta_{t}\leq 2\eta_{t+1},\forall t\geq
0$, we have:

$\mathbb{E}_{s_{t}}||\overline{v}_{t+1}-w_{t+1}||^{2}\leq\eta_{t}^{2}G^{2}$
(10)

###  6.3 Theorem proof

$\displaystyle||w_{t+1}-w^{*}||^{2}=$
$\displaystyle\underbrace{||w_{t+1}-\overline{v}_{t+1}||^{2}}_{A_{1}}+\underbrace{||\overline{v}_{t+1}-w^{*}||^{2}}_{A_{2}}$
$\displaystyle+\underbrace{2<w_{t+1}-\overline{v}_{t+1},\overline{v}_{t+1}-w^{*}>}_{A_{3}}$

For $A_{3}$, we have

$\mathbb{E}_{s_{t}}(w_{t+1})=\mathbb{E}_{s_{t}}(v_{t+1}^{s_{t}})=\frac{1}{N}\sum_{k=1}^{N}v_{t+1}^{k}=\bar{v}_{t+1}$

which proves $\mathbb{E}_{s_{t+1}}(A_{3})=0$, where $\mathbb{E}_{s_{t}}$
accounts for the expectation taken over the random sampling of $s_{t}$.

We bound $A_{1}$ with Lemma 2 and $A_{2}$ with Lemma 1. We get:

$\displaystyle\mathbb{E}||w_{t+1}-w^{*}||^{2}\leq$
$\displaystyle(1-\mu\eta_{t})\mathbb{E}||w_{t}-w^{*}||^{2}$
$\displaystyle+\frac{\eta_{t}^{2}}{N^{2}}\sum_{k=1}^{N}\sigma_{k}^{2}+2L\eta_{t}^{2}\Gamma+\eta_{t}^{2}G^{2}$

Let $\Delta_{t}=\mathbb{E}||{w_{t+1}-w^{*}}||^{2}$ and
$B=\frac{1}{N^{2}}\sum_{k=1}^{N}\sigma_{k}^{2}+2L\Gamma+G^{2}$. We obtain:

$\Delta_{t+1}\leq(1-\mu\eta_{t})\Delta_{t}+\eta_{t}^{2}B$

We show by induction that for $\beta>\frac{1}{\mu}$ and $\gamma>0$ verifying
$\eta_{1}\leq\frac{1}{L}$ and $\eta_{t}\leq 2\eta_{t+1}$, we have
$\Delta_{t}\leq\frac{v}{\gamma+t}$ with
$v=\max\\{\frac{\beta^{2}B}{\beta\mu-1},(\gamma+1)\Delta_{1}\\}$:

The affirmation directly holds for $t=1$. Then, suppose it holds for $t$, we
have:

$\displaystyle\Delta_{t+1}$
$\displaystyle\leq(1-\mu\eta_{t})\Delta_{t}+\eta_{t}^{2}B$
$\displaystyle\leq\left(1-\frac{\beta\mu}{t+\gamma}\right)\frac{v}{t+\gamma}+\frac{\beta^{2}B}{(t+\gamma)^{2}}$
$\displaystyle=\frac{t+\gamma-1}{(t+\gamma)^{2}}v+\left[\frac{\beta^{2}B}{(t+\gamma)^{2}}-\frac{\beta\mu-1}{(t+\gamma)^{2}}v\right]$
$\displaystyle\leq\frac{t+\gamma-1}{(t+\gamma)^{2}}v$
$\displaystyle\leq\frac{t+\gamma-1}{(t+\gamma+1)(t+\gamma-1)}v=\frac{v}{(t+1)+\gamma}$

where the second inequality comes from $v\geq\frac{\beta^{2}B}{\beta\mu-1}$.
This proves that the affirmation holds for $t+1$.

With the convexity of $F$, we obtain:

$\mathbb{E}[F(w_{t})]-F^{*}\leq\frac{L}{2(\gamma+t)}\left(\frac{\beta^{2}B}{\beta\mu-1}+(\gamma+1)||w_{1}-w^{*}||^{2}\right)\\\
$

which proves theorem 1 for $\beta=\frac{2}{\mu}$, and $\gamma\geq
2\frac{L}{\mu}-1$, to ensure $\eta_{1}\leq\frac{1}{L}$.

###  6.4 Proof of key lemmas

Lemma 1 We assume assumptions 1 and 2 hold, and want to find an upper bound
for $\mathbb{E}||\overline{v}_{t+1}-w^{*}||^{2}$:

$\displaystyle||{\bar{v}_{t+1}-w^{*}}||^{2}$
$\displaystyle=||{w_{t}-\eta_{t}g_{t}-w^{*}-\eta_{t}\bar{g}_{t}+\eta_{t}\bar{g}_{t}}||^{2}$
$\displaystyle=\underbrace{||{w_{t}-w^{*}-\eta_{t}\bar{g}_{t}}||^{2}}_{A_{1}}$
$\displaystyle\quad+\underbrace{2\eta_{t}<{w_{t}-w^{*}-\eta_{t}\bar{g}_{t}},{\bar{g}_{t}-g_{t}}>}_{A_{2}}$
$\displaystyle\quad+\eta_{t}^{2}||{g_{t}-\bar{g}_{t}}||^{2}$ (11)

We have $\mathbb{E}[A_{2}]=0$, so we focus on $A_{1}$:

$\displaystyle A_{1}$
$\displaystyle=||{w_{t}-w^{*}-\eta_{t}\bar{g}_{t}}||^{2}$
$\displaystyle=||{w_{t}-w^{*}}||^{2}\underbrace{-2\eta_{t}<{w_{t}-w^{*}},{\bar{g}_{t}}>}_{B_{1}}+\underbrace{\eta_{t}^{2}||{\bar{g}_{t}}||^{2}}_{B_{2}}$

By convexity of $||{.}||^{2}$ and $L$-smoothness of $F^{(k)}(.)$, we get:

$\displaystyle B_{2}$ $\displaystyle=\eta_{t}^{2}||{\bar{g}_{t}}||^{2}$
$\displaystyle\leq\frac{\eta_{t}^{2}}{N}\sum_{k=1}^{N}||{\nabla
F^{(k)}(w_{t})}||^{2}$ $\displaystyle\leq
2L\frac{\eta_{t}^{2}}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-{F^{(k)}}^{*})$

Then:

$\displaystyle B_{1}$ $\displaystyle=-2\eta_{t}<{w_{t}-w^{*}},{\bar{g}_{t}}>$
$\displaystyle=-2\frac{\eta_{t}}{N}\sum_{k=1}^{N}<{w_{t}-w^{*}},{\nabla
F^{(k)}(w_{t})}>$

The $\mu$-strong convexity of $F^{(k)}(.)$ gives:

$\displaystyle-<{w_{t}-w^{*}},{\nabla F^{(k)}(w_{t})}>\leq$
$\displaystyle-(F^{(k)}(w_{t})-F^{(k)}(w^{*}))$
$\displaystyle-\frac{\mu}{2}||{w_{t}-w^{*}}||^{2}$

we replace in $A_{1}$:

$\displaystyle A_{1}$
$\displaystyle=||{w_{t}-w^{*}-\eta_{t}\bar{g}_{t}}||^{2}$
$\displaystyle\leq||{w_{t}-w^{*}}||^{2}+2L\frac{\eta_{t}^{2}}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-{F^{(k)}}^{*})$
$\displaystyle-2\frac{\eta_{t}}{N}\sum_{k=1}^{N}\left((F^{(k)}(w_{t})-F^{(k)}(w^{*}))+\frac{\mu}{2}||{w_{t}-w^{*}}||^{2}\right)$
$\displaystyle=(1-\mu\eta_{t})||{w_{t}-w^{*}}||^{2}$
$\displaystyle+2L\frac{\eta_{t}^{2}}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-{F^{(k)}}^{*})$
$\displaystyle-2\frac{\eta_{t}}{N}\sum_{k=1}^{N}\left(F^{(k)}(w_{t})-F^{(k)}(w^{*})\right)$

We note:

$\displaystyle C$
$\displaystyle=2L\frac{\eta_{t}^{2}}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-{F^{(k)}}^{*})$
$\displaystyle\quad-2\frac{\eta_{t}}{N}\sum_{k=1}^{N}\left(F^{(k)}(w_{t})-F^{(k)}(w^{*})\right)$

Let $\gamma_{t}=2\eta_{t}(1-L\eta_{t})$. Since $\eta_{t}\leq\frac{1}{L}$, we
obtain $\gamma_{t}\geq 0$. Then:

$\displaystyle C$
$\displaystyle=2L\frac{\eta_{t}^{2}}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-{F^{(k)}}^{*})$
$\displaystyle\quad-2\frac{\eta_{t}}{N}\sum_{k=1}^{N}\left(F^{(k)}(w_{t})-F^{(k)}(w^{*})\right)$
$\displaystyle=2L\frac{\eta_{t}^{2}}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-{F^{(k)}}^{*})$
$\displaystyle\quad-2\frac{\eta_{t}}{N}\sum_{k=1}^{N}\left(F^{(k)}(w_{t})+{F^{(k)}}^{*}-{F^{(k)}}^{*}-F^{(k)}(w^{*})\right)$
$\displaystyle=\left[\frac{1}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-{F^{(k)}}^{*})\right](2L\eta_{t}^{2}-2\eta_{t})$
$\displaystyle\quad+2\frac{\eta_{t}}{N}\sum_{k=1}^{N}(F^{(k)}(w^{*})-{F^{(k)}}^{*})$
$\displaystyle=-\frac{\gamma_{t}}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-{F^{(k)}}^{*})$
$\displaystyle+2\frac{\eta_{t}}{N}\sum_{k=1}^{N}(F^{(k)}(w^{*})-{F^{(k)}}^{*})$
$\displaystyle=-\frac{\gamma_{t}}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-F^{*}+F^{*}-{F^{(k)}}^{*})$
$\displaystyle\quad+2\frac{\eta_{t}}{N}\sum_{k=1}^{N}(F^{*}-{F^{(k)}}^{*})$
$\displaystyle=-\frac{\gamma_{t}}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-F^{*})+\frac{2\eta_{t}-\gamma_{t}}{N}\sum_{k=1}^{N}(F^{*}-{F^{(k)}}^{*})$
$\displaystyle=-\frac{\gamma_{t}}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-F^{*})+2L\frac{\eta_{t}^{2}}{N}\sum_{k=1}^{N}(F^{*}-{F^{(k)}}^{*})$
$\displaystyle=-\frac{\gamma_{t}}{N}\sum_{k=1}^{N}(F^{(k)}(w_{t})-F^{*})+2L\eta_{t}^{2}\Gamma$
$\displaystyle=-\gamma_{t}(F(w_{t})-F^{*})+2L\eta_{t}^{2}\Gamma\leq
2L\eta_{t}^{2}\Gamma$

Since $(F(w_{t})-F^{*})\geq 0$ and $\gamma_{t}\geq 0$. We replace in equation
11:

$\displaystyle||{\bar{v}_{t+1}-w^{*}}||^{2}$
$\displaystyle\leq(1-\mu\eta_{t})||{w_{t}-w^{*}}||^{2}$
$\displaystyle\quad+2L\eta_{t}^{2}\Gamma+\eta_{t}^{2}||{g_{t}-\bar{g}_{t}}||^{2}$
$\displaystyle\mathbb{E}||{\bar{v}_{t+1}-w^{*}}||^{2}$
$\displaystyle\leq(1-\mu\eta_{t})\mathbb{E}||{w_{t}-w^{*}}||^{2}$
$\displaystyle\quad+2L\eta_{t}^{2}\Gamma+\eta_{t}^{2}\mathbb{E}||{g_{t}-\bar{g}_{t}}||^{2}$

Then, Assumption 3 gives us:

$\displaystyle\mathbb{E}||{g_{t}-\bar{g}_{t}}||^{2}$
$\displaystyle=\mathbb{E}||{\frac{1}{N}\sum_{k=1}^{N}(\nabla
F^{(k)}(w_{t},\xi_{t}^{k})-\nabla F^{(k)}(w_{t}))}||^{2}$
$\displaystyle=\frac{1}{N^{2}}\sum_{k=1}^{N}\mathbb{E}||{\nabla
F^{(k)}(w_{t},\xi_{t}^{k})-\nabla F^{(k)}(w_{t})}||^{2}$
$\displaystyle\leq\frac{1}{N^{2}}\sum_{k=1}^{N}\sigma_{k}^{2}$

We finally obtain:

$\displaystyle\mathbb{E}||{\bar{v}_{t+1}-w^{*}}||^{2}$
$\displaystyle\leq(1-\mu\eta_{t})\mathbb{E}||{w_{t}-w^{*}}||^{2}$
$\displaystyle\quad+2L\eta_{t}^{2}\Gamma+\frac{\eta_{t}^{2}}{N^{2}}\sum_{k=1}^{N}\sigma_{k}^{2}$

Lemma 2

We now want to find an upper bound for
$\mathbb{E}_{s_{t}}||{\bar{v}_{t+1}-w_{t+1}}||^{2}$, with $\mathbb{E}_{s_{t}}$
the expected value taken over $s_{t}$ values:

$\displaystyle\mathbb{E}_{s_{t}}||{\bar{v}_{t+1}-w_{t+1}}||^{2}$
$\displaystyle=\mathbb{E}_{s_{t}}||{v_{t+1}^{s_{t}}-\bar{v}_{t+1}}||^{2}$
$\displaystyle=\frac{1}{N}\sum_{k=1}^{N}||{v_{t+1}^{k}-\bar{v}_{t+1}}||^{2}$
$\displaystyle=\frac{1}{N}\sum_{k=1}^{N}||{(v_{t+1}^{k}-w_{t})-(\bar{v}_{t+1}-w_{t})}||^{2}$
$\displaystyle=\mathbb{E}||{(v_{t+1}^{k}-w_{t})-\mathbb{E}(v_{t+1}^{k}-w_{t})}||^{2}$

Since $\mathbb{E}||{x-\mathbb{E}x}||^{2}\leq\mathbb{E}||{x}||^{2}$, we obtain:

$\displaystyle\mathbb{E}_{s_{t}}||{\bar{v}_{t+1}-w_{t+1}}||^{2}$
$\displaystyle\leq\mathbb{E}||{v_{t+1}^{k}-w_{t}}||^{2}$
$\displaystyle=\frac{1}{N}\sum_{k=1}^{N}||{v_{t+1}^{k}-w_{t}}||^{2}$
$\displaystyle\leq\frac{1}{N}\sum_{k=1}^{N}\mathbb{E}||{v_{t+1}^{k}-w_{t}}||^{2}$
$\displaystyle\leq\frac{1}{N}\sum_{k=1}^{N}\mathbb{E}||{\eta_{t}\nabla
F^{(k)}(w_{t},\xi_{t}^{k})}||^{2}$
$\displaystyle\leq\frac{1}{N}\sum_{k=1}^{N}\eta_{t}^{2}G^{2}=\eta_{t}^{2}G^{2}$

which proves the Lemma.


DESY 21-162

October 2021

# Application of a modified chirp-taper scheme for generation of attosecond
pulses in XUV and soft X-ray FELs

E.A. Schneidmiller<EMAIL_ADDRESS>Deutsches Elektronen-
Synchrotron DESY, Notkestr. 85, 22607 Hamburg

###### Abstract

Typically, in Self-Amplified Spontaneous Emission Free Electron Laser (SASE
FEL) based short-pulse schemes, pulse duration is limited by FEL coherence
time. For hard X-ray FELs, coherence time is in a few hundred attosecond range
while for XUV and soft X-ray FELs it is in the femtosecond regime. In this
paper the modification of so-called chirp-taper scheme is developed that
allows to overcome the coherence time barrier. Numerical simulations for XUV
and soft X-ray FEL user facility FLASH demonstrate that one can generate a few
hundred attosecond long pulses in the wavelength range 2 - 10 nm with peak
power reaching hundreds of megawatts. With several thousand pulses per second
this can be a unique source for attosecond science.

###### pacs:

41.60.Cr; 29.20.-c

##  I Introduction

Attosecond science attoscience  is rapidly developing nowadays thanks to the
laser-based techniques such as chirped-pulse amplification and high-harmonic
generation. There are also different schemes proposed for generation of
attosecond X-ray pulses in free electron lasers attofel-oc ; oc-2004-2 ;
atto-b ; atto-e ; atto-f ; prstab-2006-2 ; ding ; huang-24as . Many of these
schemes make use of a few-cycle intense laser pulse to modulate electron
energy in a short undulator, and then to make only a short slice (a fraction
of wavelength) efficiently lase in a SASE undulator. In particular, in the
chirp-taper scheme prstab-2006-2 , a slice with the strongest energy chirp is
selected for lasing by application of a strong reverse undulator taper that
compensates FEL gain degradation within that slice. The lasing in the rest of
the bunch is strongly suppressed due to uncompensated reverse taper.

Creation of a short lasing slice can also be done without using a laser. In
particular, nonlinear compression of multi-GeV electron beams shuang  and
self-modulation in a wiggler of a bunch with the special temporal shape duris
allowed to generate a few hundred attosecond long pulses at the Linac Coherent
Light Source (LCLS). However, creation of sub-femtosecond features in the
electron bunch at lower electron energies ($\simeq 1$ GeV) is problematic.

Typically, pulse duration in SASE-based short-pulse schemes is limited by FEL
coherence time book . For hard X-ray FELs, coherence time is usually in a few
hundred attosecond range. For such a case an adequate choice of a laser could
be a Ti:Sapphire system providing a few mJ within 5 fs (FWHM) with the central
wavelength at 800 nm. However, for XUV and soft X-ray regimes the coherence
time is in femtosecond range, and a longer wavelength laser is needed fawley
to match a lasing slice duration and coherence time. In this paper a simple
method is developed 111A similar concept was proposed by the author as an
option for FLASH upgrade FL-CDR  but was not studied. to overcome this barrier
and to produce XUV and soft X-ray pulses that are much shorter than FEL
coherence time, and can be as short as few hundred attoseconds.

##  II Principles of operation

Figure 1:  Conceptual scheme for generation of attosecond pulses. Dashed
rectangle illustrates a particular realization of suppression (separation) of
a radiation background from the SASE undulator.

Conceptual representation of the attosecond scheme is shown in Fig. 1. Few-
cycle laser pulse is used to modulate a central part of an electron bunch in
energy in a short (typically, two-period) modulator undulator. The wavelength
$\lambda_{L}$ is chosen such that the lasing slice is much shorter than FEL
coherence length. In particular, for generation of attosecond pulses in XUV
and soft X-ray regime one can consider Ti:sapphire laser. A typical shape of
energy modulation after the modulator undulator is shown in Fig. 2.

Then the bunch enters a long SASE undulator tuned to a wavelength $\lambda$.
The undulator is operated in the same way as in the classical chirp-taper
scheme prstab-2006-2 : it is reverse-tapered to compensate for the energy
chirp within the central slice (positioned at $t=0$ in Fig. 2). In this way
the FEL gain degradation within this slice is avoided, and the amplification
proceeds up to the onset of saturation. The rest of the bunch suffers from the
uncompensated reverse taper, and the lasing is strongly suppressed (except
maybe for two satellites positioned around $t=\pm 2.7$ fs on Fig. 2 with the
negative time derivative). The difference with the standard scheme is that now
the central lasing slice is much shorter than FEL coherence time. The
distribution of bunching (density modulation amplitude) is rather narrow and
is localized at the end of that slice but the radiation slips forward, and a
relatively long pulse (on the order of coherence time) is produced.

Figure 2:  Energy modulation induced by the laser. Bunch head is on the left
side.

The next task is to get rid of this relatively long radiation pulse (as well
as of the background radiation from the rest of the bunch) while preserving
the bunching. This can be done in different ways. In Fig. 1 a possible
realization is illustrated: an offset chicane with a reflector or absorber
inside. Alternative options are discussed below in this Section: excessive
reverse taper, an achromatic bend, a kick with a quadrupole, a dogleg, and a
harmonic afterburner.

Finally, the microbunched beam radiates in a short radiator undulator. The
bunching is strong in the central slice, it is weaker in the two satellites
around $t=\pm 2.7$ fs, and much weaker in the rest of the bunch. Note that
reverse tapering is very efficient in suppression of the radiation but the
bunching can reach high values, depending on conditions rev-tap . We use
sufficiently strong chirp and taper to make sure that the the bunching stays
at low level in the whole bunch except for the mentioned slices. In addition
to that, another feature of the process is used to strongly suppress the
ratiation from unwanted parts of the bunch, including satellites. Namely, the
central slice is stretched in the main undulator due to a strong energy chirp
so that the frequency of the bunching is red-shifted with respect to the
resonance frequency at the entrance of SASE undulator. The satellites have a
weaker red shift. The rest of the bunch also has a red shift due to undulator
taper stupakov-taper  but it is even weaker. The radiator undulator is set to
the resonance with the central slice, and the number of periods is
approximately equal to the number of cycles of density modulation within that
slice. The other parts of the bunch are non-resonant and radiate very weakly.
Below in this Section the operation of the radiator is discussed in more
details.

As a result, few hundred attosecond long pulses with low background can be
produced in XUV and soft X-ray ranges. The prerequisite for operation of this
scheme is a sufficiently long SASE undulator. Note that there is always the
range of photon energies at XUV and X-ray FEL facilities for which the
saturation occurs well before the undulator end, and there is a reserve for
operation with different advanced schemes.

###  II.1 Chirp-taper compensation effect

If there is a linear energy chirp at the undulator entrance, it can have a
significant effect on SASE FEL properties, in particular on the gain. The
strength of this effect can be characterized by the energy chirp parameter
prstab-2006-2 :

$\hat{\alpha}=-\frac{d\gamma}{dt}\frac{1}{\gamma_{0}\omega_{0}\rho^{2}}\ ,$
(1)

where $\rho$ is the well-known FEL parameter bon-rho ; book , and $\gamma$ is
relativistic factor. Factor $\gamma_{0}$ for a reference particle and
reference frequency $\omega_{0}$ are connected by the FEL resonance condition:
$\omega_{0}=2ck_{w}\gamma_{0}^{2}/(1+K^{2}/2)$. Here $K$ is the undulator
parameter and $k_{w}=2\pi/\lambda_{w}$ with $\lambda_{w}$ being the undulator
period.

It was shown in prstab-2006-2  that a degrading effect of a linear energy
chirp on SASE FEL gain can be compensated for by applying a linear undulator
taper as soon as the following condition is satisfied:

$\frac{dK}{dz}=-\frac{(1+K_{0}^{2}/2)^{2}}{K_{0}}\frac{1}{\gamma_{0}^{3}}\frac{d\gamma}{cdt}$
(2)

Here $K_{0}$ is the value of undulator parameter at the undulator entrance.
Note that the condition (2) is applicable when $4\pi\rho\hat{\alpha}\ll 1$,
and a perfect compensation is only possible in the limit $\rho\to 0$. However,
for practical applications a perfect compensation is usually not required.

###  II.2 Chirp-taper compensation for a short lasing slice

Operation of SASE FELs with short bunches was studied in bon-short ; we-short
. A relevant parameter to characterize an effect of bunch length on FEL
operation is $\rho\omega\sigma_{z}/c$ with $\sigma_{z}$ being rms bunch
length. When this parameter is smaller than one (i.e. when bunch is shorter
than FEL coherence length $c(\rho\omega)^{-1}$), one can observe an increase
of saturation length and a reduction of FEL efficiency.

The condition (2) is also valid for short bunches or for short lasing slices
(as it was mentioned above, an ideal compensation is only possible when
$\rho\to 0$). In this paper we deal with long bunches but short lasing slices
having the strongest laser-induced energy chirp. For such a case, instead of
$\sigma_{z}$ one can consider a reduced laser wavelength,
$\lambda{\rule[5.0pt]{4.0pt}{0.3pt}}\hskip 1.0pt_{L}=\lambda_{L}/(2\pi)$.
Thus, a relevant parameter is now $\rho\lambda_{L}/\lambda$ where
$\lambda=2\pi c/\omega$ is the FEL wavelength. It follows from numerical
simulations with laser-modulated beam that an increase of saturation length
for a short lasing slice with respect to a normal SASE with long bunches can
be approximated as follows:

$\frac{L_{\mathrm{sat}}}{L_{\mathrm{sat}}^{\mathrm{(long\
bunch)}}}\simeq\left(\rho\frac{\lambda_{L}}{\lambda}\right)^{-1/2}\ \ \ \ \ \
\ \mathrm{for}\ \ \ \ \ \ \ \rho\frac{\lambda_{L}}{\lambda}<1$ (3)

The dependence is similar to that for short bunches bon-short ; we-short . For
the purpose of the proposed scheme, one should stop at the onset of saturation
(typically 80% to 90% of saturation length) to avoid an increase of the width
of bunching distribution within the lasing slice. Thus, a total increase of
the required undulator length can be acceptable in many practical cases even
for a small parameter $\rho\lambda_{L}/\lambda$.

###  II.3 Suppression of background from the main undulator

One of the advantages of the proposed scheme is that one can get a clean
attosecond pulse from the afterburner. However, we need to get rid of the
background produced in the main undulator. Let us consider possible ways of
doing this.

####  II.3.1 Excessive reverse taper

Reverse taper is efficient in suppression of the radiation, although under
some conditions the bunching can survive rev-tap . In case of the considered
scheme one can apply reverse taper which is stronger than the one needed for
compensation of the energy chirp in the main lasing slice. With some delay of
the saturation, one can get a strong bunching there but almost no radiation.
Excess of reverse taper would then be even stronger in the adjacent slices
with the same sign of energy chirp but a weaker amplitude. There one can
suppress the radiation even stronger, and the bunching factor saturation is
delayed also stronger than in the main slice. In general, the intensity of the
radiation from the main undulator can be made sufficiently small. Then, in the
radiator a strong power is produced only within the main slice, also due to a
frequency offset mentioned above 222Note that in some cases a regular
undulator segment (if it is sufficiently short) with optimized K value can
play a role of the radiator undulator.. A disadvantage of this method is that
it requires a longer main undulator which is not always possible. Also,
bunching within the main slice can be weaker than in a case without over-
compensation, depending on parameters.

####  II.3.2 Achromatic bend or a kick with a quadrupole

Another way to produce clean attosecond pulses in the afterburner is to create
an angle between the radiation from the main SASE undulator and from the
radiator by using an achromatic bend bend-kulipanov  or a kick with a
quadrupole macarthur . The latter technique (in combination with reverse
taper) was successfully used for generation of circularly polarized radiation
with high purity at LCLS lutman-circ .

####  II.3.3 Chicane or dogleg

One can also create an offset between the electron beam and the radiation from
the SASE undulator with the help of a chicane, as shown in Fig. 1. Then the
radiation is either absorbed directly or reflected to an absorber. A possible
difficulty is that the longitudinal dispersion, characterized by a transfer
matrix coefficient $R_{56}$, is generated in the chicane. This can be a useful
effect: additional bunching can be created, so that one can stop earlier in
SASE undulator; moreover the lasing slice is stretched even stronger which
helps in suppression of background in the radiator. However, these two
functions of the chicane (a technically reasonable offset and an optimal
$R_{56}$) should be matched which is not always easy to do. A more flexible
system could be a chicane with quadrupoles in the dispersion regions thompson
so that one can efficiently control $R_{56}$ while the required offset is
kept.

Another possible solution is a dogleg that creates a sufficient offset but the
$R_{56}$ is typically too small to influence longitudinal dynamics.

####  II.3.4 Harmonic afterburner

Radiation at the even harmonics of SASE undulator is weak. Thus, tuning the
radiator to the second harmonic, for example, would help to provide low-
background attosecond pulses. Radiation at the fundamental of the undulator
can be filtered out if it disturbs an experiment.

###  II.4 Suppression of satellites in the afterburner

One of the problems of laser-based methods for production of the attosecond
pulses is an insufficient contrast of laser modulation that leads to
generation of satellite pulses shifted in time by a cycle of the laser light
atto-f . They are weaker than the main pulse but can still be a problem for
user experiments. In the proposed scheme we rely not only on a less efficient
generation of bunching for the satellites, but also (and mainly) on the fact
the frequency of bunching in the main slice is different (more red-shifted)
from that in the satellites. The radiator is tuned to the frequency of the
main slice, and the radiation from the adjacent slices is strongly suppressed
because of the offset from resonance. Spectral properties of the radiator are
characterized by the well-known sinc-function:

$f_{1}(\omega)=\left(\frac{\sin(N_{w}\pi\frac{\omega-\omega_{r}}{\omega_{r}})}{N_{w}\pi\frac{\omega-\omega_{r}}{\omega_{r}}}\right)^{2}$
(4)

Here $\omega_{r}$ is the resonance frequency of the radiator and $N_{w}$ is
the number of periods. The latter parameter should be chosen such that it is
approximately the same as the number of cycles in the bunching distribution
within the main lasing slice. At the same time, as it can be seen from (4),
for an efficient suppression one needs to satisfy the condition
$N_{w}\geq\omega_{m}/(\omega_{s}-\omega_{m})$ with $\omega_{m}$ being the
frequency of bunching in the main slice and $\omega_{s}$ in the satellites.
One can even adjust parameters such that the satellites are positioned in
frequency domain at the zeros of the sinc function, i.e. when $N_{w}\simeq
n\omega_{m}/(\omega_{s}-\omega_{m})$, where $n$ is a natural number. In this
case the suppression will be especially effective.

The density modulations in the bulk of the beam (not modulated by the laser)
are much weaker than those on the slopes. In addition, they have a much larger
frequency offset from the resonance in the radiator, so that the radiation is
strongly suppressed. As a result, one can obtain a clean attosecond pulse from
the radiator.

##  III Numerical simulations for FLASH

In the first XUV and soft X-ray FEL user facility FLASH flash-nat-phot ; njp
the electron bunches with maximum energy of 1.25 GeV are distributed between
the two undulator lines. The facility operates in the wavelength range 4 - 60
nm with long pulse trains (several hundred pulses) following with 10 Hz
repetition rate. After the planned upgrade, the electron energy will reach
1.35 GeV, this energy is used in numerical simulations. Electron bunches with
the charge 100 pC and the following quality zemella  are considered in this
paper: peak current 1.5 kA, normalized emittance 0.5 mm mrad, uncorrelated
energy spread 200 keV. Parameters of the second undulator line FLASH2 are used
in the simulations. Segmented variable-gap undulator with the period of 3.14
cm and the maximum K about 2.7 consists of twelve 2.5 m long segments with
quadrupoles in the intersections. Average beta-function of the FODO structure
is 7 m.

Modulator undulator has two periods with the period length of 15 cm and the K
value of 12. The Ti:Sapphire laser system generates 5 fs long pulses (FWHM
intensity), a pulse energy is 0.25 mJ. The Rayleigh length is chosen to be 1
m. Energy modulation of the electron beam for this parameter set of laser-
modulator system is presented in Fig. 2, the maximum energy deviation is 4
MeV.

FEL simulations were performed with the code SIMPLEX simplex . The results of
simulations at three different wavelengths with respectively optimized
afterburners are presented below for illustration.

###  III.1 Case I: the fundamental at 4.7 nm

Figure 3:  Bunching factor at the entrance to the radiator (upper plot) and
power at its exit (lower plot) for a single shot. Bunch head is on the left
side.

Let us first consider the case when energy-modulated beam (see Fig. 2)
radiates in FLASH2 undulator tuned to 4 nm at the entrance ($K_{0}=1.25$).
Since the parameter $\rho$ for the considered beam and undulator parameters is
$1.9\times 10^{-3}$, one can use (1) to find that $\hat{\alpha}\simeq 4$ for
the central slice. The reverse step-taper is applied such that $K$ increases
by 0.025 in each undulator segment, we use ten segments in simulations. Note
that the chosen reverse taper is about $20\%$ stronger than the one needed for
the perfect compensation. This helps reduce background from the main undulator
without affecting significantly the generation of strong bunching within the
central slice. However, a much stronger excessive reverse taper would lead to
a significant increase of the undulator length and cannot be considered as the
main method of background reduction for given parameters.

Let us discuss an increase of undulator length with respect to that needed for
saturation of long bunch with the same slice parameters (which is 18 m). The
parameter $\rho\lambda_{L}/\lambda$ is 0.38 in the considered case, so that an
increase of saturation length is about 60 % according to Eq. (3). We do not
aim at reaching saturation since there is a broadening of the bunching
distribution at that point. Lasing is stopped a bit earlier, at about 90 % of
the saturation length, so that the required increase of the undulator length
is about 40 % (from 18 m to 25 m).

The distribution of bunching factor in the modulated part of the bunch at the
exit of the tenth undulator segment is shown for a single shot in Fig. 3
(upper plot). One can see not only strong bunching within the central lasing
slice but also a significant bunching in the satellites. In this simulation we
assume that the $R_{56}$ between the main undulator and the radiator is
negligible (to avoid an effect on bunching distribution it should be below
$\simeq 1{\mu}m$). Thus, the electron beam without modifications is sent to
the radiator while the radiation from the main undulator is suppressed with
the help of one of the methods discussed in the previous Section.

The radiator is the short undulator with 40 periods, period length of 2.5 cm
and the undulator parameter 1.804. In Fig. 3 (lower plot) one can see the
temporal profile of radiation pulse at 4.7 nm emitted by the beam with
bunching shown in Fig. 3 (upper plot). The wavelength increase is due to
stretching of the central slice in the main undulator. One can also see that
satellites are strongly suppressed despite a significant bunching factor, the
mechanism is explained above. Total background (that includes satellites and
the radiation produced in the bulk of the beam) does not exceed a few per cent
level.

Figure 4:  Radiation power for four representative shots at 4.7 nm. Bunch head
is on the left side.

Forty simulation runs were performed to study the properties of the attosecond
pulses. Four representative shots are shown for illustration in Fig. 4. Pulse
duration is in the range 300 - 400 as (FWHM) which is by an order of magnitude
smaller than FEL coherence time in the main undulator. The average pulse
energy is 70 nJ with the rms shot-to-shot variations about 40 %. It is
interesting to note that despite a significant variation of pulse energy, the
timing is very stable (contrary to the case of a standard single-mode lasing
of short electron bunches duesterer ). This opens up a possibility of pump-
probe experiments with near infrared (NIR) and soft X-ray pulses keeping sub-
cycle synchronization. Indeed, the laser-modulated beam after emission of an
attosecond X-ray pulse can be sent through magnetic chicane for conversion of
energy modulation into density modulation on the scale of laser wavelength.
Then it can radiate a NIR pulse in the short undulator (similar to modulator
undulator). Both pulses (soft X-ray and NIR) can be transported to a user
instrument through the same mirror system thus preserving timing. At the
experiment one can either use NIR pulse directly or (if it is too weak) do
cross-correlation with a powerful laser pulse thus preserving timing
information for every shot.

Finally, let us discuss accuracy of the predictions with FEL simulation codes
in the considered case of the strong energy chirp. Note that the parameter
$4\pi\rho\hat{\alpha}$ is about 0.1 which is sufficiently small for FEL theory
with energy-chirped beams to be applicable. At the same time, one cannot
expect a per cent level accuracy of the predictions, they are rather in the
ten per cent range. In particular, one can expect a significant stretching of
the central lasing slice in that range. The evolution of the process in time-
frequency domain is correctly simulated by FEL codes (all the necessary
information is contained in phases), however the change of average electron
density is neglected in these codes. Thus, the radiated power is somewhat
overestimated in the simulations presented here.

###  III.2 Case II: the second harmonic at 2.4 nm

The macroparticle distributions from simulation runs at 4 nm in the main
undulator were used for simulations of the radiation at the second harmonic in
the dedicated afterburner. The undulator parameters are as follows: period
length is 2 cm, number of periods is 20, K equals 1.134. Three representative
shots are shown in Fig. 5. Again, due to the stretching, the wavelength of the
second harmonic is not 2 nm but 2.4 nm. Pulse durations are in the range 250 -
300 as (FWHM), and the average pulse energy is 6 nJ.

Figure 5:  Radiation power for three representative shots at 2.4 nm. Bunch
head is on the left side.

###  III.3 Case III: the fundamental at 9.3 nm

It is also constructive to illustrate the operation of the scheme at a longer
wavelength. The electron beam is the same as it was in the previous
simulations. The main undulator is now shorter, it consists of nine segments.
The period of the afterburner is the same as that of the main undulator, but
the number of periods is only 25, so that it is shorter than a segment of the
main undulator. Four representative shots are shown in Fig. 6. Average pulse
energy is 75 nJ, and the pulse duration is about 400 as (FWHM).

Note that three different periods of the afterburners were used in three
considered cases. They are all relatively short (about 1 m) and can be placed
behind each other. In practice one can optimize the parameters and the number
of devices depending on the operating range of the attosecond facility and a
range of electron energy (in simulations only one energy was used).

Figure 6:  Radiation power for four representative shots at 9.3 nm. Bunch head
is on the left side.

##  IV Prospects for hard X-rays

The standard chirp-taper attosecond scheme was originally intended for
application in hard X-ray FELs. Indeed, the coherence time (a few hundred
attoseconds) naturally matches lasing slice when a few-cycle short wavelength
laser (like Ti:sapphire) is used. The scheme, described in this paper, can
potentially be applied to push pulse duration to few tens attoseconds regime
if the following approach is used. One can apply so-called eSASE scheme atto-f
and use the same laser to create energy modulation in the electron bunch with
a subsequent conversion of these modulations into density modulations (current
spikes). The duration of the central spike with the highest peak current is
then in sub-hundred attosecond range. A strong energy chirp is accumulated
along this spike due to longitudinal space charge field in front and inside of
the SASE undulator sc-und . Then the chirp-taper compensation is used in the
same way as in the case of laser-induced chirp. And to avoid pulse lengthening
due to the slippage one can use the method developed in this paper. Note that
in some cases the application of excessive reverse taper can be adequate, and
a regular segment of an X-ray FEL undulator can be used as a radiator. One can
also consider the second harmonic generation in one segment. In other words,
application of this method might not require any modification of existing SASE
undulators.

##  V Conclusion

A modification of the chirp-taper scheme allows to produce FEL pulses that are
much shorter than FEL coherence time. Thus, generation of attosecond pulses in
XUV and soft X-ray regimes is enabled. Application of such a scheme to a user
facility like FLASH would make it possible to create a unique source for
attosecond science.

##  VI Acknowledgment

The author would like to thank Wim Leemans for his interest in this work, and
Martin Dohlus for careful reading of the manuscript and useful suggestions.

## References

  * (1) P.B. Corkum and F. Krausz, Nat. Phys. 3(2007)381–7
  * (2) E.L. Saldin, E.A. Schneidmiller, M.V. Yurkov, Optics Communications 212(2002)377.
  * (3) E.L. Saldin, E. A. Schneidmiller and M. V. Yurkov, Optics Communications 237(2004)153-164.
  * (4) A.A. Zholents and W.M. Fawley, Phys. Rev. Lett. 92(2004)224801.
  * (5) P. Emma, Z. Huang and M. Borland, Proc. of the FEL2004 Conference, p. 333, http://www.jacow.org
  * (6) A.A. Zholents and G. Penn, Phys. Rev. ST Accel. Beams 8(2005)050704.
  * (7) E.L. Saldin, E.A. Schneidmiller and M.V. Yurkov, Phys. Rev. ST Accel. Beams 9(2006)050702.
  * (8) Y. Ding et al., Phys. Rev. ST Accel. Beams 12(2009)060703
  * (9) D. Xiang, Z. Huang, and G. Stupakov, Phys. Rev. ST Accel. Beams 12(2009)060701
  * (10) S. Huang et al., Phys. Rev. Lett. 119(2017)154801
  * (11) J. Duris et al., Nature Photonics 14(2020)30
  * (12) E.L. Saldin, E.A. Schneidmiller and M.V. Yurkov, “The Physics of Free Electron Lasers”, Springer, Berlin, 1999
  * (13) W.M. Fawley, Nucl. Instrum. and Methods A593(2008)111
  * (14) M. Beye, S. Klumpp and W. Wurth (Eds.), “FLASH2020+. Upgrade of FLASH. Conceptual Design Report”, DESY Hamburg, March 2020, DOI: 10.3204/PUBDB-2020-00465
  * (15) E.A. Schneidmiller and M.V. Yurkov, Phys. Rev. ST-AB 16(2013)110702
  * (16) Z. Huang and G. Stupakov, Phys. Rev. ST-AB 8(2005)040702
  * (17) R. Bonifacio, C. Pellegrini and L.M. Narducci, Opt. Commun. 50(1984)373
  * (18) R. Bonifacio, et al. Phys. Rev. Lett. 73(1994)70
  * (19) E.L. Saldin, E.A. Schneidmiller and M.V. Yurkov, Nucl. Instrum. and Methods A507(2003)101
  * (20) G.N. Kulipanov, A.S. Sokolov and N.A. Vinokurov, Nuclear Instruments and Methods in Physics Research A375(1996)576
  * (21) J. P. MacArthur, A. A. Lutman, J. Krzywinski, and Z. Huang, Physical Review X 8(2018)041036
  * (22) A.A. Lutman et al., Nature Photonics 10(2016)468
  * (23) N.R. Thompson, Proc. of 39th Free Electron Laser Conf., 2019, Hamburg, Germany, p. 658; doi:10.18429/JACoW-FEL2019-THP033
  * (24) W. Ackermann et al., Nature Photonics 1(2007)336
  * (25) K. Tiedtke et al., New Journal of Physics 11(2009)023029
  * (26) J. Zemella and M. Vogt, Proc. of 10th Int. Particle Accelerator Conf., 2019, Melbourne, Australia, p. 1744; doi:10.18429/JACoW-IPAC2019-TUPRB02
  * (27) T. Tanaka, J. Synchrotron Rad. 22(2015)1319
  * (28) I.J.B. Macias et al., Optics Express 29(2021)10491
  * (29) G. Geloni, E.L. Saldin, E.A. Schneidmiller and M.V. Yurkov, Nuclear Instruments and Methods in Physics Research A583(2007)228