11institutetext: University of Edinburgh 22institutetext: Meta AI

# Learn2Augment: Learning to Composite Videos for Data Augmentation in Action
Recognition

Shreyank N Gowda 11 Marcus Rohrbach 22
Frank Keller 11 Laura Sevilla-Lara 11

###### Abstract

We address the problem of data augmentation for video action recognition.
Standard augmentation strategies in video are hand-designed and sample the
space of possible augmented data points either at random, without knowing
which augmented points will be better, or through heuristics. We propose to
learn what makes a “good” video for action recognition and select only high-
quality samples for augmentation. In particular, we choose video compositing
of a foreground and a background video as the data augmentation process, which
results in diverse and realistic new samples. We learn which pairs of videos
to augment without having to actually composite them. This reduces the space
of possible augmentations, which has two advantages: it saves computational
cost and increases the accuracy of the final trained classifier, as the
augmented pairs are of higher quality than average. We present experimental
results on the entire spectrum of training settings: few-shot, semi-supervised
and fully supervised. We observe consistent improvements across all of them
over prior work and baselines on Kinetics, UCF101, HMDB51, and achieve a new
state-of-the-art on settings with limited data. We see improvements of up to
8.6% in the semi-supervised setting. Project Page:
https://sites.google.com/view/learn2augment/home

Figure 1: Standard video augmentation techniques generate data using hand-
designed heuristics (left). We propose to learn to select videos for
augmentation, based on how effective they will be for learning to classify
(middle). Our approach, Learn2Augment, improves classification across datasets
and settings, including UCF101 (right).

##  1 Introduction

Large-scale datasets have played a key role in the progress of research across
AI problems. In computer vision, neural networks have existed for decades, but
one of the enabling factors for the current revolution was the development of
the large ImageNet [8]. In the video domain, manually collecting and
annotating data can be a prohibitively expensive process. In video action
recognition, for example, collecting data requires an immense amount of manual
labor, as it involves finding suitable videos, trimming them and classifying
them.

Recent efforts in video focus on relieving the strong dependency of current
methods to the size of labeled datasets. Some of these efforts [33, 43]
involve increasing the number of data samples through data augmentation. This
strategy aims to create new videos in the training set by performing
transformations on the original annotated videos, where labels are known. This
process adds diversity to the training data, while new videos are still
realistic and plausible. In the simplest version of data augmentation in
video, new data samples are generated by flipping the input video
horizontally, or by cropping a subsection of the video. New methods [43, 38]
propose more sophisticated processes like combining two videos. VideoMix [38]
randomly crops regions of one video and pastes them onto another. ActorCut
[43] goes one step further and uses the bounding box detections of humans on
one video to paste them onto the background of another video. This increases
the diversity of the new videos, and despite the lack of visual realism of the
resulting videos, this strategy helps.

However, as datasets become larger, such data augmentation strategies become
computationally expensive. The search space of possible video pairs and
transformations is enormous and difficult to explore. The solution is often to
sample the space randomly, or to manually design augmentation heuristics. Any
exploration process is particularly burdening in the context of video data,
where the augmentation process needs to be repeated in every frame, which may
be orders of magnitude more expensive than for images.

In this paper we address the problem of sampling for data augmentation, and
propose to learn to select pairs of videos. We show that this reduces the
search space of augmented data points by orders of magnitude and improves the
final accuracy of the classifier significantly. We leverage two observations.
First, not all data points are as useful for classification. This idea has
been exploited in the context of frame or clip selection [11, 20, 16]. Second,
we can learn to predict which data points will be useful without actually
generating them. This is essential, as the space of transformations is huge,
and if we needed to create each candidate augmented video, the process would
be prohibitively expensive.

More concretely, we propose a data augmentation method which we call
Learn2Augment. The proposed method contains a “Selector” network, which
predicts a score of how useful a combination of two videos will be, without
having to actually composite them. The Selector is trained using the accuracy
of the classification as the cue. Since this metric depends on the classifier,
it is not differentiable with respect to the Selector’s parameters. Therefore
we optimize the network using reinforcement learning. Once the Selector
network is trained, we use it to choose good pairs of videos, composite them,
and train a classification network. In our experiments, for example in the
case of the UCF101 dataset, using the Selector reduces the number of augmented
videos by 92% while increasing the classification accuracy.

In the proposed method, each augmented video is created from a pair of videos
using a composition of the segmented foreground of one video, including actor
and objects, onto the background of the other video. This process yields
diverse and realistic new data samples, which we demonstrate is important for
learning. More concretely, results show an improvement of 4.4% over using a
simpler transformation.

The Selector is indeed useful to reduce the number of videos for training the
classifier. However, we also need to reduce the space of possible pairs for
training the Selector network itself. For example, the number of possible
pairs of videos in video datasets can be in the order of millions for small
datasets or billions for large datasets. For this, we leverage the natural
correlation between the occurrence of foreground activities and background
scenes [5]. This is, it is more likely to find someone playing football in a
football field than at a restaurant. Instead of sampling at random the pairs
of videos to train the Selector on, we sample pairs from classes that are
semantically similar. In particular, we use the class names to obtain a
semantic embedding, and match each class to their nearest neighbor in this
space. Experiments show that this extremely simple design choice of Semantic
Matching reduces the space of possible pairs of videos by several orders of
magnitude (from quadratic to linear on the number of videos). This yields
better results than choosing pairs at random, which may result in non-
plausible scenarios, or choosing pairs from the same class, which may not add
as much diversity.

In summary, the proposed Learn2Augment contains three core components: a
Selector that learns to choose good videos to augment, a Semantic Matching
method that improves optimization, and a Video Compositing that composites
video pairs for augmentation. Experimental results show that all components
contribute to the performance of the system in different ways, and the overall
method obtains state-of-the-art in all datasets, and in all settings that
involve limited training data. In addition, in the setting which considers the
full training set, the proposed data augmentation technique improves upon the
baseline on all datasets, including UCF101, HMDB51, and the large-scale
Kinetics-400.

##  2 Related Work

Figure 2: Overview of the proposed Learn2Augment. Given a pair of videos and
their labels, a Selector network gives a score $\omega$ of the quality of the
potential composited video. At training time, the Selector is trained with the
validation loss of the classification network. Once the Selector is trained,
pairs of videos are sampled, and only the promising combinations with high
score $\omega$ are composited and used for training the classifier.

#### Data Augmentation for Video Action Recognition.

Standard data augmentation techniques in action recognition include horizontal
flip and cropping, where new videos are created by selecting a box at each
frame, and then resizing the resulting video to have the same size as the
original one. While this strategy helps, generated videos do not add much
diversity to the training set. Recent efforts such as ActorCut [43] and
VideoMix [38] increase the diversity of new video samples by cutting and
pasting the foreground of one video onto another. This general technique of
combining two data samples has proven to be quite effective, even in the image
domain [37]. However, the resulting videos are not very realistic, and are
used for training regardless of their quality. Zhang et al. [42] go one step
further and synthesize new samples using GANs, and use “self-paced selection”
to train, starting with easy samples and progressively choosing harder
samples. Instead, we propose to create realistic data samples by segmenting,
inpainting and blending the foreground of one video onto the background of
another. Crucially, we learn to discard novel video samples that are not
expected to be useful for classification, overall producing a more accurate
data augmentation strategy.

#### Learning to Augment Data.

The idea of learning to augment data has been used in other computer vision
problems. In the image classification domain, this strategy has been done
using the final classification loss as the training criterion [25], augmenting
in feature space [9], and learning data augmentation policies [6]. As in this
paper, in the image domain it has been noted that the search space for data
samples can be large and thus expensive [7].

Other computer vision domains like low level vision, also struggle with data
dependency, as creating ground truth is particularly hard. In optical flow,
AutoFlow [33] recently introduced the strategy of learning to generate good
training data for a target dataset.

#### Semi-supervised Video Action Recognition.

Semi-supervised learning (SSL) also aims to reduce data dependence by learning
from large sets of unlabeled samples and a small set of labeled ones. SSL in
images has been widely explored. For example, some strategies include giving
pseudo-labels [1, 24] to samples where the classifier has high confidence, and
adding these to the labeled training data. Other common approaches use
consistency regularization [22, 23, 34]. Approaches that combine consistency
regularization and entropy minimization [13] have shown to be very effective
in tackling the SSL task in images such as MixMatch [3] and RemixMatch [2].

SSL in videos however, has not been explored as much. One of the early works
used extreme learning machines [17] to perform SSL on videos. Recently,
VideoSSL [18] and Temporal Contrastive Learning (TCL) [30] leverage SSL in
videos. VideoSSL [18] uses pseudo-labels and object cues from unlabeled
samples to guide the learning process. TCL [30] use a two-pathway contrastive
learning model using unlabeled videos at two different speeds with the
intuition that changing video speeds do not change the action being performed.

Data augmentation and SSL are two different families of techniques to relieve
the dependence on labeled data, and in this paper we experiment with the
combination of both, showing that they are actually complementary.

#### Sample Selection.

Recent work [16] has shown that not all data samples are as useful. Selecting
a subset of high quality frames or clips at test time shows better results
than using the entire video for action recognition. In this spirit, SMART [11]
uses an attention and relation network to learn scores for each frame in a
video and then select only the high ranked ones for inference. Similarly,
SCSampler [20] uses a lightweight clip sampling model to select the salient
clips in a video and use only those. Unlike the proposed method, these learn
to choose single videos, which are already available, while we learn to choose
pairs of videos to be composited, which are not already combined.

The most relevant work to ours is data valuation in the image domain, using RL
[36], in the image domain where each sample is given a score of how effective
the sample is, and at training time the sample is multiplied by this score. In
our work, instead of learning the effectiveness of the training set, we
leverage that knowledge for augmentation.

##  3 Learn2Augment

Figure 3: Pipeline for compositing a single frame. The foreground is from the
class “soccer juggling” and the background from the class “soccer penalty”,
which are semantic class neighbors. We can see objects such as ‘person’ and
‘ball’ are detected as objects of interest.

In this section we describe in detail the architecture of the proposed
Learn2Augment. In a nutshell, the goal is to learn to augment novel data
points which are realistic and diverse, such that we can train a better
classifier with them. For this, we train a Selector network, which predicts a
score of how useful a given pair of videos is for augmentation. We pick pairs
that have a high score to be augmented. The transformation we use for
augmentation is Video Compositing. Training the Selector using the entire
dataset is infeasible, and sampling pairs of videos at random will yield
unlikely pairs. Thus we sample pairs of videos using Semantic Matching. Figure
2 shows an overview of the proposed method and in Sec. 4 we describe how we
train our approach.

###  3.1 Selector

Given two input videos $\boldsymbol{V}_{1}$ and $\boldsymbol{V}_{2}$, the goal
of the Selector is to predict a weight $\omega$, rating the quality of the
potential composited video. Note that the input to the Selector is two
putative videos instead of the composited one. This means that at test time,
we can predict how useful the composited video will be without having to
actually create it.

The architecture of the Selector includes a standard video classification
network to extract video features, which is ResNet3D-18 [15] followed by a
simple multi-layer perceptron (MLP) with 3 hidden layers of sizes 2048, 1024
and 512. Two videos are input to the Selector at a time, and their features
and labels are concatenated and input to the MLP.

Since there is no ground truth of how “good” a video sample is for learning,
we train the Selector using the change in validation loss of the classifier.
This is, we argue that a “good” training sample is one which, if used for
training, improves the validation loss of the classification network. In other
words, if we take one optimization step training the classifier, after
updating the weights, the validation loss will go down. Section 4.1 describes
the training process in detail.

At test time, we use the Selector by sampling pairs of videos, choosing those
pairs with high score $\omega$, and input to the Video Compositing module,
which we describe in Sec. 3.3. The resulting video is finally used to augment
the training set for the classification network.

###  3.2 Semantic Matching (SM)

The number of pairs in the full dataset can be very large, as it grows with
the square of the number of videos. For Kinetics [4], for example, we would
encounter 360 billion pairs. Training the classifier using these is clearly
infeasible, and thus we use the Selector. But training the Selector itself
with all these samples is infeasible too. Sampling uniformly is a reasonable
solution, but many video pairs may not be useful for learning. We leverage the
observation that all combinations of actions and backgrounds are not equally
likely [5]. This natural correlation between actions and backgrounds helps to
prune unlikely class combinations.

For this, we make the assumption that classes that are semantically similar
are more likely to contain a foreground and a background that are plausible in
the real world, and therefore more realistic for our data augmentation
purposes. Thus, we use the class names to extract a language embedding using
sen2vec [28], and use these embeddings to match each class to its nearest
neighbor. We sample videos $\boldsymbol{V}_{1}$ and $\boldsymbol{V}_{2}$ from
class $c_{1}$ and its closest neighbor $c_{2}$ respectively. This simple
decision reduces the number of pairs to grow linearly with the size of the
dataset, and furthermore increases the accuracy significantly with respect to
sampling video pairs at random. More details on the numerical impact can be
found in Sec. 5.3. Semantic class pairs and additional experiments using
intra-class augmentation can be found in the supplementary material.

###  3.3 Video Compositing (VC)

The goal of the augmentation process is to composite two videos, to produce
realistic, plausible and diverse new videos, that will improve the
classification. Figure 3 shows the overall pipeline for compositing a single
frame.

Given two videos which will be used for foreground $\boldsymbol{V}_{f}$ and
background $\boldsymbol{V}_{b}$, we use a standard object segmentation network
(MaskRCNN [14]) to segment out people and objects in every frame of both
videos. Objects categories in action datasets are not completely contained in
the image dataset COCO [26], which is used for training MaskRCNN. However, we
observe that object detections with high confidence tend to correspond to
actual objects, even if the category is not correct (boxing bag is often
classified as fire hydrant), and therefore are useful to our purpose. We could
also have selected only the humans in the video, as action categories tend to
be focused on humans. However, we find that the presence of specific objects
is highly correlated with action categories (musical instruments in the
classes “playing guitar” or “playing violin”). Therefore removing the original
objects from the background and adding the ones from the foreground is
essential for recognition. See numerical results of the impact of these
decisions in the ablation study of Sec. 5.3.

Figure 4: Sample frames of rendered videos. While the segmentation contains
errors, such as missing limbs or portions of the object, the action category
remains clear.

We remove the segmented objects from the background video and fill in the
holes using image inpainting [27], to obtain a clean background video
$\boldsymbol{V^{\prime}}_{b}$. Finally, we combine the foreground objects and
the background at each frame by simple composition, as in:

$\boldsymbol{\widetilde{V}}^{t}=\boldsymbol{V}_{f}^{t}\odot\boldsymbol{M}_{f}^{t}+\boldsymbol{V^{\prime}}_{b}^{t}\odot(1-\boldsymbol{M}_{f}^{t}),$
(1)

where $\boldsymbol{\widetilde{V}}^{t}$ is the resulting composited frame at
time $t$, $\boldsymbol{V}_{f}^{t}$ and $\boldsymbol{V^{\prime}}_{b}^{t}$ are
frames of the foreground and background videos respectively,
$\boldsymbol{M}_{f}^{t}$ is the binary mask with the union of all detected
objects, and $\odot$ is the element-wise multiplication. Figure 4 shows sample
frames of the resulting videos.

##  4 Optimization of Learn2Augment

The optimization of the proposed Learn2Augment method has two stages. In the
first stage, we train the Selector network using RL, as described in Sec. 4.1.
Once the Selector network is trained, in the second stage, we perform data
augmentation to train the classifier. That is, we sample pairs of videos, pass
them through the trained Selector, choose the pairs with high score, create
new videos with these pairs through Video Compositing, and add them to the
training set. We now describe the details of these two training stages.

###  4.1 Training the Selector

As mentioned before, there is no ground truth to tell us how good an augmented
data sample is. Instead, we use the validation loss of the classification
network to train the Selector network. This function is not differentiable
with respect to the parameters of the Selector. A common solution to dealing
with this is to use RL [36].

Specifically, the state $s_{t}$ at time $t$ is the batch of video pairs
sampled using SM. The action $a_{t}$ is the subset of these video pairs
selected for compositing and is represented as a vector of values between 0
and 1. The environment is the classification network and the validation
process. This environment is used to compute a reward $R(\theta)$ for choosing
a particular action, where $\theta$ are the parameters of the Selector.

We calculate the reward in a single step, as the difference between the loss
in the current batch and the moving average of losses in the previous $S$
steps (where $S=5$) denoted as $\delta$, as in Eq. 2:

$\centering
R(\phi)=\left(\frac{1}{|D_{\text{val}}|}\sum_{i=1}^{|D_{\text{val}}|}\mathcal{L}_{\text{cls}}(f_{\phi}(\boldsymbol{V}_{i}),y_{i})\right)-\delta\@add@centering$
(2)

where $\mathcal{L}_{\text{cls}}$ is the classification cross-entropy loss,
$f_{\phi}$ is the classifier network of parameters $\phi$,
$\boldsymbol{V}_{i}$ and $y_{i}$ are an input video and its label
respectively, $D_{\text{val}}$ is the validation set and $|D_{\text{val}}|$ is
the number of samples in $D_{\text{val}}$. The objective function that we want
to maximize is the expected value of the reward:

$\centering J(\theta)=\mathbb{E}(R(\phi)).\@add@centering$ (3)

To find the optimal policy, we would typically differentiate the objective
function with respect to the parameters $\theta$. However, the reward function
is dependent on the validation loss, calculated with the classifier network,
which does not involve $\theta$. Instead, using REINFORCE [35], we approximate
the objective function as:

$\centering\nabla_{\theta}J(\theta)\approx\frac{1}{M}\sum_{i=1}^{M}R_{\tau^{i}}(\phi)\left(\sum_{t=0}^{T-1}\nabla_{\theta}\log\pi_{\theta}(a_{i}^{t}|s_{i}^{t})\right),\@add@centering$
(4)

where, $\tau^{i}$ is the $i^{\text{th}}$ state-action trajectory under the
policy $\pi_{\theta}$, $M$ is the number of sample trajectories and $T$ is the
number of actions performed in a trajectory. Note that as we have single-step
episodes, we can make several simplifications as $M=1$, as $T=1$, and as there
is only one trajectory $\tau^{i}$, and thus $R_{\tau^{i}}(\phi)$ is just
$R(\phi)$. With these simplifications and substituting Eq. 3 in Eq. 4, we
obtain:

$\centering\nabla_{\theta}J(\theta)\approx
R(\phi)\nabla_{\theta}\log\pi_{\theta}(D_{M}|D_{B}),\@add@centering$ (5)

where $D_{M}$ corresponds to the subset of pairs of samples to composite and
$D_{B}$ to all the pairs of samples in the batch. The Selector is updated by
$\alpha\nabla_{\theta}J(\theta)$ where $\alpha$ is the learning rate and
$\delta$ is updated with the last calculated loss as seen in Eq 6.

$\delta_{t}=\frac{S-1}{S}\delta_{t-1}+\frac{1}{|D_{\text{val}}|}\sum_{i=1}^{|D_{\text{val}}|}\left(\mathcal{L}_{cls}(f_{\theta}(\boldsymbol{V}_{i}),y_{i})\right).$
(6)

Note that this training process does involve generating the composited videos
for pairs in $D_{M}$, to input to the classifier and compute the loss.
However, crucially, during training this is a small portion (one order of
magnitude smaller) of how many videos would need to be generated if we were to
composite all pairs of videos.

Once the Selector is trained, we use it for actually filtering good pairs. At
that point, given two videos and their labels, the Selector network predicts a
policy $\pi$ of how likely it is to select the pair. The score $\omega$ is the
value of $\pi$ for each pair. We use a threshold on that score to select the
pairs of videos to augment. In our experiments, we first determine a budget on
the number of videos that we want to augment, and then pick the threshold to
select the top-ranked video pairs. We use these selected pairs of videos as
input to Video Compositing, add them to the training set, and use them to
train the classifier.

###  4.2 Training the Classifier

Similar to previous work which combines multiple samples for augmentation [37,
43], composited/mixed samples should include mixed labels. We adopt the
strategy of Cutmix [37], where the foreground label $y_{f}$ and the background
label $y_{b}$ are combined using a ratio $\lambda$, as:

$\tilde{y}=\lambda y_{f}+(1-\lambda)y_{b},$ (7)

to obtain the mixed label $\tilde{y}$. A simple way to choose $\lambda$ is to
use the ratio of the foreground mask with respect to the overall video. Given
the foreground video $\boldsymbol{V}_{f}$ of dimensions $T\times H\times W$,
and mask at each frame ${M}_{f}$, the foreground ratio would be
$\gamma=\sum\boldsymbol{M}_{f}/({THW})$. Instead of choosing $\lambda$ to be
directly proportional to the foreground ratio $\gamma$, we give slightly more
weight to the foreground [43], as in Eq. 8, where $\alpha=4$.

$\lambda=-(\gamma-1)^{\alpha}+1,\gamma\in[0,1]$ (8)

We add composited videos $\boldsymbol{\widetilde{V}}$, and their mixed labels
$\tilde{y}$ to the training set, and train the classifier network using a
standard cross-entropy loss, with stochastic gradient descent.

The choice of classifier is not tied to our method. In our experiments, we
choose the widely used 3D ResNet-18 architecture, which allows us to compare
directly to other approaches.

##  5 Experiments

We experiment extensively with Learn2Augment using three data settings, four
datasets, and two splits. We also present ablation studies. In this section we
first describe the details of the experiments and then discuss our results.

###  5.1 Experimental Details

#### Datasets.

In order to provide comparison to prior work (e.g. [43, 30]), we use standard
datasets for evaluation in action recognition, including HMDB51 [21], UCF101
[32], Kinetics-400 [4], and Kinetics-100, which includes the 100 classes with
the largest amount of samples in Kinetics, as it is used in prior work [18]
and helps us compare directly. For experiments on the effect of pre-training
the Selector, we use Kinetics-400. For the semi-supervised setting, we split
the datasets following the protocol of VideoSSL [18] and ActorCut [43]. For
few-shot we use the standard split [40] and the Truze split [12] which ensures
no overlap of novel classes with Kinetics-400.

#### Problem Settings.

We test the proposed method in three different settings. In the _semi-
supervised_ setting, a portion of the training set is artificially held out,
and the rest of the training data is assumed to be available, but unlabeled.
Tests are performed on different percentages of held out data. In the _few-
shot_ setting, some classes (novel classes) are assumed to have a very small
number of training samples (one to five instances), while other classes have
the full number of samples (seen classes). We effectively change the $n$-shot
learning problem to a $n+k$-shot problem where $k$ is the number of augmented
samples. Finally, in the standard _full set_ setting, all training data is
available.

#### Training Settings.

We use mini-batch stochastic gradient descent, with momentum of 0.9 and weight
decay 0.001. For each video, we use an 8-frame clip, where the frames are
uniformly sampled. We use batch size of 8. For UCF101 and Kinetics100 in the
SSL setting, we train the model for 400 epochs and for HMDB51, we train for
500 epochs. The initial learning rate is set to 0.1 and then decayed using
cosine annealing policy. For the SSL setting, we use the data split proposed
in VideoSSL [18]. For the few-shot setting, we use the default hyperparameters
of TRX [29], ARN [40] and C3D-PN [31], respectively. In the fully supervised
setting, we train R(2+1)D for 100 epochs on UCF101, HMDB51 and 50 epochs on
Kinetics-400.

###  5.2 Architectural Changes for Different Settings

We briefly explain the structural adaptations of our approach for each of the
settings.

#### Semi-supervised Learning.

Similar to VideoSSL [18], we first train the classifier on the available
labeled data using the categorical cross-entropy loss. Once this network is
trained, we do a forward pass of the unlabeled examples and assign pseudo-
labels to those samples with high confidence. We use these pseudo-labels as
additional data for augmentation. We also add a knowledge distillation loss
inspired by VideoSSL [18]. Details can be found in the supplementary material.

#### Few-shot Learning.

We only augment the novel classes using Learn2Augment. We also do not perform
label mixing and simply use the foreground label for the augmented sample.
This incorporates our composited samples seamlessly into the meta-learning
framework typically followed. We show results on the standard split, as on the
recently proposed TruZe [12]. TruZe ensures that the novel classes do not
overlap with Kinetics-400.

#### Fully-supervised Learning.

This is the simplest setting, where the Selector is trained on the full
training set, and used for data augmentation to train the classifier. We
explore two scenarios: training the classifier from scratch and using a model
pre-trained on Sports1M [19].

Pairs | Video | Semantic | Accuracy | #Videos
---|---|---|---|---
Selector | Compositing | Matching | in % | (S)
✓ | ✓ | ✓ | 58.9 | 12K
$\times$ | ✓ | ✓ | 55.8 | 99K
✓ | $\times$ | ✓ | 54.5 | 12K
✓ | ✓ | $\times$ | 55.2 | (1.2M)
✓ | $\times$ | $\times$ | 52.9 | (1.2M)
$\times$ | ✓ | $\times$ | 48.6 | (10.4M)
$\times$ | $\times$ | ✓ | 50.8 | 99K
$\times$ | $\times$ | $\times$ | 45.5 | (10.4M)
Table 1: Ablation study to explore the impact of each proposed component. All
settings use the same number of samples for training, so that they can be
compared fairly. The # Videos (S) corresponds to the search space in each
scenario. As we can see, we obtain the best accuracy using just 12K instead of
the standard scenario which would have had 10.4M i.e. a reduction of over
1000x.

###  5.3 Ablation Study

Table 1 shows the ablation study of Learn2Augment, which illustrates the
impact of each of the proposed elements in the design. The experiment is done
on the UCF101 dataset, using 20% of the data i.e. in a semi-supervised
setting. All three contributions (Selector, Semantic Matching and Video
Compositing) improve accuracy. Crucially, Semantic Matching and the Selector
also reduce greatly the number of possible video combinations, and the overall
reduction is around three orders of magnitude. We see that Learn2Augment
obtains a 13.4% improvement over the baseline. While there are improvements of
up to 7.4% for each component, the combination of all three gives the best
results. Further analysis can be found in the supplementary material.

The Video Compositing module also has multiple components. In Table 2, we
ablate these components and observe that removing objects is actually
essential, and has the most significant impact, followed by using segmentation
instead of a bounding box, and finally inpainting.

Although the compositing process is more computationally expensive than
previous simpler mixing strategies, it is important to note that 1) the
overall accuracy indeed improves, 2) the actual composition for training the
classifier is done on a small subset of pairs of videos and 3) the Selector
can be trained on a large dataset (e.g.: Kinetics) just once and can be reused
for the smaller datasets without the need of fine-tuning (see Table 3).

Method | Accuracy
---|---
L2A | 58.9
L2A w/o Inpaint | 57.6
L2A w/o Segmentation | 56.8
L2A w/o Objects | 55.7
L2A w/o All | 54.5
Table 2: Ablation study of compositing components. The version “w/o Inpaint”
refers to pasting the foreground without first filling in the holes of removed
objects in the background. The version “w/o Segmentation” refers to using
bounding boxes instead of object segmentations. “w/o Objects” refers to
copying and pasting only the humans in the scene, leaving the objects.

|  | Kinetics 100 | UCF101 | HMDB51
---|---|---|---|---
Method | Conference | 50% | 20% | 10% | 5% | 50% | 20% | 10% | 5% | 60% | 50% | 40%
CutMix [37] | ICCV19 | 53.7 | 46.1 | 43.2 | 39.9 | 46.1 | 36.5 | 34.6 | 25.8 | 33.9 | 30.8 | 27.8
MixUp [41] | ICLR18 | 53.4 | 45.5 | 43.0 | 39.6 | 45.8 | 36.1 | 34.2 | 25.5 | 33.7 | 31.0 | 27.5
CutOut [10] | Arxiv17 | 52.8 | 45.1 | 42.3 | 38.8 | 45.2 | 35.6 | 33.9 | 24.6 | 33.0 | 30.5 | 27.1
ST-VideoMix [38] | Arxiv21 | 55.3 | 46.6 | 43.9 | 40.4 | 46.4 | 36.4 | 35.2 | 25.9 | 34.8 | 31.3 | 28.7
PseudoLabel [24] | ICMLW13 | 59.0 | 48.0 | 38.9 | 27.9 | 47.5 | 37.0 | 24.7 | 17.6 | 33.5 | 32.4 | 27.3
MeanTeacher [34] | Neurips17 | 59.3 | 47.1 | 36.4 | 27.8 | 45.8 | 36.3 | 25.6 | 17.5 | 32.2 | 30.4 | 27.2
S4L [39] | ICCV19 | 54.6 | 51.1 | 43.3 | 33.0 | 47.9 | 37.7 | 29.1 | 22.7 | 35.6 | 31.0 | 29.8
VideoSSL [18] | WACV21 | 65.0 | 57.7 | 52.6 | 47.6 | 54.3 | 48.7 | 42.0 | 32.4 | 37.0 | 36.2 | 32.7
ActorCut [43] | Arxiv21 | 68.7 | 61.2 | 56.8 | 52.7 | 59.9 | 51.7 | 40.2 | 27.0 | 38.9 | 38.2 | 32.9
ActorCut+ID [43] | Arxiv21 | 72.2 | 68.7 | 63.9 | 59.1 | 64.7 | 57.4 | 53.0 | 45.1 | 40.8 | 39.5 | 35.7
TCL [30] | ICCV21 | 70.4 | 64.7 | 61.1 | 58.2 | 62.1 | 55.4 | 52.1 | 42.8 | 41.2 | 40.4 | 34.8
L2A |  | 75.9 | 72.1 | 67.5 | 63.7 | 72.1 | 60.3 | 56.1 | 48.0 | 44.5 | 43.2 | 37.9
L2A +Pre-training |  | - | - | - | - | 73.3 | 64.8 | 60.1 | 50.9 | 47.1 | 46.3 | 42.1

Table 3: Results on the semi-supervised setting. Results for TCL and ActorCut
are obtained by us running the author’s code. All methods are run with a 3D
ResNet-18 backbone for fair comparison. L2A +Pre-training refers to pre-
training the selector and fixing it.

###  5.4 Augmenting in the Semi-supervised Setting

In this setting we artificially hold out a portion of the training set, with
the goal of observing the behavior of different methods as the size of the
training set changes. In this setting, we use the remaining part of the
dataset by producing pseudo-labels, similar to VideoSSL [18]. Table 3 shows
results in this semi-supervised setting. The L2A version of the method uses a
Selector and a classifier trained only on the target dataset (in this case
UCF101, HMDB51 or Kinetics-100). We observe that Learn2Augment improves on all
settings over all previous methods.

The “L2A +Pre-training” row refers to Learn2Augment where the Selector has
been pre-trained on Kinetics-400, without fine-tuning on the target dataset.
We make two observations: First that pre-training on a large dataset helps, as
the results from the pre-trained model are higher for all datasets and
settings. Second that the Selector trained on Kinetics generalizes quite well
to the smaller datasets without the need for fine-tuning. We do not test on
Kinetics-100 with the pre-trained model, as this would mix training and
testing sets.

###  5.5 Augmenting in the Few-shot Setting

We also explore the impact of the proposed method on the more extreme few-shot
setting, where there are only a few examples per class. This is interesting
because few-shot methods are already designed to address data scarcity.

We compare with the current state of the art in this setting, including CD3-PN
[31], ARN [40] and TRX [29], on the UCF101 and HMDB51 datasets. We observe
that the proposed Learn2Augment method improves upon all existing approaches,
suggesting data augmentation is complementary to few-shot methods. Table 4
shows the results of the experiments.

|  | UCF101 | HMDB51
---|---|---|---
Method | Split | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5
C3D-PN [31] | S | 57.1 | 66.4 | 71.7 | 75.5 | 78.2 | 38.1 | 47.5 | 50.3 | 55.6 | 57.4
C3D-PN + L2A | S | 60.8 | 68.9 | 73.3 | 76.6 | 79.1 | 39.8 | 48.9 | 51.5 | 57.3 | 58.2
ARN [40] | S | 66.3 | 73.1 | 77.9 | 80.4 | 83.1 | 45.5 | 50.1 | 54.2 | 58.7 | 60.6
ARN + L2A | S | 67.7 | 74.2 | 79.6 | 81.1 | 84.4 | 47.3 | 51.7 | 55.5 | 60.1 | 61.8
TRX [29] | S | 77.5 | 88.8 | 92.8 | 94.7 | 96.1 | 50.5 | 62.7 | 66.9 | 73.5 | 75.6
TRX + L2A | S | 79.2 | 89.2 | 93.2 | 95.0 | 96.3 | 51.9 | 63.8 | 68.2 | 74.4 | 77.0
C3D-PN [31] | T | 50.9 | 61.9 | 67.5 | 72.9 | 75.4 | 28.8 | 38.5 | 43.4 | 46.7 | 49.1
C3D-PN + L2A | T | 52.5 | 63.8 | 70.1 | 75.2 | 78.2 | 29.9 | 40.1 | 44.5 | 47.7 | 50.8
ARN [40] | T | 61.2 | 70.7 | 75.2 | 78.8 | 80.2 | 31.9 | 42.3 | 46.5 | 49.8 | 53.2
ARN + L2A | T | 63.9 | 73.1 | 77.4 | 80.4 | 81.3 | 33.6 | 43.7 | 48.0 | 51.1 | 53.8
TRX [29] | T | 75.2 | 88.1 | 91.5 | 93.1 | 93.5 | 33.5 | 46.7 | 49.8 | 57.9 | 61.5
TRX + L2A | T | 76.8 | 88.9 | 92.7 | 93.8 | 94.1 | 35.0 | 48.1 | 51.1 | 59.2 | 62.1

Table 4: Results on UCF101 for the Few-Shot Learning setting, with different
splits. Accuracies are reported for 5-way, 1, 2, 3, 4, 5-shot classification.
S corresponds to the split used in [40, 29] and T is the TruZe split [12],
which avoids overlapping classes with Kinetics.

###  5.6 Augmenting the Full Training Set

We finally explore the effect of augmenting the full dataset, both for smaller
datasets, and the large-scale Kinetics. Results can be found on Table 5.
Again, Learn2Augment improves the performance on all datasets even for a pre-
trained model.

Augmentation | Dataset | Pretrained | Top-1
---|---|---|---
Standard | UCF101 | No Pretraining | 55.7
ActorCut [43] | UCF101 | No Pretraining | 68.3
L2A | UCF101 | No Pretraining | 73.1
Standard | HMDB51 | No Pretraining | 40.8
ActorCut [43] | HMDB51 | No Pretraining | 44.5
L2A | HMDB51 | No Pretraining | 46.4
Standard | UCF101 | Sports1M | 93.6
L2A | UCF101 | Sports1M | 95.3
Standard | HMDB51 | Sports1M | 66.6
L2A | HMDB51 | Sports1M | 68.4
Standard | Kinetics | Sports1M | 75.4
L2A | Kinetics | Sports1M | 76.3
Table 5: Augmenting standard datasets improves classification even with a
model pre-trained on the largest existing dataset (Sports1M).

##  6 Why Not Intra-class Augmentation?

One other possibility we explored is intra-class augmentation instead of using
semantic classes. However, when we followed the same procedure on 20% labeled
data of UCF101 we obtain an accuracy of 41.4% in comparison to 58.9% when
using semantically similar classes. Similarly, in Kinetics100 we obtain an
accuracy of 50.1% and 54.4% using 5% and 10% labeled data respectively. That
is 9.4% and 8.9% lower than the results using semantic neighbors. We believe
there to be two main concerns in intra-class augmentation. The first is that
Cutmix [37] has been shown to be an excellent regularization technique. This
is aided by having samples that have soft labels (since they are a ratio of
samples from different classes). However, using intra-class augmentation would
force the labels to be the same as the ground truth class. The second reason
is that samples of a particular class are clips that were part of the same
video. This is the case in both HMDB51 and UCF101 and not so in Kinetics100.
If we cut the background from one sample and paste the foreground onto this,
it results in an identical sample to the original foreground sample. This is
because the background is the same in both cases. All we end up doing then is
training the model on multiple instances of the same data which leads to
overfitting and hence a poor accuracy at test time. However, since the results
are much worse for Kinetics100 as well, we believe that this could be a
smaller contributing factor.

##  7 Distillation Loss for Semi-Supervised Learning (SSL)

Given frame $a$ from video $v$, to distill appearance information of objects
of interest, we use the softmax predictions of a ResNet [15] image classifier.
This network is pre-trained on Imagenet and not modified during training. Let
the output of the ResNet be denoted as $h(a)\in\mathbb{R}^{M}$ where $M$ =
1000 which is the number of classes in Imagenet. We randomly select a frame
from all videos (labeled, unlabeled and augmented) for training. The
classifier model in our architecture, produces an embedding
$q(v)\in\mathbb{R}^{M}$ which is of the same dimensions and space of $h(a)$.
We train $q(v)$ to match the output of $h(a)$ by using a soft cross-entropy
loss that treats the ResNet outputs as soft labels. This loss
$\mathcal{L}_{d}$ can be seen in Eq. 9. Our final loss function is a
combination of $\mathcal{L}_{d}$ and $\mathcal{L}_{s}$ (categorical cross-
entropy loss for video samples). This is done following the work in VideoSSL
[18].

$\mathcal{L}_{d}=-\sum_{v\epsilon(X\cup Z),a\epsilon
v}h(a)\log\left(q(v)\right)$ (9)

##  8 Analysis of Number of Augmented Samples

We see a common pattern when adding augmented samples to the different SSL
settings. This basically refers to increasing the number of augmented samples
in the training set. We see that the accuracy increases initially, reaches a
peak performance and then starts dropping slowly as can be seen in Figure 5.
This makes sense as we don’t expect every mixed example to be helpful for
training. In fact, this helps us to define $\omega_{i}$ for the selector. We
can see Figure 5 for the results from 0 augmentations to 5000 for 10% and 20%
labeled data on UCF101. The sweet spot for the 10% labeled data is around 1200
augmentations and for the 20% labeled data is around 2000 augmentations. Both
of which are obtained using $\omega_{i}=0.6$. We decide the value of
$\omega_{i}$ based on these and results and use the same for HMDB51 and
Kinetics100 for all settings. If we increase the value of $\omega_{i}$ we
obtain fewer samples and decreasing the value of $\omega_{i}$ results in more
number of samples for training. The value of $\omega_{i}$ thus determines the
number of augmented samples and also their quality.

Figure 5: Comparison of performance with increasing number of augmented
samples. Results are for 10% and 20% of labeled data UCF101. We see that the
performance increases initially, reaches a peak and slowly starts dropping.

##  9 Other Selector Choices

The design of the selector is a crucial aspect of our model. We want the
selector to be able to learn what makes a good pair of videos for mixing
without actually having to mix every single pair. However, for lower
percentages of labeled data, we can generate all possible samples of semantic
classes and convert a state-of-the art frame selection model (SMART) [11] to
do sample importance instead of frame importance. We also consider a simple
baseline of using a discriminator network to pick only realistic samples. We
report the results in Table 6. Another approach was to randomly pick a certain
amount of samples to train the classifier network.

We not only outperform all alternative approaches, we also do this by saving
on both memory and computation cost. For example, in the 20 percent setting,
SMART sees 99K videos and these 99k videos have to be precomputed and stored
before training SMART. However, the proposed approach only needs 12K videos
and outperforms SMART by up to 1.4%. This analysis is only to show a
comparison to possible alternatives when storing data is feasible. The idea of
trying these alternatives is only feasible in low percentage labeled data of
small datasets like UCF101 and HMDB51. Even 50% labeled data in UCF101,
results in having to mix over 400k videos while large scale datasets like
Kinetics400 would lead to millions of mixes being needed making it practically
unfeasible.

| 50% | 20% | 10% | 5%
---|---|---|---|---
Method | Acc | SS | Acc | SS | Acc | SS | Acc | SS
Random | 61.9 | 430K | 56.2 | 99K | 51.8 | 44K | 42.3 | 9.7K
Discriminator | 62.8 | 430K | 57.3 | 99K | 52.2 | 44K | 41.1 | 9.7K
SMART [11] | 68.9 | 430K | 58.9 | 99K | 57.8 | 44K | 46.5 | 9.7K
Proposed | 72.1 | 39K | 60.3 | 12K | 56.1 | 5.2K | 48.0 | 1.2K
Table 6: Comparison of approaches for the use of Selector. All results are
reported on UCF101. ’Acc’ corresponds to accuracy and ’SS’ corresponds to the
number of mixed videos that the Selector looks at. All results are on
different percentage of labeled data in UCF101.

##  10 Why Re-train the Classifier Network?

Here, we are talking about the classifier network in our proposed architecture
that the selector learns from (based on the validation loss). Training the
Selector and the Classifier together is also possible. But we decide against
this for 2 reasons. First, and the most important reason is that we want to
save out on computational cost needed to generate an augmented sample. We
showed that the selector network looks only at a fraction of samples before it
understands what makes a good pair. Hence, we first train the selector by
generating augmented samples taken from random samples of semantically similar
classes. Once the selector is trained, we don’t need to generate the mixed
sample for all possible pairs and only generate the mixed samples for good
pairs (the selector need not have seen these pairs before). We then augment
the original dataset by samples that the selector believes will improve the
classifiers performance We compare the performance of the joint training and
re-training of the classifier network in Table 7. We see that re-training the
classifier network always yields the best performance.

Method | 50% | 20% | 10% | 5%
---|---|---|---|---
Jointly trained | 66.5 | 57.4 | 53.1 | 44.7
Retrained | 72.1 | 60.3 | 56.1 | 48.0
Table 7: Comparison of jointly training classifier and re-training it. We see
that there is a consistent large improvement in re-training the classifier.

##  11 Examples of Selected and Discarded Samples

To understand what made a good sample we visualize a few samples that were
selected by the selector model and a few samples that were discarded. These
can be seen in Figure 6. The samples are displayed as 4 frames for better
visualization. Based on the small subset of examples seen, we believe that for
good pairs to be selected some of the criteria could be coherent inpainting,
similar camera movement, not too drastic a background change.

Figure 6: Visualizing selected examples. From top to bottom as (foreground,
background) pairs: (flic-flac, cartwheel), (smile, laugh), (playing violin,
playing cello), (front crawl, swimming backstroke). The first two are examples
from HMDB51 and the last two from UCF101.

We see some samples of discarded examples in Figure 7. Based on the small
subset of examples seen we think possible bad pairs are due to bad video
compositing (example 2 in Figure 7), varying camera movements (example 3 in
Figure 7) or a drastic change in background (example 1 in Figure 7). These are
however based on the few examples we see.

Figure 7: Visualizing discarded examples. From top to bottom as (foreground,
background) pairs: (somersault, diving), (climbing stairs, falling floor),
(baby crawling, walking dog), (hammering, hammer throw).

##  12 Effect of Semantic Match in generalization ability.

We test the generalization ability of the semantic matching by comparing it
with random matching which would correspond to row 4 of Table 1 in the main
paper. We observe that the performance does decrease. To strengthen this test,
we tried the same experiment in the FSL setting, which is an extreme case for
generalization. We augment data for two different methods, using the proposed
L2A, using both semantic and random matching of classes. We observe that even
in this setting, which is the most susceptible to overfitting, the semantic
matching outperforms random matching. We will add this to the final version.

Method | Class Matching | 1-shot | 3-shot | 5-shot
---|---|---|---|---
C3D-PN | Random | 28.1 | 42.9 | 47.7
C3D-PN | Semantic | 29.9 | 44.5 | 50.8
TRX | Random | 33.5 | 49.9 | 60.3
TRX | Semantic | 35.0 | 51.1 | 62.1

Table 8: Results on FSL using the proposed Semantic Matching vs random
matching using the TruZe [12] split.

##  13 Limitations and Future Work

The main area of improvement is the time needed for training. Optimizing the
Selector with RL is time-consuming, and so is compositing the initial samples
for training it. Future work could address this by parameterizing the
composition process and learn these parameters instead of compositing the
pairs directly. It could also learn to select particular frames in a video,
and avoid the computational cost of temporal redundancy. Finally, another
possible direction is to learn what samples to discard from the initial
dataset itself.

##  14 Conclusion

While standard data augmentation strategies in action recognition are hand-
crafted, we propose to learn which pairs of videos are good to composite. In
order to do this, our approach leverages three components. We train a Selector
optimized with RL to choose which pairs of videos are good to composite. We
reduce the search space by using samples from semantically similar classes. We
perform a clean segmentation for mixing samples and remove actors as well as
objects from foreground and background samples. With this, we obtain state-of-
the-art results in semi-supervised and few-shot action recognition settings,
and improve in the fully supervised setting. In particular, we see gains of up
to 8.6% and 3.7% in the semi-supervised and few-shot settings. We also see an
improvement of up to 17.4% when compared to standard augmentation in the fully
supervised setting when training from scratch.

## References

  * [1] Arazo, E., Ortego, D., Albert, P., O’Connor, N.E., McGuinness, K.: Pseudo-labeling and confirmation bias in deep semi-supervised learning. In: 2020 International Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (2020)
  * [2] Berthelot, D., Carlini, N., Cubuk, E.D., Kurakin, A., Sohn, K., Zhang, H., Raffel, C.: Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In: International Conference on Learning Representations (2019)
  * [3] Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C.A.: Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems 32 (2019)
  * [4] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6299–6308 (2017)
  * [5] Choi, J., Gao, C., Messou, J.C., Huang, J.B.: Why can’t i dance in the mall? learning to mitigate scene bias in action recognition. In: NeurIPS (2019)
  * [6] Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.: Autoaugment: Learning augmentation strategies from data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
  * [7] Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.: Randaugment: Practical automated data augmentation with a reduced search space. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 18613–18624. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper/2020/file/d85b63ef0ccb114d0a3bb7b7d808028f-Paper.pdf
  * [8] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [9] DeVries, T., Taylor, G.W.: Dataset augmentation in feature space. In: ICLR (Workshop) (2017)
  * [10] DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017)
  * [11] Gowda, S.N., Rohrbach, M., Sevilla-Lara, L.: Smart frame selection for action recognition. Proceedings of the AAAI Conference on Artificial Intelligence 35(2), 1451–1459 (May 2021), https://ojs.aaai.org/index.php/AAAI/article/view/16235
  * [12] Gowda, S.N., Sevilla-Lara, L., Kim, K., Keller, F., Rohrbach, M.: A new split for evaluating true zero-shot action recognition. In: 43rd DAGM German Conference on Pattern Recognition (2021)
  * [13] Grandvalet, Y., Bengio, Y., et al.: Semi-supervised learning by entropy minimization. CAP 367, 281–296 (2005)
  * [14] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: ICCV (2017)
  * [15] He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European conference on computer vision. pp. 630–645. Springer (2016)
  * [16] Huang, D.A., Ramanathan, V., Mahajan, D., Torresani, L., Paluri, M., Li, F.F., Niebles, J.C.: What makes a video a video: Analyzing temporal information in video understanding models and datasets. pp. 7366–7375 (06 2018). https://doi.org/10.1109/CVPR.2018.00769
  * [17] Iosifidis, A., Tefas, A., Pitas, I.: Semi-supervised classification of human actions based on neural networks. In: 2014 22nd International Conference on Pattern Recognition. pp. 1336–1341. IEEE (2014)
  * [18] Jing, L., Parag, T., Wu, Z., Tian, Y., Wang, H.: Videossl: Semi-supervised learning for video classification. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 1110–1119 (January 2021)
  * [19] Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 1725–1732 (2014)
  * [20] Korbar, B., Tran, D., Torresani, L.: Scsampler: Sampling salient clips from video for efficient action recognition. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 6232–6242 (2019)
  * [21] Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: Hmdb: a large video database for human motion recognition. In: 2011 International Conference on Computer Vision. pp. 2556–2563. IEEE (2011)
  * [22] Kuo, C.W., Ma, C.Y., Huang, J.B., Kira, Z.: Featmatch: Feature-based augmentation for semi-supervised learning. In: European Conference on Computer Vision. pp. 479–495. Springer (2020)
  * [23] Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 (2016)
  * [24] Lee, D.H., et al.: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on challenges in representation learning, ICML. vol. 3, p. 896 (2013)
  * [25] Lemley, J., Bazrafkan, S., Corcoran, P.M.: Smart augmentation learning an optimal data augmentation strategy. IEEE Access 5, 5858–5869 (2017)
  * [26] Lin, T.Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C.L., Dollár, P.: Microsoft coco: Common objects in context (2014), http://arxiv.org/abs/1405.0312
  * [27] Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 85–100 (2018)
  * [28] Pagliardini, M., Gupta, P., Jaggi, M.: Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features. In: NAACL 2018 - Conference of the North American Chapter of the Association for Computational Linguistics (2018)
  * [29] Perrett, T., Masullo, A., Burghardt, T., Mirmehdi, M., Damen, D.: Temporal-relational crosstransformers for few-shot action recognition. arXiv preprint arXiv:2101.06184 (2021)
  * [30] Singh, A., Chakraborty, O., Varshney, A., Panda, R., Feris, R., Saenko, K., Das, A.: Semi-supervised action recognition with temporal contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10389–10399 (2021)
  * [31] Snell, J., Swersky, K., Zemel, R.S.: Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175 (2017)
  * [32] Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
  * [33] Sun, D., Vlasic, D., Herrmann, C., Jampani, V., Krainin, M., Chang, H., Zabih, R., Freeman, W.T., Liu, C.: Autoflow: Learning a better training set for optical flow. In: CVPR (2021)
  * [34] Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. arXiv preprint arXiv:1703.01780 (2017)
  * [35] Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3), 229–256 (1992)
  * [36] Yoon, J., Arik, S., Pfister, T.: Data valuation using reinforcement learning. In: International Conference on Machine Learning. pp. 10842–10851. PMLR (2020)
  * [37] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: International Conference on Computer Vision (ICCV) (2019)
  * [38] Yun, S., Oh, S.J., Heo, B., Han, D., Kim, J.: Videomix: Rethinking data augmentation for video classification. arXiv preprint arXiv:2012.03457 (2020)
  * [39] Zhai, X., Oliver, A., Kolesnikov, A., Beyer, L.: S4l: Self-supervised semi-supervised learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1476–1485 (2019)
  * [40] Zhang, H., Zhang, L., Qi, X., Li, H., Torr, P.H., Koniusz, P.: Few-shot action recognition with permutation-invariant attention. In: Proceedings of the European Conference on Computer Vision (ECCV). Springer (2020)
  * [41] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: International Conference on Learning Representations (2018)
  * [42] Zhang, Y., Jia, G., Chen, L., Zhang, M., Yong, J.: Self-paced video data augmentation by generative adversarial networks with insufficient samples. In: Proceedings of the 28th ACM International Conference on Multimedia. p. 1652–1660. MM ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3394171.3414003, https://doi.org/10.1145/3394171.3414003
  * [43] Zou, Y., Choi, J., Wang, Q., Huang, J.: Learning representational invariances for data-efficient action recognition. CoRR abs/2103.16565 (2021), https://arxiv.org/abs/2103.16565


dot  1  2  3  30  31  32  20  22  31p 0.50.5 32p 0.50.5 22p 0.50.5 21p 0.50.5
21  11p 0.50.5 100

# Fractional Leibniz rule on the torus

Árpád Bényi, Tadahiro Oh, and Tengfei Zhao  Árpád Bényi, Department of
Mathematics, 516 High St, Western Washington University, Bellingham, WA 98225,
USA<EMAIL_ADDRESS>Tadahiro Oh, School of Mathematics
The University of Edinburgh
and The Maxwell Institute for the Mathematical Sciences
James Clerk Maxwell Building
The King’s Buildings
Peter Guthrie Tait Road
Edinburgh
EH9 3FD
United Kingdom<EMAIL_ADDRESS>Tengfei Zhao
School of Mathematics and Physics
University of Science and Technology Beijing
Beijing 100083, China<EMAIL_ADDRESS>
###### Abstract.

We discuss the fractional Leibniz rule for periodic functions on the
$d$-dimensional torus, including the endpoint cases.

###### Key words and phrases:

fractional Leibniz rule; Poisson summation formula

###### 2020 Mathematics Subject Classification:

42B15, 42B25, 46E35

##  1\. Introduction

In their seminal work [20] on the well-posedness theory of the Navier-Stokes
and Euler equations, Kato and Ponce proved the following commutator estimate:

$\displaystyle\|J^{s}(fg)-fJ^{s}g\|_{L^{p}(\mathbb{R}^{d})}$
$\displaystyle\lesssim\|J^{s}f\|_{L^{p}(\mathbb{R}^{d})}\|g\|_{L^{\infty}(\mathbb{R}^{d})}+\|\nabla
f\|_{L^{\infty}(\mathbb{R}^{d})}\|J^{s-1}g\|_{L^{p}(\mathbb{R}^{d})}$ (1.1)

for $s>0$, $1<p<\infty$, and $f,g\in\mathcal{S}(\mathbb{R}^{d})$. Here,
$J^{s}=J^{s}_{\mathbb{R}^{d}}=(1-\Delta)^{\frac{s}{2}}$ denotes the Fourier
multiplier operator on $\mathbb{R}^{d}$ with multiplier
$(1+4\pi^{2}|\xi|^{2})^{\frac{s}{2}}$.111In the following, when there is no
confusion, we simply write $J^{s}$ to denote the operator
$(1-\Delta)^{\frac{s}{2}}$ on either $\mathbb{R}^{d}$ or $\mathbb{T}^{d}$,
depending on the context. A similar comment applies to
$D^{s}=(-\Delta)^{\frac{s}{2}}$. One variant of the Kato-Ponce inequality
(1.1) takes the form of the fractional Leibniz rule on the Euclidean space
$\mathbb{R}^{d}$, which we now recall.

######  Theorem A.

Let $s>0$, $1\leq p_{j},q_{j}\leq\infty$, $j=1,2$, and $\frac{1}{2}\leq
r\leq\infty$ such that $\frac{1}{r}=\frac{1}{p_{j}}+\frac{1}{q_{j}}$. Suppose
that one of the following conditions holds:

  * (i)

(non-endpoint case). Let $1<p_{j},q_{j}\leq\infty$, $j=1,2$, and
$\frac{1}{2}<r<\infty$ such that $s>\frac{d}{r}-d$ or $s\in 2\mathbb{N}$.

  * (ii)

($L^{\infty}$-endpoint case). Let $p_{j}=q_{j}=r=\infty$, $j=1,2$.

  * (iii)

($L^{1}$-endpoint case). Let $p_{1}=p_{2}=1$, $1\leq q_{1}=q_{2}\leq\infty$,
and $\frac{1}{2}\leq r\leq 1$ such that $\frac{1}{r}=1+\frac{1}{q_{j}}$,
$j=1,2$.

Then, the following estimates hold:

$\displaystyle\|D^{s}(fg)\|_{L^{r}(\mathbb{R}^{d})}$
$\displaystyle\lesssim\|D^{s}f\|_{L^{p_{1}}(\mathbb{R}^{d})}\|g\|_{L^{q_{1}}(\mathbb{R}^{d})}+\|f\|_{L^{p_{2}}(\mathbb{R}^{d})}\|D^{s}g\|_{L^{q_{2}}(\mathbb{R}^{d})},$
(1.2) $\displaystyle\|J^{s}(fg)\|_{L^{r}(\mathbb{R}^{d})}$
$\displaystyle\lesssim\|J^{s}f\|_{L^{p_{1}}(\mathbb{R}^{d})}\|g\|_{L^{q_{1}}(\mathbb{R}^{d})}+\|f\|_{L^{p_{2}}(\mathbb{R}^{d})}\|J^{s}g\|_{L^{q_{2}}(\mathbb{R}^{d})}$
(1.3)

for any $f,g\in\mathcal{S}(\mathbb{R}^{d})$, where
$D^{s}=D^{s}_{\mathbb{R}^{d}}=(-\Delta)^{\frac{s}{2}}$ denotes the Fourier
multiplier operator on $\mathbb{R}^{d}$ with multiplier $(2\pi|\xi|)^{s}$.

See [7, Proposition 3.3] and [22, Theorems A.8 and A.12] for the classical
non-endpoint results, where there are further restrictions such as $0<s<1$ and
$1<p_{j},q_{j},r\lneq\infty$; see also [18, 12]. In the non-endpoint case (i),
the restriction $s>\frac{d}{r}-d$ plays no role when $r\geq 1$, since we
assume $s>0$. The non-endpoint case (i) was established independently by
Muscalu and Schlag [23, (2.1)] (for the one-dimensional case which can be
easily extended to higher dimensions) and by Grafakos and S. Oh [16]. We point
out that the condition on $s$ in the non-endpoint case (i) is sharp (with an
extra assumption $p_{j},q_{j}<\infty$, $j=1,2$); see [16, Theorem 2]. Note
that, in the non-endpoint case (i), the proofs of Theorem A in [23, 16]
involve (a) Bony’s paraproduct decomposition [3] and the Coifman-Meyer theorem
[8, 9] or (b) square function estimates and vector-valued maximal
inequalities, and hence they do not extend to the endpoint cases (ii) and
(iii). In [4], by introducing a “low frequency to high frequency switch” in
carrying out summations, Bourgain and Li established Theorem A in the
$L^{\infty}$-endpoint case (ii); see also a partial result [15]. In the
$L^{1}$-endpoint case (iii), when $q_{1}=q_{2}<\infty$, we have $r<1$, which
brings a new difficulty since $L^{r}(\mathbb{R}^{d})$ is only a quasi-Banach
space, where Young’s inequality fails. By establishing new linear and bilinear
multiplier estimates on quasi-Banach spaces, S. Oh and Wu [25] overcame this
difficulty and established Theorem A in the $L^{1}$-endpoint case (iii).

The fractional Leibniz rule on $\mathbb{R}^{d}$ (Theorem A) has played a
fundamental role in the study of nonlinear PDEs on $\mathbb{R}^{d}$ (or on
$\mathbb{R}^{d}_{x}\times\mathbb{R}_{t}$), especially in low regularity
setting; see, for example, [22]. Similarly, in studying nonlinear PDEs in the
periodic setting, i.e. posed on the $d$-dimensional torus
$\mathbb{T}^{d}=(\mathbb{R}/\mathbb{Z})^{d}$ (or on
$\mathbb{T}^{d}_{x}\times\mathbb{R}_{t}$), the fractional Leibniz rule on
$\mathbb{T}^{d}$ has played an essential role; see, for example, [19, 26, 21,
17, 6]. Unfortunately, there seems to be no standard reference for the
fractional Leibniz rule on $\mathbb{T}^{d}$. In some works, special cases of
the fractional Leibniz rule on $\mathbb{T}^{d}$ were proved in an ad hoc
manner (see, for example, [19, Lemma 9.A.2] and [17, Lemma 3.4]), whereas some
other works simply invoked the fractional Leibniz rule on $\mathbb{T}^{d}$
without any proper proof or reference.

Our main goal in this note is to provide a simple proof of the fractional
Leibniz rule on the $d$-dimensional torus $\mathbb{T}^{d}$, corresponding to
that on $\mathbb{R}^{d}$ stated in Theorem A. In [1], the first two authors
presented an elementary proof of Sobolev’s inequality on $\mathbb{T}^{d}$,
which was “part of the mathematical analysis folklore”, to provide a standard
point of reference, especially to young researchers. We hope that the
following proposition serves a similar purpose.

######  Proposition 1.

Let $s>0$, $1\leq p_{j},q_{j}\leq\infty$, $j=1,2$, and $\frac{1}{2}\leq
r\leq\infty$ such that $\frac{1}{r}=\frac{1}{p_{j}}+\frac{1}{q_{j}}$. Suppose
that one of the conditions (i), (ii), and (iii) in Theorem A holds. Then, we
have

$\displaystyle\|D^{s}(fg)\|_{L^{r}(\mathbb{T}^{d})}$
$\displaystyle\lesssim\|D^{s}f\|_{L^{p_{1}}(\mathbb{T}^{d})}\|g\|_{L^{q_{1}}(\mathbb{T}^{d})}+\|f\|_{L^{p_{2}}(\mathbb{T}^{d})}\|D^{s}g\|_{L^{q_{2}}(\mathbb{T}^{d})},$
(1.4) $\displaystyle\|J^{s}(fg)\|_{L^{r}(\mathbb{T}^{d})}$
$\displaystyle\lesssim\|J^{s}f\|_{L^{p_{1}}(\mathbb{T}^{d})}\|g\|_{L^{q_{1}}(\mathbb{T}^{d})}+\|f\|_{L^{p_{2}}(\mathbb{T}^{d})}\|J^{s}g\|_{L^{q_{2}}(\mathbb{T}^{d})}$
(1.5)

for any $f,g\in C^{\infty}(\mathbb{T}^{d})$. where
$D^{s}=D^{s}_{\mathbb{T}^{d}}=(-\Delta)^{\frac{s}{2}}$ and
$J^{s}=J^{s}_{\mathbb{T}^{d}}=(1-\Delta)^{\frac{s}{2}}$ denote the Fourier
multiplier operators on $\mathbb{T}^{d}$ with multipliers $(2\pi|n|)^{s}$ and
$(1+4\pi^{2}|n|^{2})^{\frac{s}{2}}$, respectively.

See [19, Lemma 9.A.2] and [17, Lemma 3.4], where special cases of Proposition
1 on $\mathbb{T}^{d}$ were proven. In [19, Lemma 9.A.2], Ionescu and Kenig
proved (1.5) with $0<s<1$ and $r=p_{1}=2$ (and $q_{1}=\infty$) by transferring
the corresponding estimate on $\mathbb{R}^{d}$ ([22, Theorems A.12]) via the
Poisson summation formula. In [17, Lemma 3.4], the bound (1.5) with $0<s<1$
and $1<p_{j},q_{j},r<\infty$ was shown as a corollary to [7, Proposition 3.3]
and the bilinear transference principle [10] (applied to the application of
the Coifman-Meyer theorem [8, 9]). See also [11, Theorem 3] for some non-
endpoint fractional Leibniz rule on a space of homogeneous type (which in
particular includes the $d$-dimensional torus).

In [23], the fractional Leibniz rule (1.2) on $\mathbb{R}^{d}$ in the non-
endpoint case (i) was shown via Bony’s paraproduct decomposition [3] and the
Coifman-Meyer theorem [8, 9]; see also [23, Theorem 2.15]. As such, the
corresponding result on $\mathbb{T}^{d}$ in the non-endpoint case follows from
the paraproduct decomposition and the bilinear transference principle [10]
applied to each of the paraproducts (estimated by the Coifman-Meyer theorem).
Note that, instead of invoking the bilinear transference principle, it is
possible to directly prove the Coifman-Meyer theorem on $\mathbb{T}^{d}$; see
[23, Problem 3.4] for such a result in the (more complicated) bi-parameter
setting.222Interested readers may want to directly prove (1.4) and (1.5) on
$\mathbb{T}^{d}$, at least in the non-endpoint case (i), without using a
transference result, by adapting [23, Chapter 2] to the periodic setting. We
point out, however, that such an approach via a transference principle or the
Coifman-Meyer theorem on $\mathbb{T}^{d}$ does not work in the endpoint cases
(see, for example, [13, Theorem 4.3.7] in the linear case, where a restriction
$1<p<\infty$ appears). In Section 2, we instead present a proof of Proposition
1 as a direct corollary to the fractional Leibniz rule (1.2) and (1.3) on
$\mathbb{R}^{d}$, using the Poisson summation formula.

######  Remark 1.1.

A generalization of (1.3) in the non-endpoint case to bilinear
pseudodifferential operators with symbols in the so-called exotic Hörmander
classes (which encompass the symbols of Coifman-Meyer type) was obtained in
[2, Theorem 2]; see also [5, 24] for some weighted versions and applications.
Note that [2, Theorem 2] has a natural counterpart on $\mathbb{T}^{d}$.
Interested readers may consider adapting some of the methods suggested in this
note to prove such a result.

##  2\. Proof of Proposition 1

We first establish decay estimates for the kernels of
$D^{s}=D^{s}_{\mathbb{R}^{d}}$ and $J^{s}=J^{s}_{\mathbb{R}^{d}}$ on
$\mathbb{R}^{d}$. Given $s>0$, let $\mathcal{K}_{s}$ and $\mathcal{G}_{s}$ be
the kernels of $D^{s}$ and $J^{s}$ on $\mathbb{R}^{d}$, respectively, given by

$\displaystyle D^{s}f=\mathcal{K}_{s}*f\qquad\text{and}\qquad
J^{s}f=\mathcal{G}_{s}*f.$ (2.1)

The following lemma establishes a (well-known) decay of these kernels away
from the origin. We present its proof for readers’ convenience.

######  Lemma 2.1.

Let $s>0$. Then, given $c_{0}>0$, there exists $C_{1},C_{2}>0$ such that

$\displaystyle|\mathcal{K}_{s}(x)|$ $\displaystyle\lesssim C_{1}|x|^{-d-s},$
(2.2) $\displaystyle|\mathcal{G}_{s}(x)|$ $\displaystyle\lesssim
C_{2}|x|^{-d-s}$ (2.3)

for any $x\in\mathbb{R}^{d}$ with $|x|\geq c_{0}$, where $C_{1}$ is
independent of $c_{0}$.

###### Proof.

We first consider the case $0<s<2$. By writing $D^{s}=(-\Delta)D^{-2+s}$, we
have $\mathcal{K}_{s}=(-\Delta)K_{2-s}$, where $K_{\sigma}$, $\sigma>0$,
denotes the kernel of the Riesz potential $D^{-\sigma}$ of order $\sigma>0$,
given by

$\displaystyle K_{\sigma}(x)=C_{d,\sigma}|x|^{-d+\sigma},\quad x\neq 0.$

By applying $-\Delta$ to $K_{2-s}$, we then have

$\displaystyle|\mathcal{K}_{s}(x)|=C_{d,2-s}\big{|}(-\Delta)|x|^{-d+2-s}\big{|}\lesssim|x|^{-d-s},$

yielding (2.2).

Similarly, by writing $J^{s}=(1-\Delta)J^{-2+s}$, we have

$\displaystyle\mathcal{G}_{s}=(1-\Delta)G_{2-s},$ (2.4)

where $G_{\sigma}$, $\sigma>0$, denotes the kernel of the Bessel potential
$(1-\Delta)^{-\sigma}$ of order $\sigma>0$, given by

$\displaystyle
G_{\sigma}(x)=C_{d,\sigma}\int_{0}^{\infty}e^{-t}e^{-\frac{|x|^{2}}{4t}}t^{\frac{\sigma-d}{2}}\frac{dt}{t}.$
(2.5)

See [14, p. 14]. Note that $G_{\sigma}(x)$ is smooth away from the origin
$x=0$. For $|x|\geq c_{0}$, we have

$\displaystyle t+\frac{|x|^{2}}{4t}\geq
t+\frac{c_{0}^{2}}{4t}\quad\text{and}\quad t+\frac{|x|^{2}}{4t}\geq|x|,$ (2.6)

where the latter follows from completing a square. Thus, from (2.5) and (2.6),
we have

$\displaystyle|G_{2-s}(x)|\lesssim
e^{-\frac{|x|}{2}}\int_{0}^{\infty}e^{-\frac{t}{2}}e^{-\frac{c_{0}^{2}}{8t}}t^{\frac{2-s-d}{2}}\frac{dt}{t}\lesssim
e^{-\frac{|x|}{2}}$ (2.7)

for $|x|\geq c_{0}$. Similarly, by applying $-\Delta$ to $G_{2-s}$ in (2.5)
with (2.6), we have

$\displaystyle|(-\Delta)G_{2-s}(x)|\lesssim
e^{-\frac{|x|}{2}}\int_{0}^{\infty}\Big{(}\frac{1}{t}+\frac{|x|^{2}}{t^{2}}\Big{)}e^{-\frac{t}{2}}e^{-\frac{c_{0}^{2}}{8t}}t^{\frac{2-s-d}{2}}\frac{dt}{t}\lesssim
e^{-\frac{|x|}{3}}$ (2.8)

for $|x|\geq c_{0}$. Hence, (2.3) follows from (2.4), (2.7), and (2.8).

When $s\geq 2$, by writing

$\displaystyle D^{s}=(-\Delta)^{k+1}D^{-2(k+1)+s}\quad\text{and}\quad
J^{s}=(1-\Delta)^{k+1}J^{-2(k+1)+s}$

with $k=\big{[}\frac{s}{2}\big{]}$, where $[\,\cdot\,]$ denotes the integer
part, we can repeat computations analogous to those presented above and obtain
(2.2) and (2.3). We omit the details. ∎

Before proceeding to the proof of Proposition 1, let us recall the Poisson
summation formula ([13, Theorem 3.2.8]).

######  Lemma 2.2.

Let $f\in C(\mathbb{R}^{d})$. Suppose that there exist $C,\delta>0$ such that

$|f(x)|\leq C(1+|x|)^{-d-\delta}$

for any $x\in\mathbb{R}^{d}$ and

$\sum_{n\in\mathbb{Z}^{d}}|\widehat{f}(n)|<\infty.$

Then, for any $x\in\mathbb{R}^{d}$, we have

$\displaystyle\sum_{n\in\mathbb{Z}^{d}}\widehat{f}(n)e^{2\pi in\cdot
x}=\sum_{n\in\mathbb{Z}^{d}}f(x+n).$

We now present a proof of Proposition 1.

###### Proof of Proposition 1.

We only present the proof of (1.4) since (1.5) follows in a similar manner,
using (2.3) in Lemma 2.1. Fix $s>0$ and $p_{j},q_{j},r$, $j=1,2$ as in the
statement of Theorem A.

Let us first introduce some notations. For functions on $\mathbb{T}^{d}$, let
$\mathbf{P}_{\neq 0}$ denote the projector to the non-zero frequencies,
defined by $\mathbf{P}_{\neq 0}f=f-\int_{\mathbb{T}^{d}}fdx$. Then, set
$\mathbf{P}_{0}=\mathrm{Id}-\mathbf{P}_{\neq 0}$, namely,
$\mathbf{P}_{0}f=\mathcal{F}_{\mathbb{T}^{d}}(f)(0)=\int_{\mathbb{T}^{d}}fdx$,
where $\mathcal{F}_{\mathbb{T}^{d}}$ denotes the Fourier transform on
$\mathbb{T}^{d}$.

By noting $D^{s}_{\mathbb{T}^{d}}\mathbf{P}_{0}f=0$ and applying Hölder’s
inequality (with $\text{Vol}(\mathbb{T}^{d})=1$), we have

$\displaystyle\begin{split}&\|D^{s}_{\mathbb{T}^{d}}(fg)\|_{L^{r}(\mathbb{T}^{d})}\\\
&\quad\leq\|D^{s}_{\mathbb{T}^{d}}(\mathbf{P}_{\neq 0}f\mathbf{P}_{\neq
0}g)\|_{L^{r}(\mathbb{T}^{d})}+\|\mathbf{P}_{0}g\cdot
D^{s}_{\mathbb{T}^{d}}\mathbf{P}_{\neq 0}f\|_{L^{r}(\mathbb{T}^{d})}\\\
&\hphantom{XXX}+\|\mathbf{P}_{0}f\cdot D^{s}_{\mathbb{T}^{d}}\mathbf{P}_{\neq
0}g\|_{L^{r}(\mathbb{T}^{d})}\\\
&\quad\leq\|D^{s}_{\mathbb{T}^{d}}(\mathbf{P}_{\neq 0}f\mathbf{P}_{\neq
0}g)\|_{L^{r}(\mathbb{T}^{d})}+\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{r}(\mathbb{T}^{d})}\|g\|_{L^{1}(\mathbb{T}^{d})}\\\
&\hphantom{XXX}+\|f\|_{L^{1}(\mathbb{T}^{d})}\|D^{s}_{\mathbb{T}^{d}}g\|_{L^{r}(\mathbb{T}^{d})}\\\
&\quad\leq\|D^{s}_{\mathbb{T}^{d}}(\mathbf{P}_{\neq 0}f\mathbf{P}_{\neq
0}g)\|_{L^{r}(\mathbb{T}^{d})}+\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{p_{1}}(\mathbb{T}^{d})}\|g\|_{L^{q_{1}}(\mathbb{T}^{d})}\\\
&\hphantom{XXX}+\|f\|_{L^{p_{2}}(\mathbb{T}^{d})}\|D^{s}_{\mathbb{T}^{d}}g\|_{L^{q_{2}}(\mathbb{T}^{d})}.\end{split}$
(2.9)

Hence, it suffices to prove (1.4) for mean-zero functions on $\mathbb{T}^{d}$,
which will allow us to control the first term on the right-hand side of (2.9).

In the following, we fix mean-zero functions $f$ and $g$ on $\mathbb{T}^{d}$
and prove (1.4). Let $\varphi_{1},\varphi_{2}\in
C^{\infty}(\mathbb{R}^{d};[0,1])$ be periodic,
$\varphi_{j}(x+k)=\varphi_{j}(x)$, $k\in\mathbb{Z}^{d}$, $j=1,2$ such that

$\displaystyle\varphi_{1}(x)+\varphi_{2}(x)=1$ (2.10)

for any $x\in\mathbb{R}^{d}$, $\varphi_{1}$ is supported on
$\bigcup_{m\in\mathbb{Z}^{d}}\big{(}\big{[}\frac{1}{8},\frac{7}{8}\big{]}^{d}+m\big{)}$,
and $\varphi_{2}$ is supported on
$\bigcup_{m\in\mathbb{Z}^{d}}\big{(}\big{[}-\frac{3}{8},\frac{3}{8}\big{]}^{d}+m\big{)}$.

Let $F$ be the periodic extension of $f$ on $\mathbb{R}^{d}$. We define
functions $G_{j}$, $j=1,2$, on $\mathbb{R}^{d}$ by setting

$\displaystyle G_{1}=\mathbf{1}_{[0,1]^{d}}\cdot\varphi_{1}\cdot
g\qquad\text{and}\qquad
G_{2}=\mathbf{1}_{[-\frac{1}{2},\frac{1}{2}]^{d}}\cdot\varphi_{2}\cdot g.$
(2.11)

Then, from (2.10) and (2.11), we have

$\displaystyle\mathcal{F}_{\mathbb{T}^{d}}(fg)(n)=\sum_{j=1,2}\mathcal{F}_{\mathbb{R}^{d}}(FG_{j})(n)$
(2.12)

for any $n\in\mathbb{Z}^{d}$, where $\mathcal{F}_{\mathbb{R}^{d}}$ denotes the
Fourier transform on $\mathbb{R}^{d}$. Then, from (2.12) and the Poisson
summation formula (Lemma 2.2), we have

$\displaystyle\begin{split}D^{s}_{\mathbb{T}^{d}}(fg)(x)&=\sum_{n\in\mathbb{Z}^{d}}(2\pi|n|)^{s}\mathcal{F}_{\mathbb{T}^{d}}(fg)(n)e^{2\pi
in\cdot x}\\\
&=\sum_{j=1,2}\sum_{n\in\mathbb{Z}^{d}}(2\pi|n|)^{s}\mathcal{F}_{\mathbb{R}^{d}}(FG_{j})(n)e^{2\pi
in\cdot x}\\\
&=\sum_{j=1,2}\sum_{k\in\mathbb{Z}^{d}}D^{s}_{\mathbb{R}^{d}}(FG_{j})(x+k)\end{split}$
(2.13)

for any $x\in\mathbb{T}^{d}$.

$\bullet$ Case 1: Let us first consider the case $\frac{1}{2}\leq r<1$. Then,
from $\ell^{r}(\mathbb{Z}^{d})\subset\ell^{1}(\mathbb{Z}^{d})$ and a change of
variables, we have

$\displaystyle\|D^{s}_{\mathbb{T}^{d}}(fg)\|_{L^{r}(\mathbb{T}^{d})}\lesssim\sum_{j=1,2}\|D^{s}_{\mathbb{R}^{d}}(FG_{j})(x)\|_{L^{r}(\mathbb{R}^{d})}.$
(2.14)

Let us introduce two more smooth cutoff functions $\psi_{1},\psi_{2}\in
C^{\infty}(\mathbb{R}^{d};[0,1])$ such that $\psi_{j}$ is supported on
$\big{[}\frac{1}{10}-\frac{j-1}{2},\frac{9}{10}-\frac{j-1}{2}\big{]}^{d}$ and
$\psi_{j}\equiv 1$ on
$\big{[}\frac{1}{9}-\frac{j-1}{2},\frac{8}{9}-\frac{j-1}{2}\big{]}^{d}$,
$j=1,2$. Then, by noting $\psi_{j}\varphi_{j}=\varphi_{j}$, it follows from
(2.11) that

$\displaystyle FG_{j}=(\psi_{j}F)G_{j}.$ (2.15)

Hence, from (2.14), (2.15), and the fractional Leibniz rule on
$\mathbb{R}^{d}$ ((1.2) in Theorem A), we have

$\displaystyle\begin{split}\|D^{s}_{\mathbb{T}^{d}}(fg)\|_{L^{r}(\mathbb{T}^{d})}&\lesssim\sum_{j=1,2}\|D^{s}_{\mathbb{R}^{d}}(\psi_{j}F)\|_{L^{p_{1}}(\mathbb{R}^{d})}\|G_{j}\|_{L^{q_{1}}(\mathbb{R}^{d})}\\\
&\quad+\sum_{j=1,2}\|\psi_{j}F\|_{L^{p_{2}}(\mathbb{R}^{d})}\|D^{s}_{\mathbb{R}^{d}}G_{j}\|_{L^{q_{2}}(\mathbb{R}^{d})}.\end{split}$
(2.16)

From (2.11) and the definitions of $\varphi_{j}$ and $\psi_{j}$, we have

$\displaystyle\|G_{j}\|_{L^{q_{1}}(\mathbb{R}^{d})}\leq\|g\|_{L^{q_{1}}(\mathbb{T}^{d})}\quad\text{and}\quad\|\psi_{j}F\|_{L^{p_{2}}(\mathbb{R}^{d})}\leq\|f\|_{L^{p_{2}}(\mathbb{T}^{d})}.$
(2.17)

In view of (2.16) and (2.17), in order to conclude the proof of (1.4) for
mean-zero functions $f$ and $g$ on $\mathbb{T}^{d}$, it suffices to prove

$\displaystyle\begin{split}\|D^{s}_{\mathbb{R}^{d}}(\psi_{j}F)\|_{L^{p}(\mathbb{R}^{d})}&\lesssim\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})},\\\
\|D^{s}_{\mathbb{R}^{d}}G_{j}\|_{L^{p}(\mathbb{R}^{d})}&\lesssim\|D^{s}_{\mathbb{T}^{d}}g\|_{L^{p}(\mathbb{T}^{d})}\end{split}$
(2.18)

for any $s>0$, $1\leq p\leq\infty$, and $j=1,2$.

In the following, we only prove the first bound in (2.18), since, in view of
(2.11), the second bound follows from a similar computation. Moreover, we only
consider the $j=1$ case.

Let $\mathcal{K}^{s}$ be the kernel of $D^{s}_{\mathbb{R}^{d}}$ as in (2.1).
Then, with
$\operatorname*{supp}(\psi_{1})\subset\big{[}\frac{1}{10},\frac{9}{10}\big{]}^{d}$,
we have

$\displaystyle\begin{split}D^{s}_{\mathbb{R}^{d}}(\psi_{1}F)(x)&=\int_{[\frac{1}{10},\frac{9}{10}]^{d}}\mathcal{K}^{s}(x-y)(\psi_{1}F)(y)dy\\\
&=\mathbf{1}_{[0,1)^{d}}(x)\cdot\int_{[\frac{1}{10},\frac{9}{10}]^{d}}\mathcal{K}^{s}(x-y)(\psi_{1}F)(y)dy\\\
&\quad+\mathbf{1}_{\mathbb{R}^{d}\setminus[0,1)^{d}}(x)\cdot\int_{[\frac{1}{10},\frac{9}{10}]^{d}}\mathcal{K}^{s}(x-y)(\psi_{1}F)(y)dy\\\
&=:\hskip 0.56905pt\text{I}\hskip 0.56905pt(x)+\text{I I}(x).\end{split}$
(2.19)

As for I I, we have $|x-y|\gtrsim 1$ and thus by Young’s inequality and Lemma
2.1 with $s>0$, we have

$\displaystyle\|\text{I I}\|_{L^{p}(\mathbb{R}^{d})}$
$\displaystyle\lesssim\|\mathbf{1}_{\\{|\,\cdot\,|\gtrsim
1\\}}\mathcal{K}^{s}\|_{L^{1}(\mathbb{R}^{d})}\|\psi_{1}F\|_{L^{p}(\mathbb{R}^{d})}\leq\|f\|_{L^{p}(\mathbb{T}^{d})}$
(2.20)

When $1<p<\infty$, since we assume that $f$ is a mean-zero function on
$\mathbb{T}^{d}$, we have

$\displaystyle\|f\|_{L^{p}(\mathbb{T}^{d})}\leq\|J^{s}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})}\sim\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})},$
(2.21)

where the second step follows from the Mihlin multiplier theorem (on
$\mathbb{T}^{d}$) or the Littlewood-Paley characterization of the non-
homogeneous and homogeneous Sobolev spaces, together with the fact that $f$
has mean zero. When $p=\infty$, by choosing $1<q<\infty$ with $sq>d$, it
follows from the Sobolev embedding theorem, the Mihlin multiplier theorem (as
in (2.21)), and Hölder’s inequality (with $\text{Vol}(\mathbb{T}^{d})=1$) that

$\displaystyle\|f\|_{L^{\infty}(\mathbb{T}^{d})}\lesssim\|J^{s}_{\mathbb{T}^{d}}f\|_{L^{q}(\mathbb{T}^{d})}\sim\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{q}(\mathbb{T}^{d})}\leq\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{\infty}(\mathbb{T}^{d})}.$
(2.22)

When $p=1$, by duality and (2.22) (applied to $D^{-s}_{\mathbb{T}^{d}}g$), we
have

$\displaystyle\begin{split}\|f\|_{L^{1}(\mathbb{T}^{d})}&=\sup_{\begin{subarray}{c}\|g\|_{L^{\infty}(\mathbb{T}^{d})}=1\\\
g,\text{mean zero}\end{subarray}}\bigg{|}\int_{\mathbb{T}^{d}}fgdx\bigg{|}\\\
&\leq\sup_{\begin{subarray}{c}\|g\|_{L^{\infty}(\mathbb{T}^{d})}=1\\\
g,\text{mean
zero}\end{subarray}}\|D^{-s}_{\mathbb{T}^{d}}g\|_{L^{\infty}(\mathbb{T}^{d})}\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{1}(\mathbb{T}^{d})}\\\
&\lesssim\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{1}(\mathbb{T}^{d})}.\end{split}$
(2.23)

Putting (2.20), (2.21), (2.22), and (2.23) together, we obtain

$\displaystyle\|\text{I
I}\|_{L^{p}(\mathbb{R}^{d})}\lesssim\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})}.$
(2.24)

Next, we treat I in (2.19). By noting
$\mathcal{F}_{\mathbb{T}^{d}}(\psi_{1}f)(n)=\mathcal{F}_{\mathbb{R}^{d}}(\psi_{1}F)(n)$
and proceeding as in (2.13) with the definition of I in (2.19), we have

$\displaystyle\begin{split}&\mathbf{1}_{[0,1)^{d}}(x)\cdot
D^{s}_{\mathbb{T}^{d}}(\psi_{1}f)(x)=\hskip 0.56905pt\text{I}\hskip
0.56905pt(x)+\mathbf{1}_{[0,1)^{d}}(x)\sum_{k\in\mathbb{Z}^{d}\setminus\\{0\\}}D^{s}_{\mathbb{R}^{d}}(\psi_{1}F)(x+k)\\\
&\quad=\hskip 0.56905pt\text{I}\hskip
0.56905pt(x)+\sum_{k\in\mathbb{Z}^{d}\setminus\\{0\\}}\mathbf{1}_{[0,1)^{d}}(x)\cdot\int_{[\frac{1}{10},\frac{9}{10}]^{d}}\mathcal{K}^{s}(x+k-y)(\psi_{1}F)(y)dy\\\
&\quad=:\hskip 0.56905pt\text{I}\hskip 0.56905pt(x)+\text{I I
I}(x)\end{split}$ (2.25)

for any $x\in\mathbb{R}^{d}$, where we used the identification
$\mathbb{T}^{d}\cong[0,1)^{d}$.

First, note that, for $s=2m\in 2\mathbb{N}\cup\\{0\\}$, it follows from the
classical Leibniz rule and Hölder’s inequality that

$\displaystyle\begin{split}\|D^{2m}_{\mathbb{T}^{d}}(\psi_{1}f)\|_{L^{p}(\mathbb{T}^{d})}&=\|(-\Delta)^{m}_{\mathbb{T}^{d}}(\psi_{1}f)\|_{L^{p}(\mathbb{T}^{d})}\\\
&\leq\|D^{2m}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})}+C\sum_{\ell=0}^{2m-1}\|\partial^{\ell}f\|_{L^{p}(\mathbb{T}^{d})},\end{split}$
(2.26)

where
$\partial^{\ell}=\partial_{x_{1}}^{\ell_{1}}\cdots\partial_{x_{d}}^{\ell_{d}}$
denotes (a combination of) the mixed partial derivatives of order
$\ell=\ell_{1}+\cdots+\ell_{d}$. When $1<p<\infty$, it follows from the Mihlin
multiplier theorem with (2.21) that

$\displaystyle\|\partial^{\ell}f\|_{L^{p}(\mathbb{T}^{d})}\sim\|D^{\ell}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})}\lesssim\|D^{2m}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})}.$
(2.27)

When $p=\infty$, by the Sobolev embedding theorem (with $1<q<\infty$ and small
$\sigma>0$ such that $\sigma q>d$), the Mihlin multiplier theorem (as in
(2.27); recall that $f$ has mean zero), (2.21), Hölder’s inequality (with
$\text{Vol}(\mathbb{T}^{d})=1$) that

$\displaystyle\|\partial^{\ell}f\|_{L^{\infty}(\mathbb{T}^{d})}\lesssim\|J^{\sigma}_{\mathbb{T}^{d}}\partial^{\ell}f\|_{L^{q}(\mathbb{T}^{d})(\mathbb{T}^{d})}\sim\|D^{\ell+\sigma}_{\mathbb{T}^{d}}f\|_{L^{q}(\mathbb{T}^{d})}\lesssim\|D^{2m}_{\mathbb{T}^{d}}f\|_{L^{\infty}(\mathbb{T}^{d})}.$
(2.28)

When $p=1$, by duality, integration by parts, and (2.28) (applied to
$D^{-s}_{\mathbb{T}^{d}}g$), we have

$\displaystyle\begin{split}\|\partial^{\ell}f\|_{L^{1}(\mathbb{T}^{d})}&=\sup_{\begin{subarray}{c}\|g\|_{L^{\infty}(\mathbb{T}^{d})}=1\\\
g,\text{mean
zero}\end{subarray}}\bigg{|}\int_{\mathbb{T}^{d}}(\partial^{\ell}f)gdx\bigg{|}=\sup_{\begin{subarray}{c}\|g\|_{L^{\infty}(\mathbb{T}^{d})}=1\\\
g,\text{mean
zero}\end{subarray}}\bigg{|}\int_{\mathbb{T}^{d}}f(\partial^{\ell}g)dx\bigg{|}\\\
&\leq\sup_{\begin{subarray}{c}\|g\|_{L^{\infty}(\mathbb{T}^{d})}=1\\\
g,\text{mean
zero}\end{subarray}}\|\partial^{\ell}D^{-2m}_{\mathbb{T}^{d}}g\|_{L^{\infty}(\mathbb{T}^{d})}\|D^{2m}_{\mathbb{T}^{d}}f\|_{L^{1}(\mathbb{T}^{d})}\\\
&\lesssim\|D^{2m}_{\mathbb{T}^{d}}f\|_{L^{1}(\mathbb{T}^{d})}.\end{split}$
(2.29)

Hence, from (2.26), (2.27), (2.28), and (2.29), we obtain

$\displaystyle\|D^{2m}_{\mathbb{T}^{d}}(\psi_{1}f)\|_{L^{p}(\mathbb{T}^{d})}\lesssim\|D^{2m}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})}.$
(2.30)

By interpolating (2.30) with the trivial bound when $m=0$, we then obtain

$\displaystyle\|D^{s}_{\mathbb{T}^{d}}(\psi_{1}f)\|_{L^{p}(\mathbb{T}^{d})}\lesssim\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})}$
(2.31)

for any $s\geq 0$.

For any $x\in[0,1)^{d}$, $y\in[\frac{1}{10},\frac{9}{10}]^{d}$, and
$k\in\mathbb{Z}^{d}\setminus\\{0\\}$, we have

$\displaystyle|x+k-y|\gtrsim\max(1,|k|).$ (2.32)

Then, from (2.25), (2.31), Minkowski’s inequality, and Lemma 2.1 with (2.32),
we have

$\displaystyle\begin{split}\|\hskip 0.56905pt\text{I}\hskip
0.56905pt\|_{L^{p}(\mathbb{R}^{d})}&\lesssim\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})}+\|\text{I
I I}\|_{L^{p}([0,1)^{d})}\\\
&\lesssim\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})}+\bigg{(}\sum_{k\in\mathbb{Z}^{d}\setminus\\{0\\}}|k|^{-d-s}\bigg{)}\|f\|_{L^{p}(\mathbb{T}^{d})}\\\
&\lesssim\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{p}(\mathbb{T}^{d})},\end{split}$
(2.33)

where the last step follows from (2.21), (2.22), and (2.23).

Hence, from (2.19), (2.24), and (2.33), we conclude the first bound in (2.18).
Therefore, we obtain (1.4) for mean-zero functions $f$ and $g$ on
$\mathbb{T}^{d}$ in this case.

$\bullet$ Case 2: Next, we consider the case $1\leq r\leq\infty$. In this
case, the bound (2.14) no longer holds. From (2.13), we have

$\displaystyle\begin{split}\|D^{s}_{\mathbb{T}^{d}}(fg)\|_{L^{r}(\mathbb{T}^{d})}&\leq\sum_{j=1,2}\bigg{\|}\sum_{k\in\mathbb{Z}^{d}}D^{s}_{\mathbb{R}^{d}}(FG_{j})(x+k)\bigg{\|}_{L^{r}([0,1)^{d})}\\\
&\leq\sum_{j=1,2}\|D^{s}_{\mathbb{R}^{d}}(FG_{j})(x)\|_{L^{r}(\mathbb{R}^{d})}\\\
&\quad+\sum_{j=1,2}\bigg{\|}\sum_{k\in\mathbb{Z}^{d}\setminus\\{0\\}}D^{s}_{\mathbb{R}^{d}}(FG_{j})(x+k)\bigg{\|}_{L^{r}([0,1)^{d})}\\\
&=:A_{1}+A_{2}.\end{split}$ (2.34)

As for $A_{1}$, we can proceed as in Case 1 and obtain

$\displaystyle
A_{1}\lesssim\|D^{s}f\|_{L^{p_{1}}(\mathbb{T}^{d})}\|g\|_{L^{q_{1}}(\mathbb{T}^{d})}+\|f\|_{L^{p_{2}}(\mathbb{T}^{d})}\|D^{s}g\|_{L^{q_{2}}(\mathbb{T}^{d})}.$
(2.35)

As for $A_{2}$, by applying Lemma 2.1 with (2.32), and Young’s inequality as
in (2.33) (in handling I I I) and then applying Hölder’s inequality, we have

$\displaystyle\begin{split}A_{2}&\lesssim\|FG_{j}\|_{L^{r}([0,1)^{d})}\leq\|fg\|_{L^{r}(\mathbb{T}^{d})}\leq\|f\|_{L^{p_{1}}(\mathbb{T}^{d})}\|g\|_{L^{q_{1}}(\mathbb{T}^{d})}\\\
&\lesssim\|D^{s}_{\mathbb{T}^{d}}f\|_{L^{p_{1}}(\mathbb{T}^{d})}\|g\|_{L^{q_{1}}(\mathbb{T}^{d})},\end{split}$
(2.36)

where the last step follows from (2.21), (2.22), and (2.23). Hence, from
(2.34), (2.35), and (2.36), we obtain (1.4) for mean-zero functions $f$ and
$g$ on $\mathbb{T}^{d}$ in this case. This completes the proof of Proposition
1. ∎

######  Acknowledgements.

Á.B. acknowledges the support from an AMS-Simons Research Enhancement Grant
for PUI Faculty. T.O. was supported by the European Research Council (grant
no. 864138 “SingStochDispDyn”). T.Z. was supported by the National Natural
Science Foundation of China (Grant No. 12101040, 12271051, and 12371239). T.Z.
would like to thank the University of Edinburgh for its hospitality where this
manuscript was prepared.

## References

  * [1] Á. Bényi, T. Oh, The Sobolev inequality on the torus revisited, Publ. Math. Debrecen 83 (2013), no. 3, 359–374.
  * [2] Á. Bényi, R.H. Torres, Symbolic calculus and the transposes of bilinear pseudodifferential operators, Comm. Partial Differential Equations 28 (2003), no. 5-6, 1161–1181.
  * [3] J.-M. Bony, Calcul symbolique et propagation des singularités pour les équations aux dérivées partielles non linéaires, Ann. Sci. École Norm. Sup. 14 (1981), no. 2, 209–246.
  * [4] J. Bourgain, D. Li, On an endpoint Kato-Ponce inequality, Differential Integral Equations 27 (2014), no. 11-12, 1037–1072.
  * [5] J. Brummer, V. Naibo, Weighted fractional Leibniz-type rules for bilinear multiplier operators, Potential Anal. 51 (2019), no. 1, 71–99.
  * [6] A. Chapouto, G. Li, T. Oh, D. Pilod, Deep-water limit of the intermediate long wave equation in $L^{2}$, preprint.
  * [7] M. Christ, M. Weinstein, Dispersion of small amplitude solutions of the generalized Korteweg-de Vries equation. J. Funct. Anal. 100 (1991), 87–109.
  * [8] R. Coifman, Y. Meyer, On commutators of singular integrals and bilinear singular integrals, Trans. Amer. Math. Soc. 173 (1975), 315–331.
  * [9] R. Coifman, Y. Meyer, Au delà des opérateurs pseudo-différentiels, Astérisque, 57. Société Mathématique de France, Paris, 1978. i+185 pp.
  * [10] D. Fan, S. Sato, Transference on certain multilinear multiplier operators, J. Aust. Math. Soc. 70 (2001), no. 1, 37–55.
  * [11] A.E. Gatto, Product rule and chain rule estimates for fractional derivatives on spaces that satisfy the doubling condition, J. Funct. Anal. 188 (2002), no. 1, 27–37.
  * [12] L. Grafakos, Multilinear operators in harmonic analysis and partial differential equations, Harmonic analysis and nonlinear partial differential equations, 11–27, RIMS Kôkyûroku Bessatsu, B33, Res. Inst. Math. Sci. (RIMS), Kyoto, 2012.
  * [13] L. Grafakos, Classical Fourier analysis, Third edition. Graduate Texts in Mathematics, 249. Springer, New York, 2014. xviii+638 pp.
  * [14] L. Grafakos, Modern Fourier analysis, Third edition. Graduate Texts in Mathematics, 250. Springer, New York, 2014. xvi+624 pp.
  * [15] L. Grafakos, D. Maldonado, V. Naibo, A remark on an endpoint Kato-Ponce inequality, Differential Integral Equations 27 (2014), no. 5-6, 415–424.
  * [16] L. Grafakos, S. Oh, The Kato-Ponce inequality, Comm. Partial Differential Equations 39 (2014), no. 6, 1128–1157.
  * [17] M. Gubinelli, H. Koch, T. Oh, Renormalization of the two-dimensional stochastic nonlinear wave equations, Trans. Amer. Math. Soc. 370 (2018), no 10, 7335–7359.
  * [18] A. Gulisashvili, M. Kon, Exact smoothing properties of Schrödinger semigroups, Amer. J. Math. 118 (1996), no. 6, 1215–1248.
  * [19] A. Ionescu, C. Kenig, Local and global wellposedness of periodic KP-I equations, Mathematical aspects of nonlinear dispersive equations, 181–211, Ann. of Math. Stud., 163, Princeton Univ. Press, Princeton, NJ, 2007.
  * [20] T. Kato, G. Ponce, Commutator estimates and the Euler and Navier-Stokes equations, Comm. Pure Appl. Math. 41 (1988), 891–907.
  * [21] C. Kenig, D. Pilod, Local well-posedness for the KdV hierarchy at high regularity, Adv. Differential Equations 21 (2016), no. 9-10, 801–836.
  * [22] C. Kenig, G. Ponce, L. Vega, Well-posedness and scattering results for the generalized Korteweg-de Vries equation via the contraction principle, Comm. Pure Appl. Math. 46 (1993), no. 4, 527–620.
  * [23] C. Muscalu, W. Schlag, Classical and multilinear harmonic analysis. Vol. II, Cambridge Studies in Advanced Mathematics,138. Cambridge University Press, Cambridge, 2013. xvi+324 pp.
  * [24] V. Naibo, A. Thomson, Coifman-Meyer multipliers: Leibniz-type rules and applications to scattering of solutions to PDEs, Trans. Amer. Math. Soc. 372 (2019), no. 8, 5453–5481.
  * [25] S. Oh, X. Wu, On $L^{1}$ endpoint Kato-Ponce inequality, Math. Res. Lett. 27 (2020), no. 4, 1129–1163.
  * [26] N. Tzvetkov, N. Visciglia, Invariant measures and long time behavior for the Benjamin-Ono equation, Int. Math. Res. Not. IMRN 2014, no. 17, 4679–4714.