# Enlarged Large Margin Loss for Imbalanced Classification

Sota Kato1 and Kazuhiro Hotta2 1Sota Kato is with Department of Electrical and
Electronic Engineering, Meijo University, Nagoya, Japan
<EMAIL_ADDRESS>Hotta is with Department of
Electrical and Electronic Engineering, Meijo University, Nagoya, Japan
<EMAIL_ADDRESS>
###### Abstract

We propose a novel loss function for imbalanced classification. LDAM loss,
which minimizes a margin-based generalization bound, is widely utilized for
class-imbalanced image classification. Although, by using LDAM loss, it is
possible to obtain large margins for the minority classes and small margins
for the majority classes, the relevance to a large margin, which is included
in the original softmax cross entropy loss, is not be clarified yet. In this
study, we reconvert the formula of LDAM loss using the concept of the large
margin softmax cross entropy loss based on the softplus function and confirm
that LDAM loss includes a wider large margin than softmax cross entropy loss.
Furthermore, we propose a novel Enlarged Large Margin (ELM) loss, which can
further widen the large margin of LDAM loss. ELM loss utilizes the large
margin for the maximum logit of the incorrect class in addition to the basic
margin used in LDAM loss. Through experiments conducted on imbalanced CIFAR
datasets and large-scale datasets with long-tailed distribution, we confirmed
that classification accuracy was much improved compared with LDAM loss and
conventional losses for imbalanced classification.

##  I INTRODUCTION

In recent years, convolutional neural networks (CNNs) are widely applied to
various image classification tasks with great success [13, 14]. However, there
is a problem that are great differences in the number of samples in each class
when CNNs are applied to real-world datasets [1, 2], and typically we call
this a class-imbalanced problem. In class-imbalanced image classification,
images belonging to a certain class account for the majority of datasets, and
those belonging to the remaining classes are the minority. Consequently, the
accuracy of minority classes will be quite low because a classifier is trained
on datasets with an imbalanced sample ratio between classes. To improve the
classification performance on imbalanced datasets, it is important to improve
the accuracy of minority classes.

The most common strategy for learning imbalanced data is the re-weighting
strategy [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 21]. While various approaches for
re-weighting strategy have been proposed in recent years, one of the most
effective approaches is a margin-based loss function like Label-Distribution-
Aware Margin (LDAM) loss [4]. It can regularize the minority classes more
strongly than the majority classes, and gain per-class and uniform-label test
error bounds based on the theory in [24], by accounting for the minimum margin
per class. Subsequently, by minimizing the obtained bounds, we can improve the
accuracy of minority classes without sacrificing the ability of the model to
fit the frequent classes. On the other hand, in a recent study, it has been
revealed that the softmax cross entropy loss includes a large-margin effect
which is dependent on the distribution of logits [15]. Although LDAM loss is a
loss function based on softmax cross entropy loss, the relevance between LDAM
loss and Large-Margin Softmax Cross Entropy (LMSCE) loss [15] is not clarified
yet.

Therefore, in this study, we reconvert a formula for LDAM loss in terms of a
large-margin perspective. Then, we confirm that LDAM loss includes wider and
class-specific values of large margins than LMSCE loss. In imbalanced
learning, we consider this to be the reason that LDAM loss is a better
performance than other losses based on softmax cross entropy loss.
Furthermore, inspired by the above analysis, we propose a novel Enlarged Large
Margin (ELM) loss with even wider margins than LDAM loss. A stronger
regularization than LDAM loss can be achieved by further increasing the
margins between the logit of the target class and the logit of the class with
the largest logit among the incorrect classes. Consequently, we can achieve
better-balanced learning than LDAM loss even if the dataset has a class-
imbalanced distribution.

Through experiments, we evaluated our loss function on imbalanced CIFAR
datasets and two large-scale datasets with a long-tailed distribution. From
the experimental results, we confirmed that the proposed loss function was
much improved, in comparison with LDAM loss and conventional loss functions
for class-imbalanced learning. Additionally, by comparing the feature space
between conventional margin-based losses and our proposed loss, we confirmed
ELM loss can create a better feature space where the rare classes are more
distant from each frequency class. Our code is available at
111https://github.com/usagisukisuki/ELMloss.

The main contributions of this paper are as follows:

  * •

We reconsider the original LDAM loss in terms of a large-margin perspective.
As a result, we confirm that LDAM loss includes wider and class-specific
values of large margins than LMSCE loss, and we consider LDAM loss is a good
performance under imbalanced learning.

  * •

Furthermore, we propose a novel loss function with even wider margins than
LDAM loss, called Enlarged Large Margin (ELM) loss. ELM loss has a stronger
regularization than LDAM loss, and its regularization can consist of further
increasing the margins between the logit of the target class and the maximum
logit of the other class.

This paper is organized as follows. Section $\rm{II}$ describes the related
works. Section $\rm{III}$ describes our methodology. Section $\rm{IV}$
presents the experimental results. Finally, we describe our summary and future
works in Section $\rm{V}$.

##  II RELATED STUDIES

Figure 1: Comparison of hinge loss and softplus loss.

###  II-A Margin based loss

A margin-based approach is typically obtained by using the hinge loss function
such as SVM [16], for improving the generalization performance, while
mitigating the over-fitting. The classifier benefits from margins to maximize
the inter-class margin. Especially, following [17], it is general to regard as
a classification margin the difference between the logit for the target class
and the maximum logit for the other class. In the field of deep learning,
unique loss functions with the benefits of the margin have been proposed [15,
18, 19, 20]. In particular, it has been revealed that the original softmax
cross entropy loss includes a large-margin effect that is dependent on the
distribution of logits [15], and it is formulated the regularization term to
further enhance the large-margin effect in softmax cross entropy loss.

In contrast to the above loss functions with class-independent margins,
another approach, which uses different margins for each class, has been
proposed [4, 6]. Cao et al.[4] proposed Label-Distribution-Aware Margin (LDAM)
loss that minimizes a margin-based generalization bound. The margin is
obtained from the number of data divided by the $\frac{1}{4}$ power of the
number of samples for each class. By using LDAM loss, it is possible to obtain
large margins for the categories with a small number of samples and small
margins for the categories with a large number of samples. Haeyong et al.[6]
proposed a novel Maximum Margin (MM) loss function to encourage larger sample
margins for hard negative sample classes such that the smaller the maximum
margins are the greater the shifting margins are.

However, even though these margin losses for imbalanced classification are
defined on the basis of softmax cross entropy loss, the relation to the large
margin included in softmax cross entropy loss has not been discussed. In this
study, we reconsider the connection between LDAM loss and LMSCE loss using the
softplus function and confirm that LDAM loss is assigned wider margins than
LMSCE loss.

###  II-B Re-weighting approach

The re-weighting approach is learned by imposing a large weight on
unacceptable errors and a small weight on acceptable errors [3, 4, 5, 6, 7, 8,
9, 10, 11, 12, 21]. Class-balancing weight [3] is one of the common approaches
for re-weighting. This method calculates the weights based on the number of
training samples in each class and weights the loss function to increase the
penalty for minority classes. By using class-balancing weight, it is possible
to balance the learning. Furthermore, the Deferred Re-balancing Weighting
(DRW) strategy, which trains using the vanilla loss before annealing the
learning rate and deploys a class-balancing weight with a smaller learning
rate, has also been proposed [4].

As another approach, novel loss functions to consider the distribution of
class labels have been proposed [4, 5, 6, 7, 8, 9, 10, 11, 12]. Focal loss
[12], Equalization losses [7, 8], Group softmax [9], and Adaptive Class
Suppression loss [10] are typical methods.

Although various loss functions for imbalanced classification have been
proposed, the loss using both LDAM loss and DRW is known to be effective on
many imbalanced datasets and a lot of tasks other than image classification.
Our proposed loss function is a development of LDAM loss which provides higher
performance when we used the combination with the DRW strategy.

##  III METHODOLOGY

(a) LMSCE (b) LDAM (c) ELM (Ours)
---
Figure 2: Comparison between conventional margin-based loss functions and our
proposed loss function.

In this section, we reconsider LDAM loss in terms of a large-margin
perspective and propose a novel strategy to utilize a wider large margin,
called ELM loss. By using this approach, the model can learn with stronger
regularization than LDAM loss, and we can achieve better-balanced learning
than LDAM loss even if the dataset has a class-imbalanced distribution.

###  III-A Large-margin for LDAM loss

In the multi-class classification of $C$ classes, a CNN model provides a logit
vector $\textbf{{z}}\in\mathbb{R}^{C}$. The logit vector is generally produced
by the last fully-connected layer applying a linear classifier to the feature
vector x at the penultimate layer. Given the logit vector and the ground truth
label $y\in\\{1,...,C\\}$, the original softmax cross entropy loss is
formulated as

$\displaystyle
Loss(\textbf{{z}},y)=-\log\frac{e^{z_{y}}}{\sum_{c=1}^{C}{e^{z_{c}}}}=\log\biggl{[}1+\sum_{c\neq
y}^{C}e^{(z_{c}-z_{y})}\biggl{]}$ (1)

Kobayashi [15] indicated that Eq. (1) can be converted to Eq. (2) by applying
the softplus function, which is described by $softplus(x)=log(1+e^{x})$.

$\displaystyle Loss(\textbf{{z}},y)=softplus\biggl{[}\log\Bigl{\\{}\sum_{c\neq
y}e^{z_{c}}\Bigl{\\}}-z_{y}\biggl{]}$ (2)

Subsequently, we apply to the log-sum-exp transformation that aggregates the
logits of the nontarget classes $c\neq y$ in Eq. (2), and the result of the
conversion indicates in Eq. (3).

$\displaystyle Loss(\textbf{{z}},y)=softplus(z_{c^{*}}-z_{y}+\bm{\hat{\rho}})$
(3) $\displaystyle\bm{\hat{\rho}}=\log\biggl{[}\sum_{c\neq
y}e^{(z_{c}-z_{c^{*}})}\biggl{]}$

where $z_{c^{*}}$ is the maximum logit of incorrect class and $\hat{\rho}$ is
a bias margin. The softplus function, as shown in Fig. 1, is similar to the
hinge loss. This formulation reveals that the softmax cross entropy loss
measures the significance of the target logit $z_{y}$ compared with the other
logit $z_{c^{*}}$. Then, the original softmax cross entropy loss induces a
large-margin classifier depending on the bias $\hat{\rho}$, and we call this
LMSCE loss.

Inspired by this conversion method using the softplus function, we convert
LDAM loss described in Eq. (4) to the large-margin loss in Eq. (5).

$\displaystyle
Loss_{ldam}(\textbf{{z}},y)=-\log\frac{e^{s(z_{y}-\Delta_{y})}}{e^{s(z_{y}-\Delta_{y})}+\sum_{j\neq
y}{e^{z_{j}}}}$ (4) $\displaystyle
Loss_{ldam}(\textbf{{z}},y)=softplus(s(z_{c^{*}}-z_{y})+s\Delta_{y}+\bm{\hat{\rho}})$
(5) $\displaystyle\bm{\hat{\rho}}=\log\biggl{[}\sum_{c\neq
y}e^{s(z_{c}-z_{c^{*}})}\biggl{]}$

where $\Delta_{y}$ is a class-dependent margin, which is described by
$\Delta_{j}=\frac{M}{n_{j}^{1/4}}\>for\>j\in\\{1,..,k..,C\\}$. $n_{j}$ is the
number of training samples in class $j$, $M$ is the maximum size of margin
that is tuned as a hyper-parameter, and $s$ is a scaling factor. As shown in
Eq. (5), LDAM loss can be converted by the large margin framework as well as
LMSCE loss. Additionally, compared between LDAM loss and LMSCE loss in Fig. 2
(a-b), LDAM loss is given a wider margin $\Delta_{j}$ than LMSCE loss, and the
range of margins vary from class.

###  III-B Enlarged Large Margin loss

TABLE I: Comparison of the results on imbalanced CIFAR datasets.

---
| CIFAR10-LT | CIFAR10-Step | CIFAR100-LT | CIFAR100-Step
Loss function | Ratio=10 | Ratio=100 | Ratio=10 | Ratio=100 | Ratio=10 | Ratio=100 | Ratio=10 | Ratio=100
CE | 86.92±0.06 | 70.71±0.20 | 84.44±0.35 | 64.36±0.30 | 56.54±0.35 | 38.54±0.41 | 54.48±0.12 | 38.71±0.19
LMSCE | 87.03±0.24 | 72.34±0.77 | 85.08±0.23 | 62.29±0.63 | 56.44±0.16 | 36.90±0.14 | 50.70±0.50 | 38.89±0.11
Focal | 86.41±0.18 | 71.59±0.53 | 84.54±0.28 | 66.13±2.46 | 55.98±0.19 | 38.39±0.52 | 54.88±0.36 | 38.76±0.18
Equalization v2 | 86.14±0.28 | 69.49±0.12 | 84.11±0.59 | 61.40±1.58 | 56.37±0.10 | 38.09±0.52 | 53.42±0.32 | 38.35±0.04
IB FLocal | 82.62±0.52 | 68.82±0.52 | 87.87±0.19 | 74.78±0.63 | 56.52±0.32 | 40.61±0.71 | 56.94±0.36 | 43.70±0.49
LDAM | 86.68±0.25 | 73.67±0.29 | 84.55±0.39 | 65.21±1.05 | 55.38±0.25 | 39.82±0.45 | 50.79±0.37 | 39.37±0.13
MM-LDAM | 86.23±0.24 | 71.51±0.48 | 83.86±0.25 | 56.20±1.18 | 53.25±0.30 | 36.89±0.25 | 46.34±0.32 | 38.13±0.06
LDAM + DRW | 87.36±0.28 | 77.16±0.33 | 87.65±0.27 | 77.59±0.14 | 57.71±0.32 | 42.97±0.62 | 57.03±0.09 | 45.75±0.16
MM-LDAM + DRW | 86.63±0.17 | 77.79±0.21 | 86.87±0.08 | 76.31±0.43 | 51.13±0.42 | 39.31±0.38 | 48.35±0.27 | 40.74±0.42
ELM w/o $\Delta_{y}$ ($\lambda$=1.0) | 47.30±2.01 | 60.37±0.67 | 53.89±6.27 | 64.14±0.20 | 17.69±3.68 | 33.63±1.21 | 36.64±4.96 | 37.98±0.41
ELM w/o $\Delta_{y}$ ($\lambda$=0.5) | 84.12±0.18 | 72.44±0.43 | 83.68±0.67 | 68.06±0.83 | 51.97±0.49 | 37.10±0.81 | 51.32±0.84 | 39.32±0.21
ELM w/o $\Delta_{y}$ ($\lambda$=0.1) | 86.76±0.08 | 71.61±1.08 | 85.06±0.03 | 65.61±1.08 | 55.64±0.38 | 38.04±0.39 | 53.69±0.65 | 39.47±0.23
ELM ($\lambda$=1.2) | 83.40±0.49 | 76.35±0.63 | 75.77±0.48 | 74.73±0.46 | 53.88±0.83 | 40.14±0.71 | 54.49±0.59 | 43.32±0.27
ELM ($\lambda$=1.0) | 87.61±0.04 | 77.61±0.86 | 87.53±0.35 | 76.09±0.90 | 57.45±0.45 | 40.93±0.82 | 57.59±0.46 | 43.37±0.83
ELM ($\lambda$=0.5) | 87.82±0.11 | 75.14±0.47 | 87.38±0.17 | 71.26±0.85 | 58.30±0.19 | 41.61±0.86 | 57.01±0.32 | 40.82±0.31
ELM ($\lambda$=0.1) | 87.05±0.07 | 73.34±0.84 | 84.93±0.20 | 66.44±0.46 | 56.59±0.06 | 40.36±0.45 | 51.93±0.23 | 39.62±0.12
ELM + DRW ($\lambda$=1.2) | 81.82±0.27 | 77.57±0.26 | 72.62±0.40 | 77.06±1.14 | 51.92±0.72 | 40.07±1.32 | 52.15±0.84 | 43.45±0.84
ELM + DRW ($\lambda$=1.0) | 88.16±0.17 | 78.89±0.03 | 88.42±0.11 | 78.03±0.16 | 58.93±0.33 | 41.96±0.33 | 58.17±0.60 | 43.75±0.24
ELM + DRW ($\lambda$=0.5) | 88.31±0.08 | 78.05±0.13 | 88.67±0.17 | 78.17±0.29 | 58.82±0.27 | 42.35±1.16 | 60.13±0.12 | 45.74±0.33
ELM + DRW ($\lambda$=0.1) | 87.68±0.32 | 77.84±0.61 | 88.03±0.09 | 78.17±0.49 | 58.35±0.15 | 43.07±0.55 | 57.74±0.60 | 46.00±0.18

In Section $\rm{III}$-A, we confirmed that LDAM loss is given wider and class-
specific larger margins than LMSCE loss. We consider that LDAM loss can be
balanced learning for the imbalanced dataset because it includes a large
margin depending on the label distribution. However, LDAM loss utilizes only
the large margin for the correct class, not the large margin for the maximum
logit of the incorrect class. As shown in Eq. (5), since LDAM loss implicitly
learns the difference between $z_{c^{*}}$ and $z_{y}$, we consider that LDAM
loss should also be used in the large margin for the maximum logit
$z_{c^{*}}$. Therefore, we propose a novel Enlarged Large Margin (ELM) loss to
adopt the large margin toward $z_{c^{*}}$. ELM loss defined by the softplus
function is shown in Eq. (6).

$\displaystyle
Loss_{elm}(\textbf{{z}},y)=softplus(s(z_{c^{*}}-z_{y})+s\Delta_{y}-s\lambda\Delta_{c^{*}}+\bm{\hat{\rho}})$
(6) $\displaystyle\bm{\hat{\rho}}=\log\biggl{[}\sum_{c\neq
y}e^{s(z_{c}-z_{c^{*}})}\biggl{]}$

where $\Delta_{c^{*}}$ is the margin for the maximum logit of the other class,
and it is defined by the same calculation as the margin of LDAM loss.
$\lambda$ is a hyper parameter that determine the strength of
$\Delta_{c^{*}}$. When we return Eq. (6) defined by the softplus function to
the usual cross-entropy loss form, the final loss function is indicated in Eq.
(7).

$\displaystyle
Loss_{elm}(\textbf{{z}},y)=-\log\frac{e^{s(z_{y}-\Delta_{y}+\lambda\Delta_{c^{*}})}}{e^{s(z_{y}-\Delta_{y}+\lambda\Delta_{c^{*}})}+\sum_{j\neq
y}{e^{z_{j}}}}$ (7)

As shown in Fig. 2 (c), ELM loss can take a wider large margin for
$\lambda\Delta_{c^{*}}$ compared to the conventional LMSCE loss and LDAM loss.
When the correct class is the rare class in imbalanced classification,
$z_{c^{*}}$ often takes the logit of the frequent class. Consequently, the
large margin $\Delta_{y}$ used in LDAM loss can be taken, allowing for greater
distance separation. In contrast, in ELM loss, the large margin
$\lambda\Delta_{c^{*}}$ can be used in addition to $\Delta_{y}$, and the
distance between $z_{c^{*}}$ and $z_{y}$ can be greater than LDAM loss.
Further, in the imbalance dataset, the most frequently occurring sample in the
mini-batch is the majority class, and then LDAM loss has employed the small
and the same value of $\Delta_{y}$ in most cases. In contrast, since EML loss
can use the margins of classes $z_{c^{*}}$, more variety of margins can be
used for majority classes.

##  IV EXPERIMENTS

TABLE II: Comparison of the results on ImageNet-LT dataset.

---
| Accuracy
Loss function | Top 1 | Top 3 | Top 5
CE | 47.66 | 66.87 | 74.19
LMSCE | 45.57 | 64.55 | 71.61
Focal | 47.76 | 66.68 | 73.68
Equalization v2 | 47.84 | 67.22 | 74.60
IB Focal | 47.77 | 67.65 | 74.88
LDAM | 50.53 | 69.20 | 75.64
MM-LDAM | 43.33 | 63.02 | 70.46
LDAM + DRW | 53.38 | 69.36 | 75.35
MM-LDAM + DRW | 47.74 | 65.04 | 71.38
ELM ($\lambda$=1.0) | 55.49 | 71.88 | 77.58
ELM ($\lambda$=0.5) | 54.24 | 71.42 | 77.22
ELM ($\lambda$=0.1) | 51.50 | 69.53 | 76.11
ELM + DRW ($\lambda$=1.0) | 51.22 | 71.10 | 77.23
ELM + DRW ($\lambda$=0.5) | 54.50 | 70.98 | 76.40
ELM + DRW ($\lambda$=0.1) | 53.68 | 69.88 | 75.72

TABLE III: Comparison of the results on Places-LT dataset.

---
| Accuracy
Loss function | Top 1 | Top 3 | Top 5
CE | 26.63 | 47.91 | 58.48
LMSCE | 24.34 | 44.70 | 54.83
Focal | 26.88 | 48.07 | 58.21
Equalization v2 | 26.75 | 49.13 | 59.90
IB FLocal | 32.51 | 54.86 | 65.12
LDAM | 26.28 | 46.47 | 56.06
MM-LDAM | 22.87 | 44.51 | 54.94
LDAM + DRW | 31.81 | 50.81 | 59.49
MM-LDAM + DRW | 30.58 | 50.97 | 60.39
ELM ($\lambda$=1.0) | 31.47 | 51.15 | 60.28
ELM ($\lambda$=0.5) | 28.42 | 48.57 | 58.46
ELM ($\lambda$=0.1) | 26.66 | 47.15 | 57.51
ELM + DRW ($\lambda$=1.0) | 32.16 | 53.49 | 63.07
ELM + DRW ($\lambda$=0.5) | 33.02 | 53.15 | 62.17
ELM + DRW ($\lambda$=0.1) | 32.61 | 52.06 | 61.05

###  IV-A Datasets and learning conditions

CIFAR. The original version of CIFAR-10 and CIFAR-100 contains 5,000 and 500
training images of $32\times 32$ pixels with 10 and 100 classes, respectively.
Following [3], we reduced the number of training images and make long-tailed
and step distributions. We used the imbalanced ratio to denote between the
samples of the most frequent and the least frequent categories and make two
types of datasets with different imbalance ratios $(e.g.,10,100)$. Evaluation
images comprise original test images. For data pre-processing, training
samples were randomly cropped to $32\times 32$ in the case that padding size
is 4, and flipped horizontally.

We aligned all training conditions with the rule of [3]. The network was the
ResNet32 [3] with full-scratch learning. The batch size is set to 128, the
epochs are set to 200, and the optimizer is Stochastic Gradient Descent (SGD)
with a momentum of 0.9 and weight decay of $2\times 10^{-4}$. The initial
learning rate was 0.1, and we use a standard learning schedule that was
decayed by 0.01 at the 160 epoch and again at the 180 epoch. We also used a
warm-up learning rate schedule for the first 5 epochs. Experiments were
conducted three times by altering the random initial values of the parameters.
The average accuracy of the three times experiments was used for the
evaluation.

(a) LDAM  (b) MM-LDAM (c) ELM (Ours)
---
Figure 3: Visualization of penultimate layer in ResNet32 trained on the
CIFAR10-LT with ratio=100. The color bar shows the class number and indicates
a small number of samples as the class number increase.

Large-scale dataset. We used two types of common large-scale imbalanced
datasets, ImageNet-LT [1], and Places-LT [1]. ImageNet-LT [1] contains 115,800
images for 1,000 categories, with a maximum of 1,280 images per category and a
minimum of 5 images per category. It is sampled from ImageNet-2012[13]
following the Pareto distribution. Further, a long-tailed version of Places-2
[22] is constructed in a similar way. Places-LT [1] contains 184.5K images for
365 categories, with a maximum of 4,980 images per category and a minimum of 5
images per category. The gap between the head and tail classes is even larger
than ImageNet-LT.

For data pre-processing, training samples were randomly cropped to $224\times
224$, flipped horizontally, and randomly changed brightness values. In
inference, samples were resized to $256\times 256$ and cropped $224\times 224$
in the center. We used the ResNet50 pre-trained on the ImageNet [13] as
following [4]. We set the batch size to 256 and the number of epochs to 90.
The optimizer was the SGD with momentum and weight decay ($lr$=$1\times
10^{-1}$, $momentum=0.9$, $2\times 10^{-4}$). We used a standard learning rate
schedule that is decayed by 0.1 at 50, 70, and 85 epochs, and a warm-up
learning rate schedule for the first 5 epochs for training.

Experiments were conducted one time by altering the random initial values of
the parameters, and we used accuracy, which is the percentage of images whose
labels are at the top predicted by the network (top-1, top-3, and top-5
accuracy) as the evaluation metric.

###  IV-B Results

As comparison methods, we employed the standard softmax cross entropy (CE)
loss and conventional losses for imbalanced classification, which are Focal
loss [12], Equalization v2 loss [8], and IB Focal loss [21]. Furthermore, we
also compared conventional margin-based losses, which are LMSCE loss [18],
LDAM loss [4], and MM-LDAM loss [6]. Table 1 presents the evaluation results
on test images in the imbalanced CIFAR datasets[3]. The bold letters show the
best accuracy. In imbalanced CIFARs, ELM loss was better performance compared
with conventional methods and ELM loss with DRW was the highest accuracy in
all ratios. Even though ELM loss was a simple expansion method of LDAM loss,
it was effective in comparison with the conventional loss functions for
imbalanced classification. Additionally, by setting a smaller $\lambda$, the
proposed large margin could be used more effectively because ELM loss extends
the margins further in addition to the original LDAM margin, and the margins
may become too large depending on the number of training samples in the
dataset. Moreover, when only the margins of ELM loss were used without the
margins originally used for LDAM loss (ELM w/o $\Delta_{y}$, in Table 1), the
accuracy was also better than that of CE loss and LMSCE loss by setting small
$\lambda$.

Table 2 and Table 3 present the evaluation results on test images in the
ImageNet-LT and Places-LT datasets [1]. In ImageNet-LT, EML loss had a higher
accuracy than other margin-based losses, and improved top1 accuracy by over
$4.9\%$ than LDAM loss. Furthermore, EML loss with DRW was the highest
accuracy in comparison with conventional approaches for the imbalanced
dataset. In Places-LT, ELM loss was also the better performance compared to
conventional margin-based methods with top1 accuracy. These results
demonstrate that ELM loss is an effective generalizable method on a variety of
imbalanced datasets. In ImageNet-LT, the maximum number of images is 1,280 and
the minimum number of images is 5, and the imbalance ratio is larger than in
imbalanced CIFAR and there are classes where the margins are too small. We
consider that the margins with $\lambda=1.0$ work the best in that case.
Additionally, it is possible that a class balancing weight is too strong for
the large-scale dataset if EML loss with DRW is used, and generalization
performance may decrease due to over-fitting. In that case, we consider using
only ELM loss is the best performance.

Figure 3 illustrates the visualization results of features at a penultimate
layer that is compressed to two dimensions by UMAP[23] when we evaluated
training samples. The color bar shows the class number and indicates a small
number of samples as the class number increase. As shown in Fig. 3, in
conventional loss functions such as LDAM loss and MM-LDAM loss, the distance
between minority classes and other classes is quite close, and the feature
spaces of minority classes are not independent. However, when we employed ELM
loss, the feature space is more effective for discrimination since the
distance between the minority class and the other classes is more independent.
These results indicate that ELM loss has a stronger regularization than LDAM
loss and other margin-based losses and demonstrate the effectiveness of
increasing the margins between the logit of the correct class and the maximum
logit of the wrong class.

##  V CONCLUSION

In this paper, we reconverted a formula for LDAM loss in terms of a large-
margin perspective and proposed a novel ELM loss function with even wider
margins than LDAM loss. Through experiments on imbalanced classification, ELM
loss significantly improved the accuracy compared with conventional large-
margin losses. Furthermore, we confirmed that ELM loss generated effective
feature space for classification.

To further improve the accuracy of minority classes, while maintaining the
accuracy of majority classes, the combination of EML loss with effective
strategies for long-tailed classification such as two-stage learning is one of
our future works.

## References

  * [1] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S.X. Yu, “Large-scale long-tailed recognition in an open world,” In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.2537–2546, 2019.
  * [2] G.V. Horn, O.M. Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie, “The inaturalist species classification and detection dataset,” In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.8769–8778, 2018.
  * [3] Y. Cui, M. Jia, T.Y. Lin, Y. Song, and S. Belongie, “Class-balanced loss based on effective number of samples,” In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.9268–9277, 2019.
  * [4] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma, “Learning imbalanced datasets with label-distribution-aware margin loss,” In Proc. 33rd International Conference on Neural Information Processing Systems, pp.1567-1578, 2019.
  * [5] B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis, “Decoupling representation and classifier for long-tailed recognition,” In Proc. In International Conference on Learning Representations, 2020.
  * [6] H. Kang, T. Vu and C.D. Yoo, “Learning Imbalanced Datasets With Maximum Margin Loss,” 2021 IEEE International Conference on Image Processing (ICIP), pp.1269-1273, 2021.
  * [7] J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan, ‘Equalization loss for long-tailed object recognition,” In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.11662–11671, 2020.
  * [8] J. Tan, X. Lu, G. Zhang, C. Yin, and Q. Li, “Equalization loss v2: A new gradient balance approach for long-tailed object detection,” In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.1685–1694, 2021.
  * [9] Y. Li, T. Wang, B. Kang, S. Tang, C. Wang, J. Li, and J. Feng, “Overcoming classifier imbalance for long-tail object detection with balanced group softmax,” In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.10991–11000, 2020.
  * [10] T. Wang, Y. Zhu, C. Zhao, W. Zeng, J. Wang, and M. Tang, “Adaptive class suppression loss for long-tail object detection,” In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.3103–3112, 2021.
  * [11] J. Wang, W. Zhang, Y. Zang, Y. Cao, J. Pang, T. Gong, K. Chen, Z. Liu, C.C. Loy, and D. Lin, “Seesaw loss for long-tailed instance segmentation,” In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.9695–9704, 2021.
  * [12] T.Y. Lin, P. Goyal, R. Girshick, K. He, and P. Doll ar, “Focal loss for dense object detection,” In Proc. the IEEE International Conference on Computer Vision, pp.2980–2988, 2017.
  * [13] A. Krizhevsky, I. Sutskever, and G.E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, pp. 1097–1105, 2012.
  * [14] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.770-778, 2016.
  * [15] T. Kobayashi, “Large Margin In Softmax Cross-Entropy Loss,” In Proc. British Machine Vision Conference, pp.139, 2019.
  * [16] J.A. Suykens and J. Vandewalle, “Least squares support vector machine classifiers,” Neural processing letters, vol.9, no.3, pp.293–300, 1999.
  * [17] K. Crammer and Y. Singer, “On the algorithmic implementation of multiclass kernel based vector machines,” Journal of Machine Learning Research, 2:265–292, 2001.
  * [18] W. Liu, Y. Wen, Z. Yu, and M. Yang, “Large-margin softmax loss for convolutional neural networks,” In Proc. International Conference on Machine Learning, vol.2, p.7, 2016.
  * [19] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hypersphere embedding for face recognition,” In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.212–220, 2017.
  * [20] F. Wang, J. Cheng, W. Liu, and H. Liu, “Additive margin softmax for face verification,” IEEE Signal Processing Letters, vol.25, no.7, pp.926–930, 2018.
  * [21] S. Park, J. Lim, Y. Jeon, and J.Y. Choi, “Influence-balanced loss for imbalanced visual classification,” In Proc. the IEEE/CVF International Conference on Computer Vision, pp.735-744, 2021.
  * [22] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” IEEE transactions on pattern analysis and machine intelligence, 40(6), pp.1452-1464, 2017.
  * [23] L. McInnes, J. Healy, N. Saul and L. Großberger, “UMAP: Uniform Manifold Approximation and Projection,” Journal of Open Source Software, 3(29), 2018.
  * [24] P.L. Bartlett, D.J. Foster, and M. Telgarsky, “Spectrally-normalized margin bounds for neural networks,” Advances in neural information processing systems, vol.30, pp.6241-6250, 2017.


# Realizing the ultimate scaling of the convection turbulence by spatially
decoupling the thermal and viscous boundary layers

Shufan Zou  State Key Laboratory for Turbulence and Complex Systems,
Department of Mechanics and Engineering Science, College of Engineering,
Peking University, Beijing 100871, China  Yantao Yang<EMAIL_ADDRESS>State Key Laboratory for Turbulence and Complex Systems, Department of
Mechanics and Engineering Science, College of Engineering, Peking University,
Beijing 100871, China  Beijing Innovation Center for Engineering Science and
Advanced Technology, Peking University, Beijing 100871, China  Institute of
Ocean Research, Peking University, Beijing 100871, China

###### Abstract

Turbulent convection plays a crucial role in many natural environments,
ranging from Earth ocean, mantle and outer core, to various astrophysical
systems. For such flows with extremely strong thermal driving, an ultimate
scaling was proposed for the heat flux and velocity. Despite numerous
experimental and numerical studies, a conclusive observation of the ultimate
regime has not been reached yet. Here we show that the ultimate scaling can be
perfectly realized once the thermal boundary layer is fully decoupled from the
viscous boundary layer and locates inside the turbulent bulk. The heat flux
can be greatly enhanced by one order of magnitude. Our results provide
concrete evidences for the appearance of the ultimate state when the entire
thermal boundary layer is embedded in the turbulent region, which is probably
the case in many natural convection systems.

##  I Introduction

The paradigmatic model for turbulent convection is the Rayleigh-Bénard (RB)
flow, namely the buoyancy-driven flow within a fluid layer heated from below
and cooled from above. Such flow system can be found in many natural
environments [1, 2, 3, 4]. For a given fluid with certain Prandtl number
$Pr=\nu/\kappa$, the driving force is measured by the Rayleigh number
$Ra=g\alpha\Delta H^{3}/\nu\kappa$. Here, $\nu$ is kinematic viscosity,
$\kappa$ is thermal diffusivity, $g$ is the gravitational acceleration,
$\alpha$ is thermal expansion coefficient, $\Delta$ is the temperature
difference across the layer, and $H$ is the layer height, respectively. The
most fundamental question is how the heat flux and the turbulent level of the
flow depend on $Ra$ and $Pr$ [5, 6, 7, 8]. Here, the heat flux is usually
measured by the Nusselt number $Nu$, i.e. the ratio between the convective
flux to the conductive flux. The turbulent level of the flow by the Reynolds
number $Re=UH/\nu$, with $U$ being some characteristic value of the flow
velocity.

In RB flow, thermal and viscous boundary layers develop next to the top and
bottom plates, with the convective bulk in between and full of thermal plumes.
In most early experiments and simulations the viscous boundary layer is
laminar, and the heat flux follows the “classic” scaling law $Nu\sim
Ra^{\gamma}$ with $\gamma\approx 1/3$ [9]. It was hypothesized that when $Ra$
is large enough, saying the thermal driving is extremely strong, the viscous
boundary layer becomes fully turbulent due to the shearing exposed by the
vigorous bulk flow, and the system enters the “ultimate” state where the heat
flux no longer depends on the viscosity of the fluid [10, 11, 12]. The heat
flux then follows the ultimate scaling with $\gamma=1/2$, which predicts $Nu$
many orders of magnitude larger than the classic scaling at very large $Ra$ as
in various natural environments.

Tremendous efforts have been made to achieve the ultimate scaling in RB flow
by both experiments and simulations. Fluid with very small $Pr$, such as
mercury, is used so that the thermal boundary layer is much thicker than the
viscous one and extends into the turbulent bulk [13, 14, 15]. Recent state-of-
art simulations and experiments push $Ra$ to very high values [15, 16, 17,
18]. Some of these studies reported evidences of the transition to the
ultimate regime [13, 16, 17]. Wall roughness was introduced to trigger the
transition of the momentum boundary layer to fully turbulent state, but this
only works for a limit range of $Ra$ for a given wall roughness [19, 20].
Homogeneous convection in fully periodic domain exhibits the ultimate scaling,
since in such flow all boundary layers are removed and only the turbulent bulk
plays a role [21, 22]. The radiative convection with internal heating and the
convection with background oscillation can both greatly enhance the exponent
$\gamma$ from the classic-scaling value, but such treatments change the
dynamics of the convection system by introducing extra source terms for the
thermal field or the momentum field [23, 24, 25, 26].

##  II Governing equations and numerical methods

Here by specially designed numerical experiments, we confirm the physical
conjecture of the ultimate paradigm. Once the whole thermal boundary layer is
spatially decoupled from the viscous one and locate entirely in the turbulent
flow region, the ultimate scaling of the heat flux can be perfectly realized
even at low to moderate Rayleigh numbers. Specifically, we consider the RB
flow between two parallel plates which are perpendicular to the direction of
the gravity and separated by a distance $H$. A homogeneous layer of height $h$
is introduced at top and bottom of the domain. Within this layer the
temperature is uniform and equal to the temperature of the adjacent plate,
which can be readily realized by using an immersed-boundary technique. The
buoyancy-force term takes effect only between the two homogeneous layers. By
doing so the thermal boundary layer is lifted by a height $h$ from the plate
where the viscous boundary layer occurs. If $h$ is large enough, the two
boundary layers can be fully decoupled.

Specifically, we consider a fluid layer bounded by two parallel plates which
are perpendicular to the gravity and separated by a distance $H$. The
Oberbeck-Bbuossinesq approximation is utilized to take care of the buoyancy
effects. First, the fluid density depends linearly on the scalar field as
$\rho=\rho_{0}(1-\alpha\theta)$. Here $\rho_{0}$ is the reference density at
the reference temperature $T_{0}$, and $\theta$ is the temperature deviation
from $T_{0}$. Second, the variation of density is relatively small so that all
the fluid properties can be treated as constants, and only the buoyancy force
needs to be included. The governing equations then read

$\displaystyle\partial_{t}u_{i}+\mathbf{u}\cdot\nabla\mathbf{u}$
$\displaystyle=$ $\displaystyle-\frac{1}{\rho}\nabla
p+\nu\,\nabla^{2}\mathbf{u}+\alpha\,\theta\,\mathbf{g},$ (1)
$\displaystyle\partial_{t}\theta+\mathbf{u}\cdot\nabla\theta$ $\displaystyle=$
$\displaystyle\kappa\,\nabla^{2}\theta,$ (2)
$\displaystyle\nabla\cdot\mathbf{u}$ $\displaystyle=$ $\displaystyle 0.$ (3)

Here $\mathbf{u}$ is three dimensional velocity, $p$ is pressure, and
$\partial_{t}$ is the partial derivative with respect to time, respectively.
The above equations are non-dimensionalized by the layer height $H$, the
temperature difference between two plates $\Delta$, and the free-fall velocity
$U=\sqrt{g\alpha\Delta H}$. The two plates are no-slip and the periodic
boundary conditions are applied to the two horizontal directions.

Direct numerical simulations were conducted with the in-house code, which
employs the finite difference scheme and a fraction-time-step method. The code
has been tested for various wall-turbulence and convection turbulence [27]. To
introduce the homogeneous thermal layer, an extra source term is included in
the temperature equation (2) based on the idea of the immersed boundary method
[28]. Also the buoyancy force in the momentum equation (1) is turned off
inside the homogeneous thermal layer. Grid-independence test has been
conducted in order to ensure the resolution is adequate. The three-dimensional
volume rendering in figure 1 is generated by the open source software VisIt
[29].

##  III Main results

In figure 1 we compare one thermal field of the normal RB flow and that with
homogeneous layers at $Ra=10^{6}$ and $Pr=1$. Here $h$ is chosen as $0.2H$. We
only show the fluid with temperature larger than $90\%\Delta$ and smaller than
$10\%\Delta$. For normal RB flow at this relatively low $Ra$, the large
thermal anomaly cannot be transported very far from the plates, which limits
the global heat flux. On the contrary, with two homogeneous thermal layers the
thermal plumes with large anomaly are significantly stronger, and spread over
the entire convection bulk. Accordingly, the heat flux is greatly enhanced in
the flow with homogeneous thermal layers.

Figure 1: Volume rendering of the thermal field at $Ra=10^{6}$ and $Pr=1$.
Yellow and blue indicate the temperature larger than $90\%\Delta$ and smaller
than $10\%\Delta$, respectively. A: the normal Rayleigh-Bénard flow. B: the
modified flow with two homogeneous thermal layers of the height $h=0.2H$. The
flow in B has much stronger thermal plumes in the bulk, while for the flow in
A the large temperature anomaly can hardly be transported into the bulk.

We then present the main results of the current study, i.e. the ultimate
scalings of heat flux and flow velocity, which in their full form read

$Nu\sim Pr^{1/2}Ra_{b}^{1/2},\quad Re\sim Pr^{-1/2}Ra_{b}^{1/2}.$ (4)

To account for the height of two homogeneous thermal layers, the bulk Rayleigh
number is calculated as $Ra_{b}=Ra(1-2h/H)^{3}$. The Nusselt number is defined
as $Nu=q(H-2h)/(\kappa\Delta)$ with $q$ being the total dimensional heat flux.
When $h=0$, both definitions recover the usual form of RB flow. The Reynolds
number is $Re=u_{rms}H/\nu$ and $u_{rms}$ is the root-mean-square velocity of
the whole domain, since the flow motion is confined by two plates, including
the homogeneous thermal layers. We first look at the dependences of $Nu$ and
$Re$ on $Ra_{b}$ for fixed $Pr=1$. Three decades of $Ra$ are covered from
$10^{5}$ up to $10^{8}$. The height of the homogeneous thermal layer is set at
$h=0.2H$ so that for all simulations the momentum boundary layer is fully
embedded inside the homogeneous thermal layer. Figures 2A and 2B show that
both $Nu$ and $Re$ follow the ultimate scaling law (4) perfectly, even though
the Rayleigh number is not very large. Moreover, both the heat flux and flow
velocity are greatly enhanced by introducing the homogeneous thermal layers.
For $Ra_{b}\geq 10^{6}$, the modified flow generates a heat flux more than one
order of magnitude higher than that for the normal RB flow. For instance, the
Nusselt number is comparable for the modified flow at $Ra\approx 10^{8}$ and
the normal RB flow at $Ra\approx 10^{11}$.

Figure 2: The ultimate scaling of the heat flux and flow velocity. Solid
symbols and lines depict the ultimate scalings of the modified flow with
homogeneous thermal layers. Open symbols and dashed lines show the normal RB
flow and the classic scaling. The corresponding exponents are determined by
linear fitting. A and B: The Nusselt number $Nu$ and Reynolds number $Re$
versus the increasing bulk Rayleigh number $Ra_{b}$ for a fixed Prandtl number
$Pr=1$. C and D: $Nu$ and $Re$ versus the increasing $Pr$ for fixed
$Ra=3\times 10^{6}$.

For the second group of simulations, we fix $Ra=3\times 10^{6}$ and increase
$Pr$ from $0.01$ to $10$, again covering three orders of magnitude. The height
of the homogeneous layers is $h=0.2H$. As shown in figures 2C and 2D, the
behaviors of $Nu$ and $Re$ are also very close to the ultimate scaling (4).
The scaling exponent for $Nu$ given by the linear fitting is indistinguishable
from the theoretically predicted value $1/2$. The exponent for $Re$ obtained
from the numerical data is slightly different from $-1/2$. For the moderate
$Ra=3\times 10^{6}$ considered here, the flow is not fully turbulent at high
Prandtl number regime. Therefore $Re$ decreases faster than that predicted by
the ultimate scaling. Another interesting observation is that at low $Pr$
region, the difference between the modified flow and the normal RB flow is
very small. This is expectable since for small $Pr$, the viscous boundary
layer is thinner than the thermal one in the normal RB flow, and part of the
thermal boundary layer is already in the turbulent bulk.

Figure 3: The effect of the thermal layer height on the heat-flux scaling. A:
The heat flux versus $Ra_{b}$ for different $h$, which increases from $0$ to
$0.2$ as color changes from blue to red. B: The scaling exponent $\gamma$
versus the height $h$. In both panels the solid and dashed lines mark the
ultimate scaling with the exponent $1/2$ and the classic scaling with the
exponent $1/3$, respectively.

To further demonstrate that the ultimate scaling is satisfied when the thermal
and viscous boundary layers are decoupled from each other, we run simulations
with increasing $h$ for three different Rayleigh numbers $Ra=10^{5}$,
$10^{6}$, and $10^{7}$. The Prandtl number is fixed at $Pr=1$. We focus on the
behaviors of Nusselt number, which are shown in figure 3. As $h$ increases,
i.e. the thermal boundary layers are gradually decoupled from the viscous
ones, the heat flux is enhanced and the scaling exponent transits from the
classic value of $1/3$ to the ultimate value $1/2$. In figure 3 we plot the
scaling exponent $\gamma$ versus the height of the homogeneous thermal layer
$h$. For small $h$, $\gamma$ increases rapidly to a value larger than $1/2$.
At this overshoot the thermal boundary layer already enters the turbulent bulk
for large $Ra$ but is still coupled with the viscous boundary layer for low
$Ra$. When $h$ is large enough, for all three $Ra$’s the thermal boundary
layer is fully within the turbulent region, and the exponent $\gamma$
gradually approaches the ultimate value $1/2$.

##  IV Conclusion and discussion

In summary, we demonstrate that once the thermal boundary layers decouple with
the viscous ones and locate within the turbulent convection bulk, the ultimate
scaling of the heat flux and flow velocity can be perfectly realized even at
relatively low Rayleigh numbers. Our results support the physical picture of
the ultimate state of convection turbulence, namely when the momentum boundary
layers become fully turbulent, the heat flux is independent of the fluid
viscosity. In line with the unifying model for RB convection [11, 12], the
current flow corresponds to the IVl regime. Compared to the homogeneous
convection [21, 22], the current flow configuration is more close to the
ultimate regime since the thermal boundary layers are included and the
ultimate scaling is achieved.

A definitive proof for the existence of the ultimate regime relies on the
direct observation of the ultimate scaling in the experiments and simulations
of the normal RB convection. Nevertheless, one can anticipate that, based on
the present study, convection flows will eventually enter the ultimate state
when the boundary layer region becomes fully turbulent at extremely high
thermal driving. Such conditions are very likely satisfied in natural systems
such as the Earth outer core and stars. Transition towards the ultimate regime
has been reported in recent studies [16, 17], and we may expect that a fully
developed ultimate state will be observed in experiments in near future.

Acknowledgements: This work is supported by the Major Research Plan of
National Natural Science Foundation of China for Turbulent Structures under
the Grants 91852107 and 91752202. Y.Yang also acknowledges the partial support
from the Strategic Priority Research Program of Chinese Academy of Sciences
under the Grant No. XDB42000000.

## References

  * Spiegel [1971] E. A. Spiegel, Convection in stars I. basic Boussinesq convection, Annu. Rev. Astro. Astrophys. 9, 323 (1971).
  * Tackley  _et al._ [1993] P. J. Tackley, D. J. Stevenson, G. A. Glatzmaier, and G. Schubert, Effects of an endothermic phase transition at 670 km depth in a spherical model of convection in the Earth’s mantle, Nature 361, 699 (1993).
  * Marshall and Schott [1999] J. Marshall and F. Schott, Open-ocean convection: Observations, theory, and models,, Rev. Geophys 37, 1 (1999).
  * Heimpel  _et al._ [2012] M. Heimpel, J. Aurnou, and J. Wicht, Simulation of equatorial and high-latitude jets on Jupiter in a deep convection model, Phys. Rev. Lett. 108, 024502 (2012).
  * Ahlers  _et al._ [2009] G. Ahlers, S. Grossmann, and D. Lohse, Heat transfer and large scale dynamics in turbulent Rayleigh-Bénard convection, Rev. Mod. Phys. 81, 503 (2009).
  * Chillà and Schumacher [2012] F. Chillà and J. Schumacher, New perspectives in turbulent Rayleigh?Bénard convection., Eur. Phys. J. E 35, 58 (2012).
  * Xia [2013] K.-Q. Xia, Current trends and future directions in turbulent thermal convection, Theor. Appl. Mech. Lett. 3, 052001 (2013).
  * Plumley and Julien [2019] M. Plumley and K. Julien, Scaling laws in Rayleigh-Bénard convection, Earth and Space Science 34, 1580 (2019).
  * Malkus [1954] M. V. Malkus, The heat transport and spectrum of thermal turbulence, Proc. R. Soc. Lond. A 225, 196 (1954).
  * Kraichnan [1962] R. H. Kraichnan, Turbulent thermal convection at arbitrary prandtl number, Phys. Fluids 5, 1374 (1962).
  * Grossmann and Lohse [2000] S. Grossmann and D. Lohse, Scaling in thermal convection: a unifying theory, J. Fluid Mech. 407, 27 (2000).
  * Grossmann and Lohse [2001] S. Grossmann and D. Lohse, Thermal convection for large Prandtl numbers, Phys. Rev. Lett. 86, 3316 (2001).
  * Chavanne  _et al._ [1997] X. Chavanne, F. Chilla, B. Castaing, B. Hebral, B. Chabraud, and J. Chaussy, Observation of the ultimate regime in Rayleigh-Bénard convection, Phys. Rev. Lett. 79, 3648 (1997).
  * Glazier  _et al._ [1999] J. A. Glazier, T. Segawa, A. Naert, and M. Sano, Evidence against ‘ultrahard’ thermal turbulence at very high Rayleigh numbers, Nature 398, 307 (1999).
  * Niemela  _et al._ [2000] J. J. Niemela, L. Skrbek, K. R. Sreenivasan, and R. J. Donnelly, Turbulent convection at very high Rayleigh numbers, Nature 404, 837 (2000).
  * He  _et al._ [2012] X. He, D. Funfschilling, H. Nobach, E. Bodenschatz, and G. Ahlers, Transition to the ultimate state of turbulent Rayleigh-Bénard convection, Phys. Rev. Lett. 108, 024502 (2012).
  * Zhu  _et al._ [2018] X. Zhu, V. Mathai, R. J. A. M. Stevens, R. Verzicco, and D. Lohse, Transition to the ultimate regime in two-dimensional Rayleigh-Bénard convection, Phys. Rev. Lett. 120, 144502 (2018).
  * Iyer  _et al._ [2020] K. P. Iyer, J. D. Scheel, J. Schumacher, and K. R. Sreenivasan, Classical 1/3 scaling of convection holds up to Ra=1015, Proc. Natl. Acad. Sci. U.S.A. 117, 7594 (2020).
  * Zhu  _et al._ [2017] X. Zhu, R. Stevens, R. Verzicco, and D. Lohse, Roughness-facilitated local 1/2 scaling does not imply the onset of the ultimate regime of thermal convection, Phys. Rev. Lett. 119, 154501 (2017).
  * Zhu  _et al._ [2019] X. Zhu, V. Mathai, R. J. A. M. Stevens, R. Verzicco, and D. Lohse, Nu $\sim$ Ra1/2 scaling enabled by multiscale wall roughness in Rayleigh-Bénard convection, J. Fluid Mech. 869, R4 (2019).
  * Lohse and Toschi [2003] D. Lohse and F. Toschi, The ultimate state of thermal convection, Phys. Rev. Lett. 90, 034503 (2003).
  * Calzavarini  _et al._ [2005] E. Calzavarini, D. Lohse, F. Toschi, and R. Tripiccione, Rayleigh and Prandtl number scaling in the bulk of Rayleigh-Bénard turbulence, Phys. Fluids 17, 055107 (2005).
  * Lepot  _et al._ [2018] S. Lepot, S. Aumaitre, and B. Gallet, Radiative heating achieves the ultimate regime of thermal convection, Proc. Natl Acad. Sci. USA 115, 8973 (2018).
  * Bouillaut  _et al._ [2019] V. Bouillaut, S. Lepot, S. Aumaitre, and B. Gallet, Transition to the ultimate regime in a radiatively driven convection experiment, J. Fluid Mech. 861, R5 (2019).
  * Creyssels [2020] M. Creyssels, Model for classical and ultimate regimes of radiatively driven turbulent convection, J. Fluid Mech. 900, A39 (2020).
  * Wang  _et al._ [2020] B.-F. Wang, Q. Zhou, and C. Sun, Vibration-induced boundary-layer destabilization achieves massive heat-transport enhancement, Sci. Adv. 6, ezza8239 (2020).
  * Ostilla-Mónico  _et al._ [2015] R. Ostilla-Mónico, Y. Yang, E. P. van der Poel, D. Lohse, and R. Verzicco, A multiple resolutions strategy for direct numerical simulation of scalar turbulence., J. Comput. Phys. 301, 308 (2015).
  * Fadlun  _et al._ [2000] E. A. Fadlun, R. Verzicco, P. Orlandi, and J. Mohd-Yusof, Combined immersed-boundary finite-difference methods for three-dimensional complex flow simulations, Journal of Computational Physics 161, 35 (2000).
  * Childs  _et al._ [2012] H. Childs, E. Brugger, B. Whitlock, J. Meredith, S. Ahern, D. Pugmire, K. Biagas, M. Miller, C. Harrison, G. H. Weber, H. Krishnan, T. Fogal, A. Sanderson, C. Garth, E. W. Bethel, D. Camp, O. Rübel, M. Durant, J. M. Favre, and P. Navrátil, VisIt: An End-User Tool For Visualizing and Analyzing Very Large Data, in  _High Performance Visualization–Enabling Extreme-Scale Scientific Insight_ (2012) pp. 357–372.