# Masked Visual Pre-training for Motor Control

Tete Xiao  Ilija Radosavovic  Trevor Darrell  Jitendra Malik

###### Abstract

This paper shows that self-supervised visual pre-training from real-world
images is effective for learning motor control tasks from pixels. We first
train the visual representations by masked modeling of natural images. We then
freeze the visual encoder and train neural network controllers on top with
reinforcement learning. We do not perform any task-specific fine-tuning of the
encoder; the same visual representations are used for all motor control tasks.
To the best of our knowledge, this is the first self-supervised model to
exploit real-world images at scale for motor control. To accelerate progress
in learning from pixels, we contribute a benchmark suite of hand-designed
tasks varying in movements, scenes, and robots. Without relying on labels,
state-estimation, or expert demonstrations, we consistently outperform
supervised encoders by up to 80% absolute success rate, sometimes even
matching the oracle state performance. We also find that in-the-wild images,
e.g., from YouTube or Egocentric videos, lead to better visual representations
for various manipulation tasks than ImageNet images.

Machine Learning, ICML

##  1 Introduction

The last decade of machine learning has been powered by learning
representations with large neural networks and augmenting them with a
relatively small amount of domain knowledge where appropriate. This paradigm
has led to substantial progress across a range of domains. Examples include
visual recognition (Girshick et al., 2014; He et al., 2017), natural language
(Radford et al., 2018; Devlin et al., 2019; Radford et al., 2019; Brown et
al., 2020), and audio (Van Den Oord et al., 2016). And the trend continues.
Motor control, however, remains a notable exception.

In this paper, we show that self-supervised visual pre-training on real-world
images is effective for learning motor control tasks from pixels. These self-
supervised representations consistently outperform supervised representations.

Consider tasks shown in Figure 1 (bottom). The required movement types vary
from simple reaching to object interactions. We also see variations in robots,
scene configurations, and objects. Control inputs are high-dimensional and
difficult to search (e.g., 23 DoF robot with a multi-finger hand). We explore
learning complex tasks such as these from high-dimensional pixel observations.

Figure 1: We explore learning complex motor control from pixels. We show that
we are able to solve a range of motor control tasks with variations in robots,
scenes, and objects. This is enabled by learning rich visual representations
from real-world images with masked modeling. Videos available on the project
page. Figure 2: Masked visual pre-training for motor control. _Left:_ We first
_pre-train_ visual representations using _self-supervision_ through masked
image modeling (He et al., 2021) from _real-world_ images. _Right:_ We then
_freeze_ the image encoder and train task-specific controllers on top with
reinforcement learning (RL). The _same_ visual representations are used for
all motor control tasks.

To tackle this setting, we use the neural network architecture shown in Figure
2b. Our network encodes the input image using a high-capacity visual encoder
(Dosovitskiy et al., 2020) and combines it with proprioceptive information to
obtain an embedding. A light-weight neural network controller takes in the
embedding and predicts actions. The whole system can be trained end-to-end.

Indeed, this design is akin to architectures typically used in approaches that
learn control policies end-to-end with RL, e.g., Levine et al. (2016). While
conceptually appealing, the latter has two main challenges in practice. First,
training is computationally expensive and has poor sample complexity
(especially with high-dimensional inputs and actions). Second, the learned
solutions typically overfit to the setting at hand and thus do not generalize
to new scenes and objects.

One way to offset the high sample complexity of end-to-end RL is to employ
auxiliary objectives (Jaderberg et al., 2017; Oord et al., 2018; Yarats et
al., 2019; Srinivas et al., 2020). For example, Srinivas et al. (2020) show
excellent performance in vision-based RL by using contrastive learning with
data augmentations. However, such representations are still trained using only
environment-specific experience.

The key aspect of our approach is in how we train the visual representations.
We do _not_ train the visual encoder while learning specific motor control
tasks. Instead, we _pre-train_ the visual encoder by _self-supervision_ from
_natural images_ (Figure 1 & 2). We learn the visual representation by
performing masked image modeling through the masked autoencoder (MAE) (He et
al., 2021). Thanks to the Internet and ubiquitous portable cameras, we now
have access to large amounts of unlabeled visual data for self-supervision.
MAE does not require human labels or make strong assumptions about data
distributions, e.g., centered objects or pre-defined augmentation invariances
(Xiao et al., 2021b), making it an excellent framework for learning general
visual representations from large collections of in-the-wild images.

Given the visual encoder, we train controllers on top with reinforcement
learning (Schulman et al., 2017). We keep the visual representations frozen
and do not perform _any_ task-specific fine-tuning of the encoder; all motor
control tasks use the _same_ visual representations. We call our approach MVP
(for Masked Visual Pre-training for Motor Control).

To accelerate future progress, we contribute PixMC, a new benchmark suite of
hand-designed tasks with various movement types, scene configurations, and
robots. We leverage a GPU-based simulator for fast simulation (Makoviychuk et
al., 2021), provide reward functions, baselines, and multi-GPU implementation
of learning algorithms from pixels.

Figure 3: Example reconstructions. For each triplet from left to right: the
masked image, the reconstructed image, the ground-truth target. We observe
that the autoencoder learns color, shape, and object affordance using self-
supervision from in-the-wild images.

We compare our self-supervised approach to baselines that follow the same
architecture (Figure 2b) but use different visual representations. As an upper
bound, we consider oracle hand-engineered states for solving a task (e.g., 3D
poses and direction-to-goal vectors). We also compare our method to visual
encoders trained by supervised learning on ImageNet (Deng et al., 2009), the
choice of encoder in most vision tasks. We summarize our main results as
follows:

\- 1) We show that a _single_ visual encoder pre-trained on real-world images
can solve various motor control tasks _without_ fine-tuning per-task, state
estimation, or demonstrations.

\- 2) Our _self-supervised_ approach consistently outperforms _supervised_
representations (up to 80% absolute success rate), and even matches the oracle
performance in some cases.

\- 3) We find that pre-training on _images in the wild_ , e.g., from YouTube
(Shan et al., 2020) or Egocentric (Damen et al., 2018, 2021) videos, works
better for manipulation tasks than ImageNet (Deng et al., 2009) images.

\- 4) We show that our visual representations _generalize_ in various ways.
For example, our visual encoder disentangles shape and color and is able to
handle a range of different object geometries and configurations.

We encourage researchers to evaluate visual representations not only on
downstream vision tasks but also on motor control tasks. We believe that our
work is a promising step in this direction and release the benchmark suite,
pre-trained models, and the training code on the project page.

##  2 Masked Visual Pre-training for MC

###  2.1 Masked Visual Pre-training

Thanks to the Internet and portable camera devices (e.g., phones, glasses,
etc.), we now have access to large amounts of video data to learn from in
various ways. We leverage such data for learning _visual representations_.
Specifically, we use images from the egocentric Epic Kitchens dataset (Damen
et al., 2018, 2021), the YouTube 100 Days of Hands dataset (Shan et al.,
2020), and the crowd-sourced Something-Something dataset (Goyal et al.,
2017b). Combined, these sources yield a collection of $\scriptstyle\sim$700K
images that we refer to as the Human-Object Interaction dataset (HOI). Note
that we do _not_ exploit any human labels or temporal information even if it
is possible. We also apply our approach using the ImageNet dataset (Deng et
al., 2009) for controlled comparisons with supervised baselines.

With the data in hand, we must now formulate an appropriate self-supervised
task. We adopt masked modeling as our self-supervision objective—specifically,
we use masked autoencoder (MAE) (He et al., 2021). MAEs mask-out random
patches of the input image and reconstruct the missing pixels with a Vision
Transformer (ViT) (Dosovitskiy et al., 2020). During training, only unmasked
patches are fed into the MAE; this strategy makes training more efficient. It
is critical to train MAE with a high masking ratio (e.g., masking 75% of all
patches) and use a heavy encoder with a light decoder. The most appealing
property of MAE is its simplicity and minimal reliance on dataset-specific
augmentation engineering; for example, it works well even with minimal data
augmentations (center crop and color).

In Figure 3 we show example reconstructions for HOI images. We observe that
the model learns about color, shape, and objects. Notice that the images are
representative of everyday interactions making them well suited for our needs.

###  2.2 Learning Motor Control from Pixels

Given the pre-trained visual encoder, we now turn to learning motor control
from pixels. We freeze the visual representations and use them for all
downstream motor control tasks; we do not perform any task-specific fine-
tuning of the image encoder. This design has two main benefits. First, it
prevents the encoder from overfitting to the setting at hand and thus
preserves general visual representations for learning new tasks. Second, it
leads to considerable memory and run time savings since there is no need to
back-propagate through the encoder. Freezing visual encoder makes using large
vision models in the RL loop fast and feasible.

In Figure 2b, we show our architecture for learning motor control from pixels.
We first extract a fixed-sized vector of image features using our pre-trained
visual encoder. Notice that all of the image patches are passed through the
encoder, unlike in the masked pre-training stage. We additionally compile
proprioceptive robot information in the form of joint positions and velocities
into a second vector. This proprioceptive information is readily available on
real robot hardware. We concatenate these two vectors to obtain the input
embedding for the neural network controller.

We then train task-specific motor control policies on top of this embedding
with model-free reinforcement learning. Specifically, we use the proximal
policy optimization (PPO) algorithm (Schulman et al., 2017). PPO is a state-
of-the-art policy gradient method that has shown excellent performance on
complex motor control tasks and successful transfer to real hardware (OpenAI
et al., 2020, 2019). Our policy is a small multi-layer perceptron (MLP)
network. In addition, we train a critic that has the same architecture as the
policy using the same representations. The policy and the critic do not share
weights.

| RLBench | Robosuite | MetaWorld | Ours
---|---|---|---|---
Simulator | Coppelia | MuJoCo | MuJoCo | IsaacGym
Fast |  |  |  | ✓
#Arms | 1 | 8 | 1 | 2
#Hands |  |  |  | ✓
#Tasks | 100 | 9 | 50 | 8
Rewards |  | ✓ | ✓ | ✓

Table 1: Existing benchmarks. Compared to existing benchmarks, ours features a
unique combination of hand-designed tasks, dense rewards, and complex robots
(e.g., multi-finger hands). Crucially, it leverages a fast simulator and
provides distributed training for scaling learning-based motor control from
pixel observations.

##  3 PixMC Benchmark

We construct a new benchmark suite of tasks for studying motor control from
pixels, described in this section.

###  3.1 Motivation

While there exist a number of excellent benchmarks for motor control, e.g.,
DMC (Tassa et al., 2018), RLBench (James et al., 2020), Robosuite (Zhu et al.,
2020), MetaWorld (Yu et al., 2020), they all fall short on one or more of our
requirements. In particular, there is no benchmark suite for learning motor
control algorithms that has high-resolution images, realistic robots, fast
physics simulation, efficient training, and appropriate reward functions and
metrics. To this end, we introduce a new benchmark suite for _Pixel Motor
Control_ , which we call PixMC. We compare the key aspects of PixMC to several
existing benchmarks in Table 1.

###  3.2 Simulator

We leverage the recent NVIDIA IsaacGym simulator (Makoviychuk et al., 2021) to
build our benchmark. The core design idea of IsaacGym is to perform simulation
on a GPU in a shared context. IsaacGym allows for very fast training times.
For example, we are able to train our oracle state-based models in
$\scriptstyle\sim$12 minutes and our image models in $\scriptstyle\sim$5 hours
($\scriptstyle\sim$8 million environment steps) on a single NVIDIA 2080 Ti
GPU. This training speed is considerably faster than it would be in other
simulators commonly used for motor control such as MuJoCo (Todorov et al.,
2012). Rudin et al. (2021) have shown that sim-to-real transfer is feasible
based on policies trained in IsaacGym.

###  3.3 Robots

PixMC includes two robot arms, a parallel jaw gripper, and a multi-finger hand
combined as follows: (1) _Franka:_ A Franka Emika robot commonly used for
research. It has a 7-DoF arm with a 2-DoF gripper. (2) _Kuka with Allegro:_ A
Kuka LBR iiwa arm with 7 DoFs and a 4-finger Allegro hand with 16 DoFs (4 DoFs
per finger), for 23 DoFs in total. For brevity, we refer to it as “Kuka.”

Figure 4: Example tasks. We show example trajectories for the Franka and Kuka
tasks. See the project page for video examples.

###  3.4 Observations and Actions

Our benchmark supports rendering high-resolution pixel observations for each
robot. For both the Franka and Kuka setups and each of the tasks, we use
wrist-mounted cameras by default. The benchmark provides proprioceptive
information for the robots, as well as hand-engineered states typically
including 3D poses or relevant objects, goals, and their relations. All of our
default settings use position control in joint angle space with a control
frequency of 60Hz.

Figure 5: Sample complexity. We plot the success rate as a function of
environment steps on the 8 PixMC tasks. Each task uses either the Franka arm
with a parallel gripper or the Kuka arm with a multi-finger hand. The MVP
approach significantly outperforms the supervised baseline on 7 tasks and
closely matches the oracle state model (considered the upper bound of RL) on 5
tasks at convergence. The result shows that self-supervised pre-training
markedly improves representation quality for motor control tasks.

###  3.5 Tasks, Rewards, and Metrics

PixMC tasks include several movement types from basic reaching to interacting
with objects. The objects in the environment vary in positions, scale, color,
and shape. Figure 1 and 4 show a few example scenes. We hand-design task-
specific dense reward functions for training RL policies. We define reward-
independent success metrics that typically quantify the distance from the
agent or an object to a specified goal location over sufficient time steps.

###  3.6 Distributed Training

The scarcity of GPU memory is a bottleneck for learning motor control from
pixels. For our typical setup with 224$\times$224 images, we can fit at most
256 environments on a single 2080 Ti GPU. We implement PPO with distributed
training to support large batch sizes. Similar to data parallel training, we
create a model replica per-GPU, collect rollouts on each GPU, and synchronize
gradients across GPUs.

##  4 Experimental Setup

#### Data for pre-training.

We consider two kinds of pre-training data: ImageNet (Deng et al., 2009) and a
joint Human-Object Interaction (HOI) dataset. We construct the HOI data by
combining Epic-Kitchens (Damen et al., 2018, 2021), Something-Something (Goyal
et al., 2017b), and 100 Days of Hands (100-DOH) (Shan et al., 2020). To build
HOI, we sample frames from Epic-Kitchens and Something-Something at 1fps and
0.3fps, respectively. This yields 700k images including 100k from 100-DOH.

#### Encoder.

The image encoder follows standard ViT architecture (Dosovitskiy et al.,
2020). ViT partitions an image into patches and linearly projects each patch
into features, followed by standard Transformer blocks (Vaswani et al., 2017;
Wang et al., 2019). We use the ViT-Small model with a 16$\times$16 patch size,
384 hidden size, 6 attention heads, an MLP multiplier of 4, and 12 blocks. The
model runs at 4.6 gigaflops for input images of 224$\times$224, approximately
1.2$\times$ as many as the ResNet-50 (He et al., 2016) model.

We pre-train supervised and self-supervised variants of the ViT model. For the
supervised model, we use the recipe in (Xiao et al., 2021a) and train on the
ImageNet dataset for 400 epochs. We use the MAE framework for the self-
supervised counterpart (He et al., 2021). We use an auxiliary dummy
classification token in the MAE for downstream finetuning and transfer (He et
al., 2021). We use a crop ratio of [0.2, 1.0] for ImageNet and [0.1, 0.75] for
HOI, due to the larger width-over-height aspect ratio of HOI images. We train
the MAE models for 1600 epochs on 16 GPUs for both HOI and ImageNet datasets.

#### Controller.

The controller is a simple MLP with each hidden layer followed by a SeLU
(Klambauer et al., 2017) activation function. We use a four-layer MLP with
hidden layers of size [256, 128, 64] for all tasks, following (Makoviychuk et
al., 2021). The (dummy) classification token of the ViT encoder yields the
image features and a linear layer projects the features to 128 dimensions. The
controller takes in the linearly-projected (128-d) proprioceptive state of the
robot along with the projected image features. The controller outputs delta
joint angles.

#### Training with RL.

We freeze the visual encoder throughout the entire training horizon. We train
for 500 iterations for reach, 1000 iterations for cabinet, and 2000 iterations
for pick and relocate, respectively. In each iteration, we collect samples
from 256 environments which have 32 steps each. We train for 10 epochs on
these collected samples per iteration. We compose 4 minibatches per epoch,
i.e., 4 gradient updates, leading to a minibatch size of 2048 per gradient
update. We choose this configuration because it maximizes the memory on a
single NVIDIA 2080 Ti GPU. In all experiments we train with a cosine learning
rate decay schedule (Goyal et al., 2017a). To reduce randomness in the RL
experiments (Agarwal et al., 2021), for each task and model we search for the
best learning rate in {0.0005, 0.001, 0.0015} with 5 seeds per learning rate
(15 runs for each task and model). We always report the performance yielded by
the best learning rate aggregated over seeds unless otherwise specified. Other
hyperparams use defaults: Adam optimizer with $\beta_{1}=0.9$ and
$\beta_{2}=0.999$, gradient norm of 1, initial noise standard deviation of
1.0.

##  5 Experimental Results

###  5.1 Sample Complexity

Figure 5 shows success rates over training on 8 challenging tasks from PixMC.
We consider the oracle state model (i.e., position, orientation, and velocity
of the object, goal and robot in world-coordinate system, which is difficult
to estimate in real-world settings) as the upper bound of RL. MVP
significantly outperforms the supervised baseline on 7 out of 8 tasks and
matches the baseline on the 8th task. At convergence, MVP closely matches the
oracle state model on 5 tasks. The supervised baseline is flat at zero success
rate on the pick and move tasks with both robots; MVP rivals the oracle on the
pick task and achieves high success rate on the relocate task. These results
show that self-supervised pre-training markedly improves representation
quality for motor control tasks.

Figure 6: Robustness to distractors. The robots are trained to pick up a blue
box of 4.5cm side length. At _test time_ , we add a distractor object
differing from the training object in terms of color (blue vs. green), shape
(cube vs. sphere), or size (4.5cm vs. 6cm), shown at the top. MVP maintains
high success rates for color and shape. The model is less sensitive to size
likely due to the scale ambiguity from single first-person camera setup.

###  5.2 Generalization

We design various experiments on the pick task with both the Franka and Kuka
arms to demonstrate the degree to which MVP is able to generalize.

Figure 7: Generalization to objects of various geometries. We import three
additional objects (i.e., can, mug, and banana) from the YCB dataset and re-
train the controller to pick up the individual object category. The Kuka robot
with the Allegro hand can pick up all objects with $\scriptstyle\sim$50%
success rate.

#### Robustness to distractors.

In the pick task the robots are trained to pick up a blue box of 4.5cm side
length. In this experiment we add a distractor object that differs from the
training object in terms of color, shape, or scale, _at testing time_. The
models are not retrained for new testing configurations. A robust model from
pixels should pick up the object used for training. Specifically, we have 1) a
_green_ box of 4.5cm side length (color distractor); 2) a blue _sphere_ of
4.5cm diameter (shape distractor); and 3) a blue box of _6cm side length_
(scale distractor). Figure 6 shows the results of our MVP model. Color and
shape distractors only marginally decrease the success rate, implying that MVP
is able to recognize the color and shape of objects. The model, however, is
less sensitive to scale variation as the 50% success rate suggests that the
distractor or the original box is picked up by chance. We believe it is due to
scale ambiguity from single first-person camera setup. Note that the oracle
state model would not be able to function in these testing cases due to
changes in state dimensions.

#### Generality of the framework.

We import various objects from the YCB dataset (Calli et al., 2015)—box, can,
mug, and banana—for the pick task and re-train the model for each individual
object. Figure 7 visualizes the experiment setup and shows the results. The
Kuka robot with the Allegro hand can pick up all of the objects with at least
a 50% success rate. This shows MVP as a framework can generalize to objects of
different geometries and the multi-finger hand’s strength in object
manipulation.

Figure 8: Pre-training data: HOI vs. ImageNet. We compare the performance of
our approach when using HOI and ImageNet images as the source of pre-training
data. Overall, we find that the representations learned from HOI data perform
better on motor control tasks.

###  5.3 Ablations

#### Pre-training data.

We train MVP on HOI and ImageNet data, respectively. Figure 8 shows the
results. MVP trained on HOI data outperforms the counterpart trained on
ImageNet data on 7 out of 8 tasks. Whereas ImageNet is dominated by images of
animals and objects, HOI contains many images demonstrating object
manipulation from a first-person camera view. We hypothesize that this
difference is why HOI is empirically the superior choice for motor control
tasks.

#### Random features.

We compare our MVP with a randomly initialized and frozen image encoder. The
random model yields flat-zero or close-to-zero success rates on 6 out of 8
tasks. This indicates that random features are insufficient to solve complex
motor control tasks. We show the results of one successful and one failed task
in Figure 9.

#### Stability.

We characterize how sensitive different models are to changes in learning rate
and random seed. Representations of high quality should be less sensitive and
yield consistently-high performance. Figure 10 shows the reach task results
over 3 learning rates (0.0005, 0.001, 0.0015) and 5 seeds for a total of 15
runs for each model. Although the supervised baseline with the best choice of
learning rate performs close to MVP in Franka reach and has over 40% success
rate in Kuka reach (see Figure 5), it exhibits much worse stability across a
range of learning rates. Our MVP, in contrast, shows good stability in the
learning rate hyperparameter, a sign of superior optimizability in training.

Figure 9: Random features. We compare our learnt representations to a random
features baseline. We use the same visual encoder, initialize it randomly, and
freeze. The random model fails on 6 out of 8 PixMC tasks (0 success rate).
Here we show one task that fails (FrankaPick) and one that succeeds
(KukaPick). Figure 10: Learning rate and seed stability. For each model, we
train 15 instances of the model with 3 learning rates and 5 seeds. We show the
results on tasks where the supervised baseline achieves nontrivial
performance. MVP significantly improves stability over the supervised model.
Figure 11: Larger encoders. We pre-train a ViT-Base model (18Gflops) and use
the representations for the pick task. We do not observe clear gains from
preliminary model scaling and believe that scaling data and model size is an
exciting area for future work. Figure 12: CURL and RAD comparisons. We compare
MVP to environment training with our reimplementation of CURL and RAD (denoted
with star; see text for details). We observe that, for a fixed number of
steps, MVP outperforms both while being less computationally expensive to
train (1 vs. 8 GPU training).

#### Unfreezing encoders.

We experiment with training the encoder end-to-end with RL in downstream motor
control tasks. Unfreezing the encoder significantly increases GPU compute and
memory consumption. To maintain the number of environments, we increase the
number of GPUs used from 1 to 8 via our distributed training pipeline. We
choose the Franka pick task as the testbed and run 3 runs with different seeds
per model. We test with two initialization: (1) initializing the visual
encoder randomly, or (2) initializing with the pre-trained MAE weights. We
observed unstable training (the loss goes to NaN), and we decreased the
learning rate until training successfully completed. Still, both models
yielded flat zero success rate on the task at all seeds. This result is
somewhat counter-intuitive as one would expect end-to-end training should
yield better results than training on frozen representations. We conjecture
that it may be due to (a) the RL signal being unstable and hard to tune; (b)
noisy gradients from the RL objective interfering with pre-trained visual
representation; or (c) a high capacity vision model like ViT requiring
significantly more samples and environments to train end-to-end with RL.
Freezing the visual encoder as in MVP preserves the quality of visual
representations while yielding faster RL training.

#### Larger encoders.

We pre-train a ViT-Base encoder (18 gigaflops) and conduct a preliminary
transfer study on the Franka/Kuka pick tasks. Figure 11 shows the results. We
observe that the larger encoder does not improve performance. A larger encoder
potentially requires more data and/or a different training recipe. Overall,
scaling data and models in the context of self-supervised representations for
motor control remains an exciting area for future work.

###  5.4 Additional Comparisons

#### CURL and RAD comparisons.

We compare MVP to two state-of-the-art methods that train the vision encoder
with environment data: CURL (Srinivas et al., 2020) and RAD (Laskin et al.,
2020). Due to differences in their original settings and ours, e.g., small
ConvNet vs. large ViT, we compare to our reimplementations (denoted with
star). In particular, we adopt PPO, ViT-Small visual encoder, and MoCo-v3
(Chen et al., 2021b) data augmentation recipe. We observe that, for a fixed
number of steps, MVP outperforms both baselines while being less
computationally expensive (1 vs. 8 GPUs). We note that environment training
with enough data might yield better performance than our image pre-training.
Indeed, we do not see our approach as the ultimate answer for any one setting
but rather as a solid baseline, or a starting point, for many varying
settings.

#### CLIP comparisons.

Next, we experiment with substituting a pre-trained CLIP visual encoder
(Radford et al., 2021) in place of our MAE encoders. The CLIP encoder is
trained using 400M labeled text-image pairs and has shown excellent
performance across a wide range of visual tasks. We opt to use the ViT-Base
CLIP encoder as it is closest in size to our ViT-Small encoders. We note that
this comparison is imperfect due to difference in data distributions and
encoders but believe it is still instructive to see. In Figure 13 we show the
results. We observe a promising signal that self-supervised representations
can outperform strong CLIP encoders. We believe it would be interesting to
perform a controlled study on in-the-wild images with text annotations, like
from the recently released Ego4D dataset (Grauman et al., 2021).

#### MoCo-v3 comparisons.

Finally, we compare visual encoders trained with the Momentum Contrastive
(MoCo) self-supervised learning framework instead of MAE used in MVP. We opt
to use the latest MoCo-v3 (Chen et al., 2021b) designed for ViT models. We
show results in Figure 14. We observe that MoCo-v3 can achieve good
performance on one of the tasks and non-trivial on the other. This suggests
that our approach may be more general and applicable to other self-supervised
pre-training techniques as well. However, in contrast to MAE, it may be harder
to adapt techniques like MoCo-v3 to in-the-wild images (see Figure 8).

Figure 13: CLIP comparisons. We compare our visual representations to the CLIP
visual encoder trained with large scale language supervision. The results show
a promising signal that self-supervised representations can outperform CLIP
encoders. Figure 14: MoCo-v3 comparisons. We compare MAE, used in MVP, to the
MoCo-v3 alternative for self-supervised pre-training _on ImageNet_. We see
that MoCo-v3 can achieve non-trivial performance, showing the generality of
our approach. However, in contrast to MAE, it may be harder to adapt to in-
the-wild images.

##  6 Related Work

#### Dexterous manipulation.

Recently, OpenAI (OpenAI et al., 2020, 2019) has shown impressive results in
dexterous in-hand manipulation with large scale domain randomization. A number
of works consider related manipulation problems with multi-finger hands but
rely on explicit state estimation (Handa et al., 2020; Huang et al., 2021),
expert policies (Chen et al., 2021a), human demonstrations (Rajeswaran et al.,
2018; Radosavovic et al., 2021; Qin et al., 2021), human priors (Mandikal &
Grauman, 2021), or models (Nagabandi et al., 2019). In contrast, we do not use
on any of the aforementioned components in our approach.

#### Representations in RL.

One way to learn representations for motor control is to rely on the task
signal. This is commonly done in end-to-end RL (Mnih et al., 2015; Levine et
al., 2016; Kalashnikov et al., 2018). However, it results in high sample
complexity, particularly in the case of high-dimensional observations like
images. Furthermore, such representations may get overly adapted to the
problem at hand and not generalize to new settings (e.g., new objects).

#### RL with self-supervision.

One way to overcome the high sample complexity of RL is to employ auxiliary
objectives. In particular, in addition to learning the task, learn to predict
some property of the environment, e.g., depth, that may lead to learning good
representations as a side effect (Mirowski et al., 2017; Jaderberg et al.,
2017; Shelhamer et al., 2017; Lample & Chaplot, 2017). Rather than predicting
hand-designed environment properties, researchers explored using more general
self-supervised objectives (Oord et al., 2018; Yarats et al., 2019; Srinivas
et al., 2020). For example, Srinivas et al. (2020) show excellent performance
in vision-based RL tasks. Representations can also be pre-trained on the data
from the environment (Ha & Schmidhuber, 2018; Srinivas et al., 2020). Overall,
we share the goal of learning good visual representations with self-
supervision. In contrast, we learn representations from large collections of
natural images rather than environment-specific experience.

#### Self-supervision in robotics.

Self-supervised learning has also been used in various robotic settings.
(Pinto & Gupta, 2016) and Agrawal et al. (2016) learn representations through
interaction. Sermanet et al. (2018) learn representations from multiview video
using contrastive learning and use them for imitation learning. Florence et
al. (2018) learn dense image descriptors with self-supervision. Zhan et al.
(2020) learn robotic manipulation with RAD (Laskin et al., 2020) and CURL
(Srinivas et al., 2020). Pari et al. (2021) show the effectiveness of visual
representations with non-parametric nearest neighbor controllers. All of these
approaches learn representations in a specific robotic setting of interest
(e.g., videos in the lab), rather than general visual representations from
image collections like ours.

#### Supervised pre-training.

Sax et al. (2018) and Chen et al. (2020a) show that representations learned
from performing a set of mid-level vision tasks using label supervision
benefits downstream navigation and manipulation tasks, respectively. Zhou et
al. (2019) show the effectiveness on visual representations in driving
settings. Yen-Chen et al. (2020) transfer image models trained on supervised
vision tasks, e.g., edge detection and semantic segmentation, to affordance
prediction models for object manipulation. Shah & Kumar (2021) use ImageNet
representations for dexterous manipulation. In contrast to all of these, our
approach is self-supervised and does not rely on labeled datasets.

#### Self-supervised pre-training in computer vision.

Self-supervised learning has been gaining momentum in computer vision. The
approaches often rely on pretext tasks for pre-training (Doersch et al., 2015;
Wang & Gupta, 2015; Noroozi & Favaro, 2016; Zhang et al., 2016; Pathak et al.,
2016; Komodakis & Gidaris, 2018). More recently, contrastive learning methods,
e.g., (Hadsell et al., 2006; Oord et al., 2018; Wu et al., 2018; Henaff, 2020;
He et al., 2020; Chen et al., 2020c; Jabri et al., 2020), have been popular.
These techniques try to learn to be invariant to a set of hand-crafted
augmentations. Xiao et al. (2021b) have shown that the augmentations introduce
inductive bias and may harm downstream transfer. Masked image autoencoding
(Chen et al., 2020b; Bao et al., 2022; He et al., 2021) pursues a different
direction by learning to recover masked pixels. Specifically, we adopt the
Masked Autoencoders (MAE) (He et al., 2021) have shown excellent performance
on recognition tasks. We adopt the MAE as our visual pre-training strategy for
learning motor control.

##  7 Conclusion

In this paper, we show that self-supervised visual pre-training is effective
for motor control. We use a single vision encoder to learn various motor
control tasks from pixels, without per-task fine-tuning, explicit state
estimation, or expert demonstrations. We further show large sample complexity
improvements compared to supervised baselines (up to 80% absolute success
rate) and sometimes even match the oracle state performance. Finally, we show
that in-the-wild images, e.g., from YouTube or Egocentric videos, can lead to
better visual representations than ImageNet images.

## Acknowledgements

We thank William Peebles, Matthew Tancik, Anastasios Angelopoulos, Aravind
Srinivas, and Agrim Gupta for helpful discussions. This work was supported in
part by DOD including DARPA’s MCS, XAI, LwLL, and/or SemaFor programs; ONR
MURI program (N00014-14-1-0671), as well as BAIR’s industrial alliance
programs.

## References

  * Agarwal et al. (2021) Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M.  Deep reinforcement learning at the edge of the statistical precipice.  _NeurIPS_ , 2021.
  * Agrawal et al. (2016) Agrawal, P., Nair, A. V., Abbeel, P., Malik, J., and Levine, S.  Learning to poke by poking: Experiential learning of intuitive physics.  _NeurIPS_ , 2016.
  * Bao et al. (2022) Bao, H., Dong, L., and Wei, F.  Beit: Bert pre-training of image transformers.  In _ICLR_ , 2022.
  * Brown et al. (2020) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _NeurIPS_ , 2020.
  * Calli et al. (2015) Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., and Dollar, A. M.  The ycb object and model set: Towards common benchmarks for manipulation research.  In _ICAR_ , 2015.
  * Chen et al. (2020a) Chen, B., Sax, A., Lewis, G., Armeni, I., Savarese, S., Zamir, A., Malik, J., and Pinto, L.  Robust policies via mid-level visual representations: An experimental study in manipulation and navigation.  In _CoRL_ , 2020a.
  * Chen et al. (2020b) Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I.  Generative pretraining from pixels.  In _ICML_ , 2020b.
  * Chen et al. (2020c) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.  A simple framework for contrastive learning of visual representations.  In _ICML_ , 2020c.
  * Chen et al. (2021a) Chen, T., Xu, J., and Agrawal, P.  A system for general in-hand object re-orientation.  In _CoRL_ , 2021a.
  * Chen et al. (2021b) Chen, X., Xie, S., and He, K.  An empirical study of training self-supervised vision transformers.  In _ICCV_ , 2021b.
  * Damen et al. (2018) Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al.  Scaling egocentric vision: The epic-kitchens dataset.  In _ECCV_ , 2018.
  * Damen et al. (2021) Damen, D., Doughty, H., Farinella, G. M., , Furnari, A., Ma, J., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., and Wray, M.  Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.  _IJCV_ , 2021.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HCT_ , 2019.
  * Doersch et al. (2015) Doersch, C., Gupta, A., and Efros, A. A.  Unsupervised visual representation learning by context prediction.  In _ICCV_ , 2015.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2020.
  * Florence et al. (2018) Florence, P. R., Manuelli, L., and Tedrake, R.  Dense object nets: Learning dense visual object descriptors by and for robotic manipulation.  In _CoRL_ , 2018.
  * Girshick et al. (2014) Girshick, R., Donahue, J., Darrell, T., and Malik, J.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In _CVPR_ , 2014.
  * Goyal et al. (2017a) Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  _arXiv:1706.02677_ , 2017a.
  * Goyal et al. (2017b) Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al.  The” something something” video database for learning and evaluating visual common sense.  In _ICCV_ , 2017b.
  * Grauman et al. (2021) Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  _arXiv:2110.07058_ , 2021.
  * Ha & Schmidhuber (2018) Ha, D. and Schmidhuber, J.  World models.  _arXiv:1803.10122_ , 2018.
  * Hadsell et al. (2006) Hadsell, R., Chopra, S., and LeCun, Y.  Dimensionality reduction by learning an invariant mapping.  In _CVPR_ , 2006.
  * Handa et al. (2020) Handa, A., Van Wyk, K., Yang, W., Liang, J., Chao, Y.-W., Wan, Q., Birchfield, S., Ratliff, N., and Fox, D.  Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system.  In _ICRA_ , 2020.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * He et al. (2017) He, K., Gkioxari, G., Dollár, P., and Girshick, R.  Mask r-cnn.  In _ICCV_ , 2017.
  * He et al. (2020) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R.  Momentum contrast for unsupervised visual representation learning.  In _CVPR_ , 2020.
  * He et al. (2021) He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.  Masked autoencoders are scalable vision learners.  _arXiv:2111.06377_ , 2021.
  * Henaff (2020) Henaff, O.  Data-efficient image recognition with contrastive predictive coding.  In _ICML_ , 2020.
  * Huang et al. (2021) Huang, W., Mordatch, I., Abbeel, P., and Pathak, D.  Generalization in dexterous manipulation via geometry-aware multi-task learning.  _arXiv:2111.03062_ , 2021.
  * Jabri et al. (2020) Jabri, A., Owens, A., and Efros, A.  Space-time correspondence as a contrastive random walk.  _NeurIPS_ , 2020.
  * Jaderberg et al. (2017) Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K.  Reinforcement learning with unsupervised auxiliary tasks.  In _ICLR_ , 2017.
  * James et al. (2020) James, S., Ma, Z., Arrojo, D. R., and Davison, A. J.  Rlbench: The robot learning benchmark & learning environment.  _RA-L_ , 2020.
  * Kalashnikov et al. (2018) Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al.  Scalable deep reinforcement learning for vision-based robotic manipulation.  In _CoRL_ , 2018.
  * Klambauer et al. (2017) Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S.  Self-normalizing neural networks.  In _NeurIPS_ , 2017.
  * Komodakis & Gidaris (2018) Komodakis, N. and Gidaris, S.  Unsupervised representation learning by predicting image rotations.  In _ICLR_ , 2018.
  * Lample & Chaplot (2017) Lample, G. and Chaplot, D. S.  Playing fps games with deep reinforcement learning.  In _AAAI_ , 2017.
  * Laskin et al. (2020) Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A.  Reinforcement learning with augmented data.  _NeurIPS_ , 2020.
  * Levine et al. (2016) Levine, S., Finn, C., Darrell, T., and Abbeel, P.  End-to-end training of deep visuomotor policies.  _JMLR_ , 2016.
  * Makoviychuk et al. (2021) Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., et al.  Isaac gym: High performance gpu-based physics simulation for robot learning.  In _NeurIPS_ , 2021.
  * Mandikal & Grauman (2021) Mandikal, P. and Grauman, K.  Dexvip: Learning dexterous grasping with human hand pose priors from video.  In _CoRL_ , 2021.
  * Mirowski et al. (2017) Mirowski, P., Pascanu, R., Viola, F., Soyer, H., Ballard, A. J., Banino, A., Denil, M., Goroshin, R., Sifre, L., Kavukcuoglu, K., et al.  Learning to navigate in complex environments.  In _ICLR_ , 2017.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al.  Human-level control through deep reinforcement learning.  _Nature_ , 2015.
  * Nagabandi et al. (2019) Nagabandi, A., Konolige, K., Levine, S., and Kumar, V.  Deep dynamics models for learning dexterous manipulation.  In _CoRL_ , 2019.
  * Noroozi & Favaro (2016) Noroozi, M. and Favaro, P.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In _ECCV_ , 2016.
  * Oord et al. (2018) Oord, A. v. d., Li, Y., and Vinyals, O.  Representation learning with contrastive predictive coding.  _arXiv:1807.03748_ , 2018.
  * OpenAI et al. (2019) OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., and Zhang, L.  Solving rubik’s cube with a robot hand.  _arXiv:1910.07113_ , 2019.
  * OpenAI et al. (2020) OpenAI, Andrychowicz, M., Baker, B., Chociej, M., Józefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., Schneider, J., Sidor, S., Tobin, J., Welinder, P., Weng, L., and Zaremba, W.  Learning dexterous in-hand manipulation.  _The International Journal of Robotics Research_ , 2020.
  * Pari et al. (2021) Pari, J., Muhammad, N., Arunachalam, S. P., Pinto, L., et al.  The surprising effectiveness of representation learning for visual imitation.  _arXiv:2112.01511_ , 2021.
  * Pathak et al. (2016) Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A. A.  Context encoders: Feature learning by inpainting.  In _CVPR_ , 2016.
  * Pinto & Gupta (2016) Pinto, L. and Gupta, A.  Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours.  In _ICRA_ , 2016.
  * Qin et al. (2021) Qin, Y., Wu, Y.-H., Liu, S., Jiang, H., Yang, R., Fu, Y., and Wang, X.  Dexmv: Imitation learning for dexterous manipulation from human videos.  _arXiv:2108.05877_ , 2021.
  * Radford et al. (2018) Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.  Language models are unsupervised multitask learners.  2019\.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Radosavovic et al. (2021) Radosavovic, I., Wang, X., Pinto, L., and Malik, J.  State-only imitation learning for dexterous manipulation.  In _IROS_ , 2021.
  * Rajeswaran et al. (2018) Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  _RSS_ , 2018.
  * Rudin et al. (2021) Rudin, N., Hoeller, D., Reist, P., and Hutter, M.  Learning to walk in minutes using massively parallel deep reinforcement learning.  In _CoRL_ , 2021.
  * Sax et al. (2018) Sax, A., Emi, B., Zamir, A. R., Guibas, L., Savarese, S., and Malik, J.  Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies.  _arXiv:1812.11971_ , 2018.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv:1707.06347_ , 2017.
  * Sermanet et al. (2018) Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., and Brain, G.  Time-contrastive networks: Self-supervised learning from video.  In _ICRA_ , 2018.
  * Shah & Kumar (2021) Shah, R. and Kumar, V.  Rrl: Resnet as representation for reinforcement learning.  _arXiv:2107.03380_ , 2021.
  * Shan et al. (2020) Shan, D., Geng, J., Shu, M., and Fouhey, D. F.  Understanding human hands in contact at internet scale.  In _CVPR_ , 2020.
  * Shelhamer et al. (2017) Shelhamer, E., Mahmoudieh, P., Argus, M., and Darrell, T.  Loss is its own reward: Self-supervision for reinforcement learning.  In _ICLR_ , 2017.
  * Srinivas et al. (2020) Srinivas, A., Laskin, M., and Abbeel, P.  Curl: Contrastive unsupervised representations for reinforcement learning.  In _ICML_ , 2020.
  * Tassa et al. (2018) Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al.  Deepmind control suite.  _arXiv:1801.00690_ , 2018.
  * Todorov et al. (2012) Todorov, E., Erez, T., and Tassa, Y.  Mujoco: A physics engine for model-based control.  In _IROS_ , 2012.
  * Van Den Oord et al. (2016) Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A. W., and Kavukcuoglu, K.  Wavenet: A generative model for raw audio.  _SSW_ , 2016.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Wang et al. (2019) Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., and Chao, L. S.  Learning deep transformer models for machine translation.  In _ACL_ , 2019.
  * Wang & Gupta (2015) Wang, X. and Gupta, A.  Unsupervised learning of visual representations using videos.  In _ICCV_ , 2015.
  * Wu et al. (2018) Wu, Z., Xiong, Y., Yu, S. X., and Lin, D.  Unsupervised feature learning via non-parametric instance discrimination.  In _CVPR_ , 2018.
  * Xiao et al. (2021a) Xiao, T., Dollar, P., Singh, M., Mintun, E., Darrell, T., and Girshick, R.  Early convolutions help transformers see better.  In _NeurIPS_ , 2021a.
  * Xiao et al. (2021b) Xiao, T., Wang, X., Efros, A. A., and Darrell, T.  What should not be contrastive in contrastive learning.  In _ICLR_ , 2021b.
  * Yarats et al. (2019) Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., and Fergus, R.  Improving sample efficiency in model-free reinforcement learning from images.  _arXiv:1910.01741_ , 2019.
  * Yen-Chen et al. (2020) Yen-Chen, L., Zeng, A., Song, S., Isola, P., and Lin, T.-Y.  Learning to see before learning to act: Visual pre-training for manipulation.  In _ICRA_ , 2020.
  * Yu et al. (2020) Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In _CoRL_ , 2020.
  * Zhan et al. (2020) Zhan, A., Zhao, P., Pinto, L., Abbeel, P., and Laskin, M.  A framework for efficient robotic manipulation.  _arXiv:2012.07975_ , 2020.
  * Zhang et al. (2016) Zhang, R., Isola, P., and Efros, A. A.  Colorful image colorization.  In _ECCV_ , 2016.
  * Zhou et al. (2019) Zhou, B., Krähenbühl, P., and Koltun, V.  Does computer vision matter for action?  _Science Robotics_ , 2019.
  * Zhu et al. (2020) Zhu, Y., Wong, J., Mandlekar, A., and Martín-Martín, R.  robosuite: A modular simulation framework and benchmark for robot learning.  _arXiv:2009.12293_ , 2020.


electron-electron interactions and present multiple cuts of the global phase
diagram at zero and finite temperature (see Figs. 3 and 16) and finite
chemical doping (see Figs. 4, 5 and 6).

Using the renormalization group analysis, we show that in an isotropic system
an $s$-wave pairing and two nematic orders are the prominent candidates for a
broken symmetry phase at zero temperature and chemical doping. While the
$s$-wave pairing only breaks the global $U(1)$ symmetry and produces a uniform
mass gap, two nematic orders, transforming under the $T_{2g}$ and $E_{g}$
representations, produce lattice distortion along the $C_{3v}$ and $C_{4v}$
axes, respectively. Such ordered phases can describe either time-reversal
symmetry preserving insulators or topological Dirac semimetals. However, a
collection of strongly correlated gapless spin-3/2 fermions do not show any
noticable propensity toward the nucleation of any magnetic order or $d$-wave
pairings in an isotropic system at least when $t=0$. This is so, because the
magnetic orders ($d$-wave pairings) produce gapless Weyl nodes (nodal loops)
around which the density of states vanishes as $\varrho(E)\sim|E|^{2}(|E|)$,
while the former three orders support gapped spectra. Hence, the magnetic
orders and $d$-wave pairings are energetically inferior to $s$-wave pairing
and electronic nematicities. However, with increasing temperature, one finds a
smooth crossover from nematic to magnetic phases, as shown in Fig. 3. Hence,
_energetically superior orders are found at low temperatures, whereas at
higher temperature broken symmetry phases possess larger entropy_. This
energy-entropy competition is discussed in Sec. IV.4 and summarized in Fig. 2.

We identify that the mass anisotropy ($\alpha$), measuring the quadrupolar
distortion in the Luttinger system, can be a useful non-thermal tuning
parameter to further explore the territory of strongly interacting spin-3/2
fermions goswami-roy-dassarma ; Herbut-2 . In particular, we find that strong
quadrupolar distortions can be conducive for various magnetic orderings even
at zero temperature. Specifically, when the electronic dispersion along the
$C_{3v}$ axes becomes almost flat (realized as $\alpha\to\frac{\pi}{2}$) the
singlet $A_{2u}$ magnetic order stabilizes at $t=0$, see Figs. 6 and 16. On
the other hand, when Luttinger fermions are almost non-dispersive along the
$C_{4v}$ axes (realized when $\alpha\to 0$), the system becomes susceptible
toward the formation of a triplet $T_{1u}$ magnetic order, see Figs. 7 and 16.
For these two limiting scenarios the above two magnetic orders become (almost)
_mass_ [see Secs. V.1.2 and V.1.3], and their nucleation becomes energetically
beneficiary even at zero temperature.

Irrespective of these details, we realize that all quantum phase transitions
from the Luttinger semimetal to symmetry-breaking phases are continuous and
controlled by various critical critical points, see Sec. VI.1 and Table 2. To
the leading order in the $\epsilon$-expansion, the universality class of these
transitions is characterized by the (a) correlation length exponent
$\nu^{-1}=\epsilon$ and (b) dynamic scaling exponent $z=2$ [see Sec. VI.2].
The presence of such quantum critical points manifests itself through the
scaling of the transition temperature $t_{c}\sim|\delta|^{\nu z}$, yielding
$t_{c}\sim|\delta|^{2}$ for $d=3$ or $\epsilon=1$, see Fig. 17, where $\delta$
is the reduced distance from the critical point.

Finally, we introduce (chemical) doping as another non-thermal tuning
parameter to map out the global phase diagram of an interacting Luttinger
metal, see Sec. VI.4. Since any paired state maximally gaps the Fermi surface,
its appearance at the lowest temperature is quite natural, at least when
$|\mu|>0$. By contrast, excitonic phases (insulators or semimetals) become
metallike (possessing a finite density of states) at finite chemical doping,
according to the Luttinger theorem Luttinger_Theorem . Therefore, particle-
hole orders are accompanied by higher entropy due to the presence of a Fermi
surface (with a constant density of states). To demonstrate the energy-entropy
competition in a metallic system, we choose the mass anisotropy parameter
$\alpha$ such that at $\mu=0$ the system only supports excitonic orders. Upon
raising (lowering) the chemical potential to the conduction (valence) band, we
observe that a superconducting order develops at low temperature and the
excitonic order gets pushed toward higher temperature and stronger
interactions, see Figs. 5 and 6. Therefore, the overall structure of the
global phase diagram is compatible with the energy-entropy competition,
dictated by the emergent band topology of competing broken symmetry phases.

Furthermore, we also identify a definite “selection rule” among competing
phases [see Sec. VI.5]. From multiple cuts of the global phase diagram of a
collection of strongly interacting spin-3/2 fermions, we find that two phases
can reside in close vicinity of each other if the order-parameters describing
two distinct phases can be combined to form a composite order-parameter. In
other words, two ordered phases, respectively described by $O(M_{1})$ and
$O(M_{2})$ symmetric order-parameters [see footnote 13], can reside next to
each other only if one can construct an $O(M)$ symmetric composite-vector,
where $M_{1},M_{2}\leq M\leq M_{1}+M_{2}$, from the elements of two individual
order parameters. This is so because when an interaction favors $O(M_{1})$
symmetry breaking order, it also enhances the _scaling dimension_ of an
$O(M_{2})$ symmetry breaking order and vice-versa, when the above selection
rule is satisfied. Therefore, by tuning a suitable parameter (such as
temperature, mass anisotropy parameter, chemical potential) one can induce a
transition between two competing phases. As an immediate outcome of this
selection rule we realize that while repulsive interactions (short-range) in
the nematic channels are conducive to $s$-wave pairing [see Figs. 3, 3, 4, 6,
6], magnetic interactions favor nucleation of $d$-wave pairings [see Figs. 5,
6, 6] among Luttinger fermions.

A few specific cuts of the global phase diagram corroborate with ones
extracted experimentally in Ln2Ir2O7 takagi  and half-Heusler compounds
Exp:Paglione-1 . For exmaple, a finite temperature phase transition from
$A_{2u}$ or all-in all-out magnetic order to a Luttinger semimetal has been
observed in majority of 227 pyrochlore iridates (except for Ln=Pr) takagi ,
and Fig. 3 qualitatively captures this phenomena (for strong enough
$g_{{}_{3}}$). By contrast, Pr2Ir2O7 supports a large anomalous Hall effect
below $1.5$K AHE-1 ; AHE-2 ; AHE-3 . Note that a triplet $T_{1u}$ or 3-in
1-out magnetic order supports anomalous Hall effect due to the presence of
Weyl nodes in the ordered state (see Sec. IV.3 and Ref. goswami-roy-dassarma )
and the phase diagram from Fig. 3 shows the appearance of $T_{1u}$ magnetic
order at finite temperature for strong enough interaction ($g_{{}_{4}}$). It
is worth recalling that ARPES measurements strongly suggests that isotropic
Luttinger semimetal describes the normal states of both Nd2Ir2O7 and Pr2Ir2O7
Exp:Nakatsuji-2 ; Exp:Nakatsuji-1 . On the other hand, half-Heusler compounds
LnPdBi display a confluence of magnetic order and superconductivity
Exp:Paglione-1  and the phase diagrams shown in Figs. 5, 6 and 6 capture such
competition (at least qualitative). This connection can be further
substantiated from a recent penetration depth ($\Delta\lambda$) measurement in
YPtBi Exp:Paglione-2 , suggesting $\Delta\lambda\sim T/T_{c}$ (roughly) at low
enough temperature, where $T_{c}=0.78$K in the superconducting transition
temperature. Such a T-linear dependence of the penetration depth can result
from gapless BdG fermions yielding $\varrho(E)\sim|E|$. Any $d$-wave pairing,
producing two nodal loops (see Table 1), is therefore a natural candidate for
the paired state in half-Heuslers (see also Refs. brydon-1 ; Herbut-3 ; roy-
nevidomskyy ; savary-2 ). It is admitted that more microscopic analysis is
needed to gain further insights into the global phase diagram of strongly
interacting spin-3/2 fermions in various materials, which we leave for future
investigation.

The energy-entropy competition and the proposed selection rule among competing
orders provide valuable insights into the overall structure of the global
phase diagram of strongly interacting spin-3/2 fermions. Various cuts of the
phase diagram, which we exposed by pursuing an unbiased renormalization group
analysis, corroborate (at least qualitatively) the former two approaches.
These approaches are not limited to interacting spin-3/2 fermions living in
three dimensions. The methodology is applicable for a large set of strongly
interacting multi-band systems, among which two dimensional Dirac
(semi)metals, doped Bernal-stacked bilayer graphene (supporting bi-quadratic
band touchings in two dimensions) graphene-RMP ; Balatsky ; vafek , twisted
bilayer graphene near so called the magic angle tblg-1 ; tblg-2 ; tblg-3 ,
three-dimensional Dirac or Weyl materials Weyl-RMP , doped topological
(Kondo-)insulators (described by massive Dirac fermions) TI-RMP-1 ; TI-RMP-2 ;
TKI-Review  and nodal loop metals roy-NLSM ; Shouvik-NLSM  are the prominent
and experimentally pertinent ones. In the future we will systematically study
these systems, which should allow us to gain further insights into the global
phase diagram of correlated materials, appreciate the role of emergent
topology inside various broken symmetry phases, and search for possible routes
to realize unconventional high temperature superconductors.

###### Acknowledgements.

This work was in part supported by Deutsche Forschungsgemeinschaft under grant
SFB 1143.

##  Appendix A Specific heat and compressibility

This appendix is devoted to present the scaling of specific heat ($C_{V}$) and
compressibility ($\kappa$) in a non-interacting Luttinger system, when the
chemical potential ($\mu$) is placed away from the band touching point. We
begin with the expression for the free-energy density in this system, given by
(after setting $k_{B}=1$)

$f=-DT\sum_{\tau=\pm}\int\frac{d^{3}{\bf
k}}{(2\pi)^{3}}\;\ln\left[2\cosh\left(\frac{E^{\tau}_{\bf
k}}{2T}\right)\right],$ (67)

where $D$ is the degeneracy of the valence and conduction band, hence $D=2$,
and

$E^{\tau}_{\bf k}=\frac{k^{2}}{2m}+\tau\mu.$ (68)

The chemical potential and momentum ($k$) are measured from the band touching
point. One can rewrite the above expression for the free energy density as

$\displaystyle f$ $\displaystyle=$
$\displaystyle-\frac{D}{2}\sum_{\tau=\pm}\int\frac{d^{3}{\bf
k}}{(2\pi)^{3}}E^{\tau}_{\bf k}$ (69) $\displaystyle-$ $\displaystyle
DT\sum_{\tau=\pm}\int\frac{d^{3}{\bf
k}}{(2\pi)^{3}}\ln\left[1+\exp\left(-\frac{E^{\tau}_{\bf
k}}{T}\right)\right].$

The first term is independent of temperature and not important for the
thermodynamic properties of the system. So we focus only on the second term.
After proper rescaling of variables the free-energy density becomes

$\displaystyle f$ $\displaystyle=$ $\displaystyle-
DT^{5/2}\frac{(2m)^{3/2}}{4\pi^{2}}\;\sum_{\tau=\pm}\int^{\infty}_{0}dy\;\sqrt{y}\;\ln\left[1+e^{-y-\tau\tilde{\mu}}\right],$

leading to Eq. (3), where $\tilde{\mu}=\mu/T$, and ${\rm Li}_{s}(z)$
represents the polylogarithm function of order $s$ and argument $z$. For
$\tilde{\mu}\ll 1$ the free-energy density reads as

$f=-DT^{5/2}\frac{(2m)^{3/2}}{8\pi^{3/2}}\left[a+b\;\frac{\mu^{2}}{T^{2}}+c\;\frac{\mu^{4}}{T^{4}}+\;\cdots\right],$
(71)

where

$\displaystyle a$ $\displaystyle=$
$\displaystyle\frac{1}{2}\left(4-\sqrt{2}\right)\;\zeta\left(\frac{5}{2}\right)\approx
1.7344,$ $\displaystyle b$ $\displaystyle=$
$\displaystyle\left(1-\sqrt{2}\right)\;\zeta\left(\frac{1}{2}\right)\approx
0.6049,$ $\displaystyle c$ $\displaystyle=$
$\displaystyle\frac{1}{12}\left(1-4\sqrt{2}\right)\;\zeta\left(-\frac{3}{2}\right)\approx
0.00989.$ (72)

From the above expression of the free-energy density we arrive the expression
for specific heat [see Eq. (4)] and compressibility [see Eq. (5)]. From these
two expressions we finally arrive at the following universal ratio

$\frac{C_{V}/T}{\kappa}=\frac{15\left(4-\sqrt{2}\right)}{16\left(1-\sqrt{2}\right)}\times\frac{\zeta\left(5/2\right)}{\zeta\left(1/2\right)}\approx
5.37611,$ (73)

reported in Eq. (6). This number is a characteristic of a $z=2$ scale
invariant fixed point in $d=3$.

Figure 21: Scaling of the universal function $F_{\rm Dr}(x)$ with its argument
$x$, appearing in the expression for the Drude conductivity of Luttinger
fermions, see Eq. (7).

##  Appendix B Dynamic conductivity

This appendix is devoted to disclose some key steps of the computation of the
dynamic conductivity in Luttinger system. We focus on an isotropic system (for
the sake of simplicity) and explicitly compute the $z$-component of the
conductivity ($\sigma_{zz}$). Due to the cubic symmetry
$\sigma_{zz}=\sigma_{xx}=\sigma_{yy}\equiv\sigma$ (say). To this end we use
the Kubo formula and compute the polarization bubble as a function of external
(Matsubara) frequency

$\displaystyle\Pi_{zz}(i\omega_{n})$ $\displaystyle=$
$\displaystyle-\frac{e^{2}}{\beta}\sum_{m}\int_{\bf k}\;{\bf
Tr}\bigg{[}\hat{j}_{z}\>G_{0}\left(ip_{m},{\bf k}\right)\>\hat{j}_{z}$ (74)
$\displaystyle\times$ $\displaystyle G_{0}\left(ip_{m}+i\omega_{n},{\bf
k}\right)\bigg{]},$

where $e$ is the electronic charge, $\beta$ is the inverse temperature and the
current operator along the $z$ direction is given by

$\hat{j}_{z}=-\frac{1}{2m}\left[\sqrt{3}\;k_{y}\Gamma_{1}+\sqrt{3}\;k_{x}\Gamma_{2}+2\;k_{z}\Gamma_{5}\right].$
(75)

In the spectral representation the noninteracting Greens function reads as

$G_{0}(i\omega_{n},{\bf
k})=\int^{\infty}_{-\infty}\>\frac{d\epsilon}{2\pi}\;\frac{{\mathcal{A}}(\epsilon,{\bf
k})}{i\omega_{n}-\epsilon},$ (76)

where

$\displaystyle{\mathcal{A}}(\epsilon,{\bf k})$ $\displaystyle=$
$\displaystyle\pi\left(1+\sum^{5}_{j=1}\Gamma_{j}\hat{d}_{j}\right)\>\delta\left(\omega+\mu-\frac{k^{2}}{2m}\right)$
$\displaystyle+$
$\displaystyle\pi\left(1-\sum^{5}_{j=1}\Gamma_{j}\hat{d}_{j}\right)\>\delta\left(\omega+\mu+\frac{k^{2}}{2m}\right).$

Components of $\hat{\bf d}$ are shown in Eq. (C). After performing the
analytic continuation $i\omega\to\omega+i\eta$, we find the dynamic
conductivity using the Kubo formula

$\sigma_{zz}(\omega)=\frac{\Im\>\Pi_{zz}\left(i\omega\to\omega+i\eta\right)}{\omega}.$
(78)

From the above expression we obtain both Drude and inter-band components of
the dynamic conductivity, respectively shown in Eqs. (7) and (8). The
universal function $F_{\rm Dr}(x)$ appearing in the expression of the Drude
component in Eq. (7) is given by

$~{}F_{\rm
Dr}(x)=\frac{32}{3}\int^{\infty}_{0}\;du\;u^{3/2}\>\sum_{\tau=\pm}\operatorname{sech}^{2}\left(u+\tau\frac{\mu}{2T}\right).$
(79)

The scaling of this function is shown in Fig. 21.

##  Appendix C Details of Luttinger model

In this Appendix we present some essential details of the Luttinger model,
introduced in Sec. III. The components of a five-dimensional unit vector
$\hat{\bf d}(\hat{\bf k})$, appearing in the Luttinger Hamiltonian [see Eq.
(13)], are given by

$\displaystyle~{}\hat{d}_{1}$ $\displaystyle=$
$\displaystyle\frac{i\left[Y^{1}_{2}+Y^{-1}_{2}\right]}{\sqrt{2}}=\frac{\sqrt{3}}{2}\sin
2\theta\,\sin\phi=\sqrt{3}\;\hat{k}_{y}\hat{k}_{z},$
$\displaystyle\hat{d}_{2}$ $\displaystyle=$
$\displaystyle\frac{\left[Y^{-1}_{2}+Y^{1}_{2}\right]}{\sqrt{2}}=\frac{\sqrt{3}}{2}\sin
2\theta\cos\phi=\sqrt{3}\;\hat{k}_{x}\hat{k}_{z},$ $\displaystyle\hat{d}_{3}$
$\displaystyle=$
$\displaystyle\frac{i\left[Y^{-2}_{2}+Y^{-2}_{2}\right]}{\sqrt{2}}=\frac{\sqrt{3}}{2}\sin^{2}\theta\sin
2\phi=\sqrt{3}\;\hat{k}_{y}\hat{k}_{x},$ $\displaystyle\hat{d}_{4}$
$\displaystyle=$
$\displaystyle\frac{\left[Y^{-2}_{2}+Y^{2}_{2}\right]}{\sqrt{2}}=\frac{\sqrt{3}}{2}\sin^{2}\theta\cos
2\phi=\frac{\sqrt{3}}{2}\left[\hat{k}^{2}_{x}-\hat{k}^{2}_{y}\right],$
$\displaystyle\hat{d}_{5}$ $\displaystyle=$ $\displaystyle
Y^{0}_{2}=\frac{1}{2}\left(3\cos^{2}\theta-1\right)=\frac{1}{2}\left[2\hat{k}^{2}_{z}-\hat{k}^{2}_{x}-\hat{k}^{2}_{y}\right],$
(80)

where $Y^{m}_{l}\equiv Y^{m}_{l}(\theta,\phi)$,
$\hat{d}_{j}\equiv\hat{d}_{j}(\hat{k})$, and $\theta$ and $\phi$ are the polar
and azimuthal angles in the momentum space, respectively. Five mutually anti-
commuting $\Gamma$ matrices are [see also Eq. (III.1)]

$\displaystyle\Gamma_{1}$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{3}}\\{J_{y},J_{z}\\},\Gamma_{2}=\frac{1}{\sqrt{3}}\\{J_{x},J_{z}\\},\Gamma_{3}=\frac{1}{\sqrt{3}}\\{J_{x},J_{y}\\},$
$\displaystyle\Gamma_{4}$ $\displaystyle=$
$\displaystyle\,\frac{1}{\sqrt{3}}\left[J_{x}^{2}-J_{y}^{2}\right],\Gamma_{5}=\frac{1}{3}\left[2J_{z}^{2}-J_{x}^{2}-J_{y}^{2}\right],$
(81)

where $\\{A,B\\}\equiv AB+BA$, and ${\bf J}$ are three spin-3/2 matrices. Both
$\hat{{\bf d}}$ and ${\bf\Gamma}$ transform as vectors under the cubic point
group ($O_{h}$), and their scalar product yields the Luttinger Hamiltonian, an
$A_{1g}$ quantity.

On the other hand, ten commutators $\Gamma_{jk}=[\Gamma_{j},\Gamma_{k}]/(2i)$
with $j>k$ can be expressed in terms of the products of _odd_ number of
spin-3/2 matrices as follows

$\displaystyle\Gamma_{45}$ $\displaystyle=$
$\displaystyle-\frac{2}{\sqrt{3}}\left[J_{x}J_{y}J_{z}+J_{z}J_{y}J_{x}\right],\Gamma_{12}=\frac{1}{3}\left[7J_{z}-4J^{3}_{z}\right],$
$\displaystyle\Gamma_{13}$ $\displaystyle=$
$\displaystyle-\frac{1}{3}\left[7J_{y}-4J^{3}_{y}\right],\Gamma_{23}=\frac{1}{3}\left[7J_{x}-4J^{3}_{x}\right],$
$\displaystyle\Gamma_{34}$ $\displaystyle=$
$\displaystyle-\frac{1}{6}\left[13J_{z}-4J^{3}_{z}\right],\Gamma_{35}=\frac{1}{\sqrt{3}}\left\\{J_{z},J^{2}_{x}-J^{2}_{y}\right\\},$
$\displaystyle\Gamma_{14}$ $\displaystyle=$
$\displaystyle\frac{1}{12}\left[13J_{x}-4J^{3}_{x}\right]+\frac{1}{2}\left\\{J_{x},J^{2}_{y}-J^{2}_{z}\right\\},$
(82) $\displaystyle\Gamma_{15}$ $\displaystyle=$
$\displaystyle\frac{1}{4\sqrt{3}}\left[13J_{x}-4J^{3}_{x}\right]-\frac{1}{2\sqrt{3}}\left\\{J_{x},J^{2}_{y}-J^{2}_{z}\right\\},$
$\displaystyle\Gamma_{24}$ $\displaystyle=$
$\displaystyle\frac{1}{12}\left[13J_{y}-4J^{3}_{y}\right]-\frac{1}{2}\left\\{J_{y},J^{2}_{z}-J^{2}_{x}\right\\},$
$\displaystyle\Gamma_{25}$ $\displaystyle=$
$\displaystyle-\frac{1}{4\sqrt{3}}\left[13J_{y}-4J^{3}_{y}\right]-\frac{1}{2\sqrt{3}}\left\\{J_{y},J^{2}_{z}-J^{2}_{x}\right\\}.$

The four-dimensional identity matrix can be written as

$\Gamma_{0}=\frac{4}{15}\left(J^{2}_{x}+J^{2}_{y}+J^{2}_{z}\right).$ (83)

##  Appendix D Fierz Identity

In this Appendix we present the Fierz reduction of the number of linearly
independent quartic terms for the interacting Luttinger system. To perform
this exercise for generic local density-density interactions we introduce a
_six_ component vector

$\displaystyle X^{\top}$ $\displaystyle=$
$\displaystyle\bigg{[}(\Psi^{\dagger}\Gamma_{0}\Psi)^{2},\sum^{3}_{j=1}(\Psi^{\dagger}\Gamma_{j}\Psi)^{2},\sum^{5}_{j=4}(\Psi^{\dagger}\Gamma_{j}\Psi)^{2},$
$\displaystyle(\Psi^{\dagger}\Gamma_{45}\Psi)^{2},\sum^{3}_{j=1}(\Psi^{\dagger}\Gamma_{j}\Gamma_{45}\Psi)^{2},\sum^{3}_{j=1}\sum^{5}_{k=4}(\Psi^{\dagger}\Gamma_{jk}\Psi)^{2}\bigg{]},$

constituted by the quartic terms appearing in $H_{\rm int}$, see Eq. (V). The
Fierz identity allows us to rewrite each quartic term appearing in $X$ as a
linear combination of the remaining ones according to

$\displaystyle\left[\Psi^{\dagger}(x)M\Psi(x)\right]\left[\Psi^{\dagger}(y)N\Psi(y)\right]=$
$\displaystyle-\frac{1}{16}\left(\mathrm{Tr}M\Gamma^{a}N\Gamma^{b}\right)\left[\Psi^{\dagger}(x)\Gamma^{b}\Psi(y)\right]\left[\Psi^{\dagger}(y)\Gamma^{a}\Psi(x)\right],$
(85)

where $M$ and $N$ are four-dimensional Hermitian matrices and for local
interactions $x=y$. The set of sixteen matrices
$\\{\Gamma_{a},\>a=1,\cdots,16\\}$ closes the $U(4)$ Clifford algebra of four-
dimensional matrices, and we choose
$\Gamma^{\dagger}_{a}=\Gamma_{a}=(\Gamma_{a})^{-1}$. The above Fierz
constraint then takes a compact form $FX=0$, where $F$ is the Fierz matrix
given by

$F=\begin{bmatrix}5&1&1&1&1&1\\\ 3&3&-3&3&-1&1\\\ 2&-2&4&-2&2&0\\\
1&1&-1&5&1&-1\\\ 3&-1&3&3&3&-1\\\ 6&2&0&-6&-2&4\end{bmatrix}.$ (86)

The rank of $F$ is 3. Hence, the number of lienarly independent quartic terms
is $3=6$ (dimensionality of $F$) – $3$ (rank of $F$). We chose four-fermion
interactions proportional to $\lambda_{0}$, $\lambda_{1}$ and $\lambda_{3}$ as
_three_ linearly independent quartic terms. The remaining three quartic terms
can then be expressed in terms of linear combinations of the above three
according to

$\displaystyle~{}(\Psi^{\dagger}\Gamma_{45}\Psi)^{2}$ $\displaystyle=$
$\displaystyle-\frac{1}{2}(\Psi^{\dagger}\Psi)^{2}-\frac{1}{2}\sum^{3}_{j=1}(\Psi^{\dagger}\Gamma_{j}\Psi)^{2}$
$\displaystyle+$
$\displaystyle\frac{1}{2}\sum^{5}_{j=4}(\Psi^{\dagger}\Gamma_{j}\Psi)^{2},$
$\displaystyle\sum^{3}_{j=1}(\Psi^{\dagger}\Gamma_{j}\Gamma_{45}\Psi)^{2}$
$\displaystyle=$
$\displaystyle-\frac{3}{2}(\Psi^{\dagger}\Psi)^{2}+\frac{1}{2}\sum^{3}_{j=1}(\Psi^{\dagger}\Gamma_{j}\Psi)^{2}$
$\displaystyle-$
$\displaystyle\frac{3}{2}\sum^{5}_{j=4}(\Psi^{\dagger}\Gamma_{j}\Psi)^{2},$
$\displaystyle\sum^{3}_{j=1}\sum^{5}_{k=4}(\Psi^{\dagger}\Gamma_{jk}\Psi)^{2}$
$\displaystyle=$
$\displaystyle-3(\Psi^{\dagger}\Psi)^{2}-\sum^{3}_{j=1}(\Psi^{\dagger}\Gamma_{j}\Psi)^{2}.$
(87)

Therefore, during the RG analysis whenever we generate any one of the above
three quartic terms we can immediately express them in terms of the ones
proportional to $\lambda_{0}$, $\lambda_{1}$ and $\lambda_{2}$. Therefore, the
interacting model [see Eq. (36)] remains closed under coarse garining to any
order in the perturbation theory.

##  Appendix E Details of RG flow equations

The RG flow equations, displayed in Eq. (45), are expressed in terms of the
functions $H^{j}_{km}(\alpha,t,\mu)$, where $j,k,m=0,1,2$. For brevity we here
drop the explicit dependence of these functions on $\alpha,t$ and $\mu$. The
functions appearing in the $\beta$ function of $g_{{}_{0}}$ are given by

$\displaystyle H_{00}^{0}$ $\displaystyle=$
$\displaystyle\frac{5}{4}\left[-f_{0}-\tilde{f}_{0}-4f_{g}-6f_{t}\right],\>H_{11}^{0}=\frac{5}{4}\left[-6\tilde{f}_{0}-6f_{g}+6\tilde{f}_{g}-3f_{t}+3\tilde{f}_{t}\right],$
$\displaystyle H_{22}^{0}$ $\displaystyle=$
$\displaystyle\frac{5}{4}\left[f_{0}-3\tilde{f}_{0}-3f_{t}+3\tilde{f}_{t}\right],\>H_{01}^{0}=\frac{5}{4}\left[6f_{0}+6f_{g}+6\tilde{f}_{g}+18f_{t}+12\tilde{f}_{t}\right],$
$\displaystyle H_{02}^{0}$ $\displaystyle=$
$\displaystyle\frac{5}{2}\left[2f_{0}+5f_{g}+3f_{t}+3\tilde{f}_{g}+3\tilde{f}_{t}\right],\>H_{12}^{0}=\frac{15}{2}\left[-f_{0}-\tilde{f}_{0}-f_{g}+\tilde{f}_{g}-2f_{t}+2\tilde{f}_{t}\right].$
(88)

The six functions appearing in the RG flow equation for $g_{{}_{1}}$ are given
by

$\displaystyle H_{00}^{1}$ $\displaystyle=$
$\displaystyle\frac{5}{4}(f_{t}+\tilde{f}_{t}),\>H_{11}^{1}=\frac{5}{4}(-5f_{0}+\tilde{f}_{0}+10f_{g}+2\tilde{f}_{g}+10f_{t}+10\tilde{f}_{t}),\>H_{22}^{1}=\frac{5}{4}(-f_{0}-\tilde{f}_{0}+3f_{t}+\tilde{f}_{t}),$
$\displaystyle H_{01}^{1}$ $\displaystyle=$
$\displaystyle\frac{5}{2}(2f_{0}-\tilde{f}_{0}-3f_{g}+\tilde{f}_{g}-\tilde{f}_{t}),\>H_{02}^{1}=-\frac{5}{2}(f_{g}-\tilde{f}_{g}+f_{t}-\tilde{f}_{t}),\>H_{12}^{1}=-\frac{5}{2}(3f_{0}+\tilde{f}_{0}-7f_{g}-\tilde{f}_{g}-2\tilde{f}_{t}).$
(89)

Finally, the flow equation for $g_{{}_{2}}$ is expressed in terms of the
following six functions

$\displaystyle H_{00}^{2}$ $\displaystyle=$
$\displaystyle\frac{5}{4}(f_{g}+\tilde{f}_{g}),\>H_{11}^{2}=\frac{5}{4}(-3f_{0}-3\tilde{f}_{0}+3f_{g}+3\tilde{f}_{g}+3f_{t}-3\tilde{f}_{t}),\>H_{22}^{2}=\frac{5}{4}(-3f_{0}+\tilde{f}_{0}+4f_{g}+4\tilde{f}_{g}+9f_{t}+3\tilde{f}_{t}),$
$\displaystyle H_{01}^{2}$ $\displaystyle=$
$\displaystyle\frac{5}{4}(-6f_{t}+6\tilde{f}_{t}),\>H_{02}^{2}=\frac{5}{2}(2f_{0}-\tilde{f}_{0}+f_{g}-\tilde{f}_{g}-3f_{t}),\>H_{12}^{2}=\frac{15}{2}(-f_{0}-f_{g}+\tilde{f}_{g}+4f_{t}+\tilde{f}_{t}).$
(90)

Note that in the above expression $f_{j}\equiv f_{j}(\alpha,t,\mu)$ and
$\tilde{f}_{j}=\tilde{f}_{j}(\alpha,t,\mu)$s for $j=0,t,g$, and

$\displaystyle f_{0}(\alpha,t,\mu)$ $\displaystyle=$
$\displaystyle-\frac{1}{2}\int\mathrm{d}\Omega\>\frac{1}{2t}\sum_{\tau=\pm
1}\left[\mathrm{sech}^{2}\left(\frac{f+\tau\mu}{2t}\right)+\frac{2t}{f}\tanh\left(\frac{f+\tau\mu}{2t}\right)\right],$
(91) $\displaystyle f_{t}(\alpha,t,\mu)$ $\displaystyle=$
$\displaystyle-\frac{1}{2}\int\mathrm{d}\Omega\>\frac{\cos^{2}\alpha}{2t}\sum_{\tau=\pm
1}\left[\frac{1}{f^{2}}\mathrm{sech}^{2}\left(\frac{f+\tau\mu}{2t}\right)-\frac{2t}{f^{3}}\tanh\left(\frac{f+\tau\mu}{2t}\right)\right]\frac{\hat{d}_{1}^{2}+\hat{d}_{2}^{2}+\hat{d}_{3}^{2}}{3},$
(92) $\displaystyle f_{g}(\alpha,t,\mu)$ $\displaystyle=$
$\displaystyle-\frac{1}{2}\int\mathrm{d}\Omega\>\frac{\sin^{2}\alpha}{2t}\sum_{\tau=\pm
1}\left[\frac{1}{f^{2}}\mathrm{sech}^{2}\left(\frac{f+\tau\mu}{2t}\right)-\frac{2t}{f^{3}}\tanh\left(\frac{f+\tau\mu}{2t}\right)\right]\frac{\hat{d}_{4}^{2}+\hat{d}_{5}^{2}}{2},$
(93) $\displaystyle\tilde{f}_{0}(\alpha,t,\mu)$ $\displaystyle=$
$\displaystyle-\frac{1}{2}\int\mathrm{d}\Omega\>\frac{1}{\mu\left(1-(\frac{\mu}{f})^{2}\right)}\sum_{\tau=\pm
1}\tanh\left(\frac{f+\tau\mu}{2t}\right)\left[\tau+\frac{\mu}{f}-2\tau\left(\frac{\mu}{f}\right)^{2}\right],$
(94) $\displaystyle\tilde{f}_{t}(\alpha,t,\mu)$ $\displaystyle=$
$\displaystyle\frac{1}{2}\int\mathrm{d}\Omega\>\frac{\cos^{2}\alpha}{f^{2}\mu\left(1-(\frac{\mu}{f})^{2}\right)}\sum_{\tau=\pm
1}\tanh\left(\frac{f+\tau\mu}{2t}\right)\left[-\tau+\frac{\mu}{f}\right]\frac{\hat{d}_{1}^{2}+\hat{d}_{2}^{2}+\hat{d}_{3}^{2}}{3},$
(95) $\displaystyle\tilde{f}_{g}(\alpha,t,\mu)$ $\displaystyle=$
$\displaystyle\frac{1}{2}\int\mathrm{d}\Omega\>\frac{\sin^{2}\alpha}{f^{2}\mu\left(1-(\frac{\mu}{f})^{2}\right)}\sum_{\tau=\pm
1}\tanh\left(\frac{f+\tau\mu}{2t}\right)\left[-\tau+\frac{\mu}{f}\right]\frac{\hat{d}_{4}^{2}+\hat{d}_{5}^{2}}{2},$
(96)

where $d\Omega=\sin\theta d\theta d\phi$, $f\equiv f(\alpha,\theta,\phi)$ and

$f(\alpha,\theta,\phi)=\sqrt{\cos^{2}\alpha(\hat{d}_{1}^{2}+\hat{d}_{2}^{2}+\hat{d}_{3}^{2})+\sin^{2}\alpha(\hat{d}_{4}^{2}+\hat{d}_{5}^{2})}.$
(97)

The definition of $\hat{\bf d}$ is already provided in Eq. (C).

In terms of $f_{j}$s and $\tilde{f}_{j}$s we can express the leading order RG
flow equations for all source terms [see Eq. (48)] as

$\displaystyle\frac{d\ln\Delta_{0}}{dl}-2$ $\displaystyle=$
$\displaystyle-\frac{5}{4}(f_{0}+2f_{g}+3f_{t})(3g_{0}-3g_{1}-2g_{2}),$
$\displaystyle\frac{d\ln\Delta_{1}}{dl}-2$ $\displaystyle=$
$\displaystyle\frac{5}{4}(f_{0}-2f_{g}-f_{t})(g_{0}-5g_{1}-2g_{2}),$
$\displaystyle\frac{d\ln\Delta_{2}}{dl}-2$ $\displaystyle=$
$\displaystyle\frac{5}{4}(f_{0}-3f_{t})(g_{0}-3g_{1}-4g_{2}),$
$\displaystyle\frac{d\ln\Delta_{3}}{dl}-2$ $\displaystyle=$
$\displaystyle\frac{5}{4}(f_{0}-2f_{g}+3f_{t})(g_{0}+3g_{1}-2g_{2}),$
$\displaystyle\frac{d\ln\Delta_{4}}{dl}-2$ $\displaystyle=$
$\displaystyle\frac{5}{4}(f_{0}+2f_{g}-f_{t})(g_{0}-g_{1}+2g_{2}),$
$\displaystyle\frac{d\ln\Delta_{5}}{dl}-2$ $\displaystyle=$
$\displaystyle\frac{5}{4}(f_{0}+f_{t})(g_{0}+g_{1}),$
$\displaystyle\frac{d\ln\Delta^{\rm p}_{A_{1g}}}{dl}-2$ $\displaystyle=$
$\displaystyle-\frac{5}{4}(\tilde{f}_{0}-2\tilde{f}_{g}-3\tilde{f}_{t})(g_{0}+3g_{1}+2g_{2}),$
$\displaystyle\frac{d\ln\Delta^{\rm p}_{T_{2g}}}{dl}-2$ $\displaystyle=$
$\displaystyle-\frac{5}{4}(\tilde{f}_{0}+2\tilde{f}_{g}+\tilde{f}_{t})(g_{0}-g_{1}-2g_{2}),$
$\displaystyle\frac{d\ln\Delta^{\rm p}_{E_{g}}}{dl}-2$ $\displaystyle=$
$\displaystyle-\frac{5}{4}(\tilde{f}_{0}+3\tilde{f}_{t})(g_{0}-3g_{1}).$ (98)

These flow equations are schematically shown in Eq. (49).

##  Appendix F Stability matrix analysis

FPs | $\left(g^{\ast}_{0},g^{\ast}_{1},g^{\ast}_{2}\right)\times 10^{3}$ | EVs of SM
---|---|---
Gaussian | $(0,0,0)$ | $(-1,-1,-1)\epsilon$
QCP${}^{1}_{\frac{\pi}{4}}$ | $\left(2.24,2.03,2.03\right)\epsilon$ | $(1,-0.870,-0.704)\epsilon$
QCP${}^{1}_{\frac{\pi}{2}}$ | $\left(1.49,1.47,1.26\right)\epsilon$ | $(-1.075,1,-0.654)\epsilon$
QCP${}^{2}_{\frac{\pi}{2}}$ | $\left(-2.93,-2.84,3.76\right)\epsilon$ | $(-1.169,1,-0.933)\epsilon$
BCP${}_{\frac{\pi}{2}}$ | $\left(-1.58,-1.33,5.54\right)\epsilon$ | $(1.271,1,-0.527)\epsilon$
QCP${}^{1}_{0}$ | $\left(1.95,1.71,1.94\right)\epsilon$ | $(1,-0.889,-0.642)\epsilon$
QCP${}^{2}_{0}$ | $\left(-5.98,4.80,-5.87\right)\epsilon$ | $(-1.131,1,-0.134)\epsilon$
BCP0 | $\left(-5.60,4.97,-5.48\right)\epsilon$ | $(-1.075,1,0.132)\epsilon$
Table 4:  Three eigenvalues (EVs) of the stability matrix (SM)
$M\left(g_{{}_{0}},g_{{}_{1}},g_{{}_{2}}\right)$, defined in Eq. (99) at
various fixed points (FPs) located at
$\left(g^{\ast}_{{}_{0}},g^{\ast}_{{}_{1}},g^{\ast}_{{}_{2}}\right)$, see
Table 2. The trivial Guassian fixed point is found for any arbitrary value of
$\alpha$.

To gain insight into the local structure of the RG flow trajectories in the
close proximity to any fixed point, we perform the _stability_ analysis around
them. To proceed with this analysis we introduce a $3\times 3$ stability
matrix $M\left(g_{{}_{0}},g_{{}_{1}},g_{{}_{2}}\right)$ as follows

$~{}M\left(g_{{}_{0}},g_{{}_{1}},g_{{}_{2}}\right)=\left[\begin{array}[]{ccc}\frac{d\beta_{g_{{}_{0}}}}{dg_{{}_{0}}}&\frac{d\beta_{g_{{}_{1}}}}{dg_{{}_{0}}}&\frac{d\beta_{g_{{}_{2}}}}{dg_{{}_{0}}}\\\
\frac{d\beta_{g_{{}_{0}}}}{dg_{{}_{1}}}&\frac{d\beta_{g_{{}_{1}}}}{dg_{{}_{1}}}&\frac{d\beta_{g_{{}_{2}}}}{dg_{{}_{1}}}\\\
\frac{d\beta_{g_{{}_{0}}}}{dg_{{}_{2}}}&\frac{d\beta_{g_{{}_{1}}}}{dg_{{}_{2}}}&\frac{d\beta_{g_{{}_{2}}}}{dg_{{}_{2}}}\end{array}\right].$
(99)

We classify the fixed points, located at
$(g^{\ast}_{{}_{0}},g^{\ast}_{{}_{1}},g^{\ast}_{{}_{2}})$, according to the
number of positive and negative eignevalues of
$M\left(g^{\ast}_{{}_{0}},g^{\ast}_{{}_{1}},g^{\ast}_{{}_{2}}\right)$.

1\. There exists only one fixed point at
$\left(g^{\ast}_{{}_{0}},g^{\ast}_{{}_{1}},g^{\ast}_{{}_{2}}\right)=(0,0,0)$
with three negative eigenvalues $(-1,-1,-1)\epsilon$ of the stability matrix.
This fixed point, referred as _Gaussian_ in Table 4, is stable from all
directions and represents the noninteracting LSM for sufficiently weak but
generic short-range interactions.

2\. Fixed points possessing one positive and two negative eigenvalues for the
stability matrix are the QCPs. Such fixed points control QPTs from LSM to
BSPs. Since such QPTs can be triggered by tuning only one parameter (the
interaction strength along the relevant direction, determined by the
eigenvector associated with the positive eigenvalue of the stability matrix),
they are continuous in nature. Concomitantly, a single parameter scaling
emerges in the vicinity of all QCPs. The positive eigenvalue of the stability
matrix also determines the correlation length exponent ($\nu$) at the QCPs
[see Sec. VI.2].

3\. Fixed points with two positive and one negative eigenvalues for the
stability matrix are the BCPs. Typically BCPs are found when there exists more
than one QCP in the multi-dimensional coupling constant space. For example,
only around $\alpha=\frac{\pi}{2}$ and $0$, when the coupled RG flow equations
support two QCPs, there exists one BCP (respectively denoted by
BCP${}_{\frac{\pi}{2}}$ and BCP0). Existence of such a BCP is necessary to
ensure the continuity of the RG flow trajectories. A BCP also separates the
basin of attractions of two QCPs.

A summary of the eigenvalues for the stability matrix is presented in Table 4.
We did not find any fixed point with three positive eigenvalues of the
corresponding stability matrix.

## References

  * (1) P. M. Chaikin and T. C. Lubensky, _Principles of Condensed Matter Physics_ (Cambridge University Press, 1st ed., Cambridge, UK, 1995).
  * (2) M. Tinkham, _Introduction to Superconductivity_ , (Dover Publications, 2nd ed., 2004).
  * (3) J. M. Luttinger, _Quantum Theory of Cyclotron Resonance in Semiconductors: General Theory_ , Phys. Rev. 102, 1030 (1956).
  * (4) S. Murakami, S-C. Zhang, and N. Nagaosa, _SU(2) non-Abelian holonomy and dissipationless spin current in semiconductors_ , Phys. Rev. B 69, 235206 (2004).
  * (5) R. Dornhaus, G. Nimtz, and B. Schlicht, _Narrow-Gap Semicounductors_ , (Springer-Verlag, 1983).
  * (6) S. Groves and W. Paul, _Band Structure of Gray Tin_ , Phys. Rev. Lett. 11, 194 (1963).
  * (7) B. J. Roman and A. W. Ewald, _Stress-Induced Band Gap and Related Phenomena in Gray Tin_ , Phys. Rev. B 5, 3914 (1974).
  * (8) X. Wan, A. M. Turner, A. Vishwanath, and S. Y. Savrasov, _Topological semimetal and Fermi-arc surface states in the electronic structure of pyrochlore iridates_ , Phys. Rev. B 83, 205101 (2011).
  * (9) W. Witczak-Krempa, G. Chen, Y. B. Kim, and L. Balents, _Correlated quantum phenomena in the strong spin-orbit regime_ , Annu. Rev. Condens. Matter Phys. 5, 57 (2014).
  * (10) T. Kondo, M. Nakayama, R. Chen, J. J. Ishikawa, E.-G. Moon, T. Yamamoto, Y. Ota, W. Malaeb, H. Kanai, Y. Nakashima, Y. Ishida, R. Yoshida, H. Yamamoto, M. Matsunami, S. Kimura, N. Inami, K. Ono, H. Kumigashira, S. Nakatsuji, L. Balents, and S. Shin, _Quadratic Fermi node in a 3D strongly correlated semimetal_ , Nat. Commun. 6, 10042 (2015).
  * (11) M. Nakayama, T. Kondo, Z. Tian, J. J. Ishikawa, M. Halim, C. Bareille, W. Malaeb, K. Kuroda, T. Tomita, S. Ideta, K. Tanaka, M. Matsunami, S. Kimura, N. Inami, K. Ono, H. Kumigashira, L. Balents, S. Nakatsuji, and S. Shin, _Slater to Mott Crossover in the Metal to Insulator Transition of Nd 2Ir2O7_, Phys. Rev. Lett. 117, 056403 (2016).
  * (12) B. Cheng, T. Ohtsuki, D. Chaudhuri, S. Nakatsuji, M. Lippmaa, N. P. Armitage, _Dielectric anomalies and interactions in the three-dimensional quadratic band touching Luttinger semimetal Pr 2Ir2O7_, Nat. Commun. 8, 2097 (2017).
  * (13) H. Lin, L. A. Wray, Y. Xia, S. Xu, S. Jia, R. J. Cava, A. Bansil, and M. Z. Hasan, _Half-Heusler ternary compounds as new multifunctional experimental platforms for topological quantum phenomena_ , Nat. Mater. 9, 546 (2010).
  * (14) Z. Liu, L. Yang, S-C. Wu, C. Shekhar, J. Jiang, H. Yang, Y. Zhang, S-K. Mo, Z. Hussain, B. Yan, C. Felser, and Y. Chen, _Observation of Unusual Topological Surface States in Half-Heusler Compounds LnPtBi (Ln=Lu, Y)_ , Nat. Commun. 7, 12924 (2016).
  * (15) J. Yu, B. Yan, and C-X. Liu, _Model Hamiltonian and time reversal breaking topological phases of antiferromagnetic half-Heusler materials_ , Phys. Rev. B 95, 235158 (2017).
  * (16) K. Matsuhira, M. Wakeshima, Y. Hinatsu, and S. Takagi, _Metal–Insulator Transitions in Pyrochlore Oxides Ln 2Ir2O7_, J. Phys. Soc. Jpn. 80, 094701 (2011).
  * (17) W. Witczak-Krempa and Y. B. Kim, _Topological and magnetic phases of interacting electrons in the pyrochlore iri- dates_ , Phys. Rev. B 85, 045124 (2012).
  * (18) W. Witczak-Krempa, A. Go, and Y. B. Kim, _Pyrochlore electrons under pressure, heat, and field: Shedding light on the iridates_ , Phys. Rev. B 87, 155101 (2013).
  * (19) B.-J. Yang and N. Nagaosa, _Emergent Topological Phenomena in Thin Films of Pyrochlore Iridates_ , Phys. Rev. Lett. 112, 246402 (2014).
  * (20) L. Savary, E.-G. Moon, and L. Balents, _New Type of Quantum Criticality in the Pyrochlore Iridates_ , Phys. Rev. X 4, 041027 (2014).
  * (21) Y. Yamaji and M. Imada, _Metallic Interface Emerging at Magnetic Domain Wall of Antiferromagnetic Insulator: Fate of Extinct Weyl Electrons_ , Phys. Rev. X 4, 021035 (2014).
  * (22) H. Shinaoka, S. Hoshino, M. Troyer, and P. Werner, _Phase Diagram of Pyrochlore Iridates: All-in All-out Magnetic Ordering and Non-Fermi-Liquid Properties_ , Phys. Rev. Lett. 115, 156401 (2015).
  * (23) K. Ueda, J. Fujioka, Y. Takahashi, T. Suzuki, S. Ishiwata, Y. Taguchi, and Y. Tokura, _Variation of Charge Dynamics in the Course of Metal-Insulator Transition for Pyrochlore-Type Nd 2Ir2O7_, Phys. Rev. Lett. 109, 136402 (2012).
  * (24) K. Ueda, J. Fujioka, B.-J. Yang, J. Shiogai, A. Tsukazaki, S. Nakamura, S. Awaji, N. Nagaosa, and Y. Tokura, _Magnetic Field-Induced Insulator-Semimetal Transition in a Pyrochlore Nd 2Ir2O7_, Phys. Rev. Lett. 115, 056402 (2015).
  * (25) Y. Machida, S. Nakatsuji, Y. Maeno, T. Tayama, T. Sakakibara, and S. Onoda, _Unconventional Anomalous Hall Effect Enhanced by a Noncoplanar Spin Texture in the Frustrated Kondo Lattice Pr 2Ir2O7_, Phys. Rev. Lett. 98, 057203 (2007).
  * (26) Y. Machida, S. Nakatsuji, S. Onoda, T. Tayama, and T. Sakakibara, _Time-reversal symmetry breaking and sponta- neous Hall effect without magnetic dipole order_ , Nature (London) 463, 210 (2010).
  * (27) L. Balicas, S. Nakatsuji, Y. Machida, and S. Onoda, _Anisotropic Hysteretic Hall Effect and Magnetic Control of Chiral Domains in the Chiral Spin States of Pr 2Ir2O7_, Phys. Rev. Lett. 106, 217204 (2011).
  * (28) P. Goswami, B. Roy, and S. Das Sarma, _Competing orders and topology in the global phase diagram of pyrochlore iridates_ , Phys. Rev. B 95, 085120 (2017).
  * (29) G. Goll, M. Marz, A. Hamann, T. Tomanic, K. Grube, T. Yoshino, and T. Takabatake, _Thermodynamic and transport properties of the non-centrosymmetric superconductor LaBiPt_ , Physica B 403, 1065 (2008).
  * (30) N. P. Butch, P. Syers, K. Kirshenbaum, A. P. Hope, and J. Paglione, _Superconductivity in the topological semimetal YPtBi_ , Phys. Rev. B 84, 220504 (2011).
  * (31) T. V. Bay, T. Naka, Y. K. Huang, and A. de Visser, _Superconductivity in noncentrosymmetric YPtBi under pressure_ , Phys. Rev. B 86, 064515 (2012).
  * (32) Y. Pan, A. M. Nikitin, T. V. Bay, Y. K. Huang, C. Paulsen, B. H. Yan, and A. de Visser, _Superconductivity and magnetic order in the non-centrosymmetric Half Heusler compound ErPdBi_ , Eur. Phys. Lett. 104, 27001 (2013).
  * (33) F. F. Tafti, T. Fujii, A. Juneau-Fecteau, S. René de Cotret, N. Doiron-Leyraud, A. Asamitsu, and L. Taillefer, _Superconductivity in the noncentrosymmetric half-Heusler compound LuPtBi: A candidate for topological superconductivity_ , Phys. Rev. B 87, 184504 (2013).
  * (34) G. Xu, W. Wang, X. Zhang, Y. Du, E. Liu, S. Wang, G. Wu, Z. Liu, and X.-X. Zhang, _Weak Antilocalization Effect and Noncentrosymmetric Superconductivity in a Topologically Nontrivial Semimetal LuPdBi_ , Sci. Rep. 4, 5709 (2014).
  * (35) A. M. Nikitin, Y. Pan, X. Mao, R. Jehee, G. K. Araizi, Y. K. Huang, C. Paulsen, S. C. Wu, B. H. Yan, and A. de Visser, _Magnetic and superconducting phase diagram of the half-Heusler topological semimetal HoPdBi_ , J. Phys.: Condens. Matter 27, 275701 (2015).
  * (36) Y. Nakajima, R. Hu, K. Kirshenbaum, A. Hughes, P. Syers, X. Wang, K. Wang, R. Wang, S. Saha, D. Pratt, J. W. Lynn, and J. Paglione, _Topological $R$PdBi half-Heusler semimetals: a new family of non-centrosymmetric magnetic superconductors_, Sci. Adv. 5, e1500242 (2015).
  * (37) H. Kim, K. Wang, Y. Nakajima, R. Hu, S. Ziemak, P. Syers, L. Wang, H. Hodovanets, J. D. Denlinger, P. M. R. Brydon, D. F. Agterberg, M. A. Tanatar, R. Prozorov, and J. Paglione, _Beyond Triplet: Unconventional Superconductivity in a Spin-3/2 Topological Semimetal_ , Sci. Adv. 4, eaao4513 (2018).
  * (38) H. Xiao, T. Hu, W. Liu, Y. L. Zhu, P. G. Li, G. Mu, J. Su, K. Li, and Z. Q. Mao, _Superconductivity in the half-Heusler compound TbPdBi_ , Phys. Rev. B 97, 224511 (2018).
  * (39) E-G. Moon, C. Xu, Y-B. Kim, and L. Balents, _Non-Fermi-Liquid and Topological States with Strong Spin-Orbit Coupling_ , Phys. Rev. Lett. 111, 206401 (2013).
  * (40) I. F. Herbut and L. Janssen, _Topological Mott insulator in three-dimensional systems with quadratic band touching_ , Phys. Rev. Lett. 113, 106401 (2014).
  * (41) H-H. Lai, B. Roy, P. Goswami, _Disordered and interacting parabolic semimetals in two and three dimensions_ , arXiv:1409.8675 (unpublished).
  * (42) L. Janssen, I. F. Herbut, _Excitonic instability of three-dimensional gapless semiconductors: Large-N theory_ , Phys. Rev. B 93, 165109 (2016).
  * (43) I. Boettcher, and I. F. Herbut, _Anisotropy induces non-Fermi-liquid behavior and nematic magnetic order in three-dimensional Luttinger semimetals_ , Phys. Rev. B 95, 075149 (2017).
  * (44) M. Kharitonov, J-B. Mayer, E. M. Hankiewicz, _Universality and Stability of the Edge States of Chiral-Symmetric Topological Semimetals and Surface States of the Luttinger Semimetal_ , Phys. Rev. Lett. 119, 266402 (2017).
  * (45) R. Wang, A. Go, A. Millis, _Weyl Rings and enhanced susceptibilities in Pyrochlore Iridates: $k\cdot p$ Analysis of Cluster Dynamical Mean-Field Theory Results_, Phys. Rev. B 96, 195158 (2017).
  * (46) T. Oh, H. Ishizuka, and B.-J. Yang, _Magnetic field induced topological semimetals near the quantum critical point of pyrochlore iridates_ , Phys. Rev. B 98, 144409 (2018).
  * (47) C. Fang, B. A. Bernevig, and M. J. Gilbert, _Tri-Dirac surface modes in topological superconductors_ , Phys. Rev. B 91, 165421 (2015).
  * (48) W. Yang, Y. Li, and C. Wu, _Topological Septet Pairing with Spin- $3/2$ Fermions: High-Partial-Wave Channel Counterpart of the 3He-$B$ Phase_, Phys. Rev. Lett. 117, 075301 (2016).
  * (49) P. M. R. Brydon, L. Wang, M. Weinert, and D. F. Agterberg, _Pairing of $j=3/2$ fermions in half-Heusler superconductors_, Phys. Rev. Lett. 116, 177001 (2016).
  * (50) C. Timm, A. P. Schnyder, D. F. Agterberg, and P. M. R. Brydon, _Inflated nodes and surface states in superconducting half-Heusler compounds_ , Phys. Rev. B 96, 094526 (2017).
  * (51) I. Boettcher and I. F. Herbut, _Unconventional Superconductivity in Luttinger Semimetals: Theory of Complex Tensor Order and Emergence of the Uniaxial Nematic State_ , Phys. Rev. Lett. 120, 057002 (2018).
  * (52) L. Savary, J. Ruhman, J. W. F. Venderbos, L. Fu, and P. A. Lee, _Superconductivity in three-dimensional spin-orbit coupled semimetals_ , Phys. Rev. B 96, 214514 (2017).
  * (53) B. Roy, S. A. A. Ghorashi, M. S. Foster, A. H. Nevidomskyy, _Topological superconductivity of spin-3/2 carriers in a three-dimensional doped Luttinger semimetal_ , arXiv:1708.07825
  * (54) J. W. F. Venderbos, L. Savary, J. Ruhman, P. A. Lee, L. Fu, _Pairing states of spin-3/2 fermions: Symmetry-enforced topological gap functions_ , Phys. Rev. X 8, 011029 (2018).
  * (55) T. Kawakami, T. Okamura, S. Kobayashi, and M. Sato, _Topological Crystalline Materials of $J=3/2$ Electrons: Antiperovskites, Dirac points, and High Winding Topological Superconductivity_, Phys. Rev. X 8, 041026 (2018).
  * (56) Q-Z. Wang, J. Yu, C-X. Liu, _Unconventional superconductivity and Surface pairing symmetry in Half-Heusler Compounds_ , Phys. Rev. B 97, 224507 (2018).
  * (57) J. Yu, and C-X. Liu, _Singlet-Quintet Mixing in Spin-Orbit Coupled Superconductors with $j=3/2$ Fermions_, Phys. Rev. B 98, 104514 (2018).
  * (58) P. M. R. Brydon, D. F. Agterberg, H. Menke, C. Timm, _Bogoliubov Fermi surfaces: General theory, magnetic order, and topology_ , arXiv:1806.03773
  * (59) G-B. Sim, A. Mishra, M. J. Park, Y-B. Kim, G. Y. Cho, S-B. Lee, _Topological $d+s$ wave superconductors in a multi-orbital quadratic band touching system_, arXiv:1811.04046
  * (60) A. A. Abrikosov and S. D. Beneslavskii, _Possible Existence of Substances Intermediate Between Metals and Dielectrics_ , Sov. Phys. JETP 32, 699 (1971).
  * (61) A. A. Abrikosov, _Calculation of critical indices for zero-gap semiconductors_ , Sov. Phys. JETP 66, 1443 (1974).
  * (62) D. Sherrington and W. Kohn, _Speculations about Grey Tin_ , Rev. Mod. Phys. 40, 767 (1968).
  * (63) A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov, A. K. Geim, _The electronic properties of graphene_ , Rev. Mod. Phys. 81, 109 (2009).
  * (64) T. O. Wehling, A. M Black-Schaffer, A. V Balatsky, _Dirac Materials_ , Adv. Phys. 63 1 (2014).
  * (65) M. Z Hasan, and C. L. Kane, _Colloquium: Topological insulators_ , Rev. Mod. Phys. 82, 3045 (2010).
  * (66) X. L. Qi, and S.-C. Zhang, _Topological insulators and superconductors_ , Rev. Mod. Phys. 83, 1057 (2011).
  * (67) M. Dzero, J. Xia, V. Galitski, P. Coleman, _Topological Kondo Insulators_ , Annu. Rev. Condens. Matter Phys. 7, 249 (2016).
  * (68) N. P. Armitage, E. J. Mele, A. Vishwanath, _Weyl and Dirac Semimetals in Three Dimensional Solids_ , Rev. Mod. Phys. 90, 15001 (2018).
  * (69) Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, R. C. Ashoori, and P. Jarillo-Herrero, _Correlated insulator behaviour at half-filling in magic-angle graphene superlattices_ , Nature 556, 80 (2018).
  * (70) Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, _Unconventional superconductivity in magic-angle graphene superlattices_ , Nature 556, 43 (2018).
  * (71) M. Yankowitz, S. Chen, H. Polshyn, K. Watanabe, T. Taniguchi, D. Graf, A. F. Young, C. R. Dean, _Tuning superconductivity in twisted bilayer graphene_ , arXiv:1808.07865
  * (72) R. Shankar, _Renormalization-group approach to interacting fermions_ , Rev. Mod. Phys. 66, 129 (1994).
  * (73) W. Kohn and J. M. Luttinger, _New Mechanism for Superconductivity_ , Phys. Rev. Lett. 15, 524 (1965).
  * (74) M. A. Baranov, A. V. Chubukov, and M. Yu. Kagan, _Superconductivity and Superfluidity in Fermi Systems with Repulsive Interactions_ , Int. Jour. of Mod. Phys. B 06, 2471 (1992).
  * (75) D. Zanchi and H. J. Schulz, _Superconducting instabilities of the non-half-filled Hubbard model in two dimensions_ , Phys. Rev. B 54, 9509 (1996).
  * (76) J. Mráz, and R. Hlubina, _Kohn-Luttinger instability of the $t-t^{\prime}$ Hubbard model in two dimensions: Variational approach_, Phys. Rev. B 67, 174518 (2003).
  * (77) S. Raghu, S. A. Kivelson, and D. J. Scalapino, _Superconductivity in the repulsive Hubbard model: An asymptotically exact weak-coupling solution_ , Phys. Rev. B 81, 224505 (2010).
  * (78) J. M. Luttinger, _Fermi Surface and Some Simple Equilibrium Properties of a System of Interacting Fermions_ , Phys. Rev. 119, 1153 (1960).
  * (79) S. Chakravarty, B. I. Halperin, and D. R. Nelson, _Two-dimensional quantum Heisenberg antiferromagnet at low temperatures_ , Phys. Rev. B 39, 2344 (1989).
  * (80) O. Vafek, J. M. Murray, V. Cvetkovic, _Superconductivity on the brink of spin-charge order in doped honeycomb bilayer_ , Phys. Rev. Lett. 112, 147002 (2014).
  * (81) S. L. Sondhi, S. M. Girvin, J. P. Carini, and D. Shahar, _Continuous quantum phase transitions_ , Rev. Mod. Phys. 69, 315 (1997).
  * (82) S. Sachdev, _Quantum Phase Transitions_ (Cambridge University Press, 2nd ed., England, 2007).
  * (83) S. Borisenko, Q. Gibson, D. Evtushinsky, V. Zabolotnyy, B. Büchner, and R. J. Cava, _Experimental Realization of a Three-Dimensional Dirac Semimetal_ , Phys. Rev. Lett. 113, 027603 (2014).
  * (84) Z. K. Liu, B. Zhou, Y. Zhang, Z. J. Wang, H. M. Weng, D. Prabhakaran, S.-K. Mo, Z. X. Shen, Z. Fang, X. Dai, Z. Hussain, Y. L. Chen, _Discovery of a Three-Dimensional Topological Dirac Semimetal, Na 3Bi_, Science 343, 864 (2014).
  * (85) M. Sigrist and K. Ueda, _Phenomenological theory of unconventional superconductivity_ , Rev. Mod. Phys. 63, 239 (1991).
  * (86) I. F. Herbut, V. Juričić and B. Roy, _Theory of interacting electrons on the honeycomb lattice_ , Phys. Rev. B 79, 085116 (2009).
  * (87) B. Roy, and S. Das Sarma, _Quantum phases of interacting electrons in three-dimensional dirty Dirac semimetals_ , Phys. Rev. B 94, 115137 (2016).
  * (88) B. Roy, P. Goswami and V. Juričić, _Interacting Weyl fermions: Phases, phase transitions, and global phase diagram_ , Phys. Rev. B 95, 201102 (2017).
  * (89) B. Roy, _Interacting nodal-line semimetal: Proximity effect and spontaneous symmetry breaking_ , Phys. Rev. B 96, 041113 (2017).
  * (90) S. Sur and R. Nandkishore, _Instabilities of Weyl loop semimetals_ , New J. Phys. 18, 115006 (2016).