Hong<EMAIL_ADDRESS>Janarbek<EMAIL_ADDRESS>Shubhankar<EMAIL_ADDRESS>Yizhe<EMAIL_ADDRESS>Amin<EMAIL_ADDRESS>Fatih<EMAIL_ADDRESS>Qualcomm AI Research
San Diego, CA, USA
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
Second and third authors contributed equally to this work Nanjing University
of Science and Technology
Nanjing, China
Work done at Qualcomm AI Research Qualcomm Technologies, Inc.
San Diego, CA, USA X-Distill

# X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task
Distillation

###### Abstract

In this paper, we propose a novel method, X-Distill, to improve the self-
supervised training of monocular depth via cross-task knowledge distillation
from semantic segmentation to depth estimation. More specifically, during
training, we utilize a pretrained semantic segmentation teacher network and
transfer its semantic knowledge to the depth network. In order to enable such
knowledge distillation across two different visual tasks, we introduce a
small, trainable network that translates the predicted depth map to a semantic
segmentation map, which can then be supervised by the teacher network. In this
way, this small network enables the backpropagation from the semantic
segmentation teacher’s supervision to the depth network during training. In
addition, since the commonly used object classes in semantic segmentation are
not directly transferable to depth, we study the visual and geometric
characteristics of the objects and design a new way of grouping them that can
be shared by both tasks. It is noteworthy that our approach only modifies the
training process and does not incur additional computation during inference.
We extensively evaluate the efficacy of our proposed approach on the standard
KITTI benchmark and compare it with the latest state of the art. We further
test the generalizability of our approach on Make3D. Overall, the results show
that our approach significantly improves the depth estimation accuracy and
outperforms the state of the art.

##  1 Introduction

Accurate monocular depth estimation plays a critical role in 3D visual scene
understanding and is of great importance for a variety of application domains,
such as self-driving, AR/VR, and robotics. Thanks to the advancement of deep
learning algorithms, recent years have seen considerable progress in this area
[Ming et al.(2021)Ming, Meng, Fan, and Yu]. However, training accurate deep
learning models in a supervised manner requires high-quality (e.g., dense and
correctly aligned) ground-truth depth maps, which are difficult and costly to
obtain.

In order to overcome this challenge, self-supervision has emerged as a new
paradigm for training monocular depth estimation models [Zhou et
al.(2017)Zhou, Brown, Snavely, and Lowe, Godard et al.(2017)Godard, Mac Aodha,
and Brostow, Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow]. Since
the inception of such self-supervised training, researchers have looked at
various directions in order to further improve the depth estimation accuracy,
such as designing more complex architectures [Guizilini et
al.(2020a)Guizilini, Ambrus, Pillai, Raventos, and Gaidon, Johnston and
Carneiro(2020), Lyu et al.(2021)Lyu, Liu, Wang, Kong, Liu, Liu, Chen, and
Yuan], improving the photometric matching [Shu et al.(2020)Shu, Yu, Duan, and
Yang, Jiang et al.(2020)Jiang, Ding, Sun, and Huang], handling dynamic objects
[Casser et al.(2019)Casser, Pirk, Mahjourian, and Angelova, Gordon et
al.(2019)Gordon, Li, Jonschkowski, and Angelova, Dai et al.(2020)Dai, Patil,
Hecker, Dai, Van Gool, and Schindler, Klingner et al.(2020)Klingner,
Termöhlen, Mikolajczyk, and Fingscheidt], utilizing edge information [Ramirez
et al.(2018)Ramirez, Poggi, Tosi, Mattoccia, and Di Stefano, Zhu et
al.(2020)Zhu, Brazil, and Liu, Saeedan and Roth(2021)], multi-task learning
[Yin and Shi(2018), Chen et al.(2019b)Chen, Schmid, and Sminchisescu, Ranjan
et al.(2019)Ranjan, Jampani, Balles, Kim, Sun, Wulff, and Black, Luo et
al.(2019)Luo, Yang, Wang, Wang, Xu, Nevatia, and Yuille, Tosi et
al.(2020)Tosi, Aleotti, Ramirez, Poggi, Salti, Stefano, and Mattoccia], and
exploiting temporal information [Patil et al.(2020)Patil, Van Gansbeke, Dai,
and Van Gool].

Given the importance of visual scene understanding for depth estimation,
researchers have recently started to study how to utilize semantic
segmentation information to improve accuracy. In [Yue et al.(2020)Yue, Fu, Wu,
and Wang, Guizilini et al.(2020b)Guizilini, Hou, Li, Ambrus, and Gaidon, Kumar
et al.(2021)Kumar, Klingner, Yogamani, Milz, Fingscheidt, and Mader], the
authors use pretrained or jointly trained semantic segmentation networks to
assist the depth network during both training and test. While such approaches
can considerably improve accuracy, they incur significant extra computation
during inference as they require running a separate and usually heavy-weight
segmentation network. Another route is to incorporate the semantic information
into the loss function, which only requires the extra computation of semantic
information during training. One possible way is to include semantic
segmentation as an auxiliary task, by co-training a semantic network and a
depth network that share a set of layers [Tosi et al.(2020)Tosi, Aleotti,
Ramirez, Poggi, Salti, Stefano, and Mattoccia]. Other papers compare the
semantic segmentations on both the warped and actual versions of a frame, and
enforce a consistency regularization [Yang et al.(2018)Yang, Zhao, Shi, Deng,
and Jia, Chen et al.(2019a)Chen, Liu, Liu, and Wang]. However, this requires
running the segmentation network in every training iteration, which still
incurs considerable overhead. In [Zhu et al.(2020)Zhu, Brazil, and Liu], the
authors use the segmentation masks to explicitly regularize the edges on the
depth map, but their approach requires semantic labels on the same dataset and
introduces many additional hyper-parameters.

In this paper, we propose a novel cross-task knowledge distillation approach,
X-Distill, to utilize semantic information to improve self-supervised
monocular depth estimation. Given a pretrained semantic segmentation teacher
network, our goal is to transfer the semantic knowledge from this teacher
network to the depth network during training, in order to enhance the depth
network’s visual scene understanding capability. Note that our setting is
different from the conventional knowledge distillation where the teacher and
student networks share the same visual task. In our case, the outputs of the
depth network and the semantic segmentation network are not directly
comparable. In order to enable such cross-task distillation, we utilize a
small neural network to connect segmentation and depth, by generating semantic
segmentation based on the predicted depth. The resulting depth-based semantic
segmentation is then supervised by the teacher network. The small network is
trained together with the depth network and as such, allows backpropagation
from the semantic segmentation teacher’s supervision to the depth network.

In addition to enabling gradient flow across the two tasks, it is necessary to
redesign the semantic classes such that they are compatible with the visual
information in the depth map. In particular, the classes commonly used in
semantic segmentation are usually too fine-grained for depth. For instance,
road and sidewalk are typically treated as two separate classes in semantic
segmentation. However, it is not necessary to treat them differently on the
depth map since both of them are on the ground surface and have highly similar
depth variation patterns in the field of view. As such, we regroup the objects
based on their visual and geometric characteristics. This allows the depth
network to distill the key depth-relevant semantic information, without
introducing unnecessary difficulties to the learning process.

We next summarize our main contributions as follows:

• We propose a novel method, X-Distill, to exploit semantic information to
improve self-supervised monocular depth estimation. X-Distill enables the
depth network to distill semantic knowledge in a cross-task manner from a
segmentation teacher network during training. At inference time, the depth
network then runs in a standalone manner, without requiring extra computation
to process/generate semantic information.

• In order to make the semantic segmentation knowledge compatible with the
visual information in depth, we regroup the semantic classes based on the
visual and geometric characteristics of the objects. This allows the depth
network to distill the key semantic knowledge while removing the unnecessary
complexities in the learning.

• We evaluate our proposed approach on KITTI and Make3D, and compare it with
the state of the art. We further conduct extensive ablation studies on our
method. Overall, our proposed approach achieves considerably more accurate
depth estimation, e.g., outperforming [Godard et al.(2019)Godard, Mac Aodha,
Firman, and Brostow] by $14\%$ on KITTI (in terms of squared relative error).

##  2 Related Work

Self-Supervised Monocular Depth Estimation: Due to the difficulty of
collecting dense, high-quality ground-truth depth maps, researchers have
proposed self-supervised training to obtain monocular depth estimation models.
Such self-supervision leverages the geometric relationship among neighboring
video frames [Zhou et al.(2017)Zhou, Brown, Snavely, and Lowe, Godard et
al.(2019)Godard, Mac Aodha, Firman, and Brostow] or between the left and right
cameras in a stereo setting [Godard et al.(2017)Godard, Mac Aodha, and
Brostow]. While these methods provide a new way to train a depth network
without labels, factors such as moving objects, occlusion, poor lighting, and
low texture can considerably degrade their performance.

Utilizing Semantic Information for Depth Estimation: Given the high
correlation between semantic and depth information, researchers have studied
how to incorporate semantic information to improve depth accuracy. One way is
to run an additional (sub)network to generate semantic information at
inference time, which can be fed to the depth network [Yue et al.(2020)Yue,
Fu, Wu, and Wang, Guizilini et al.(2020b)Guizilini, Hou, Li, Ambrus, and
Gaidon, Kumar et al.(2021)Kumar, Klingner, Yogamani, Milz, Fingscheidt, and
Mader]. While this approach can considerably improve the depth estimation
performance, it incurs significantly more computation. Other works include new
loss functions during training, either via multi-task training [Tosi et
al.(2020)Tosi, Aleotti, Ramirez, Poggi, Salti, Stefano, and Mattoccia] or by
enforcing segmentation consistency between the warped and real images [Yang et
al.(2018)Yang, Zhao, Shi, Deng, and Jia, Chen et al.(2019a)Chen, Liu, Liu, and
Wang, Meng et al.(2019)Meng, Lu, Raj, Sunarjo, Guo, Javidi, Bansal, and
Bharadia]. These methods do not require extra semantic computation during
test, but require running a semantic network at every training iteration,
which still generates a considerable overhead.

Knowledge Distillation: Knowledge Distillation is usually used to transfer the
knowledge from a more complex model to a smaller model, where both of them are
designed for the same visual task [Gou et al.(2021)Gou, Yu, Maybank, and Tao].
Few papers have looked at knowledge distillation across two different visual
tasks, e.g., classification tasks with non-overlapping classes [Ye et
al.(2020)Ye, Lu, and Zhan], classification and text-to-image synthesis [Yuan
and Peng(2019)], RGB-based depth estimation and depth super resolution [Sun et
al.(2021)Sun, Ye, Li, Li, Wang, and Xu]. None of the existing works has
studied cross-task distillation from semantic segmentation to depth and we
show how to enable it in this paper.

##  3 Proposed Method

In this section, we present our novel take, X-Distill, on utilizing semantic
segmentation to improve self-supervised monocular depth estimation, through
cross-task distillation. In order to transfer the relevant knowledge from a
semantic segmentation teacher network to the depth network during training, we
use a small network to translate depth to segmentation, thus enabling gradient
flow across the two visual tasks. In addition, we redesign the semantic
classes to make them compatible with the visual information contained in
depth.

###  3.1 Self-Supervised Monocular Depth Estimation

We utilize self-supervision to train a monocular depth estimation model, based
on single-view video sequences [Godard et al.(2019)Godard, Mac Aodha, Firman,
and Brostow, Zhou et al.(2017)Zhou, Brown, Snavely, and Lowe].

Geometric Modeling: Consider two neighboring video frames, $I_{t}$ and
$I_{s}$. Suppose that pixel $p_{t}\in I_{t}$ and pixel $p_{s}\in I_{s}$ are
two different views of the same point of an object, then $p_{t}$ and $p_{s}$
are related geometrically as follows:

$d(p_{s})h(p_{s})=\mathbf{K}(\mathbf{R}_{t\rightarrow
s}\mathbf{K}^{-1}d(p_{t})h(p_{t})+\mathbf{t}_{t\rightarrow s}),\vspace{-7pt}$
(1)

where $h(p)=[h,\,w,\,1]$ denotes the homogeneous coordinates of a pixel $p$
with $h$ and $w$ being its vertical and horizontal positions on the image,
$d(p)$ is the depth at $p$, $\mathbf{K}\in\mathbb{R}^{3\times 3}$ is the
camera intrinsic matrix, and $\mathbf{T}_{t\rightarrow
s}=[\mathbf{R}_{t\rightarrow s}|\mathbf{t}_{t\rightarrow
s}]\in\mathbb{R}^{3\times 4}$ is the 6DoF relative camera motion/pose from $t$
to $s$, with $\mathbf{R}_{t\rightarrow s}\in\mathbb{R}^{3\times 3}$ and
$\mathbf{t}_{t\rightarrow s}\in\mathbb{R}^{3\times 1}$ being the rotation
matrix and translation vector.

Given the depth map of $I_{t}$, denoted by $D_{t}$, and the relative camera
pose from $I_{t}$ to $I_{s}$, we can synthesize $I_{t}$ from $I_{s}$ based on
Eq. 1, assuming that the 3D points captured in $I_{t}$ are also present in
$I_{s}$. We denote the synthesized/warped version of $I_{t}$ as
$\widehat{I}_{t}$.

Self-Supervised Training: Suppose that the depth map and the relative camera
pose are provided by a depth network and a pose network, respectively. By
minimizing the difference between the warped and actual versions of $I_{t}$,
we can train these two networks. A common photometric loss function for
comparing $I_{t}$ and $\widehat{I}_{t}$ is given by

$\mathcal{L}_{\text{PH}}(I_{t},\,\widehat{I}_{t})=\alpha\|I_{t}-\widehat{I}_{t}\|_{1}+(1-\alpha)\frac{1-\text{SSIM}(I_{t},\,\widehat{I}_{t})}{2},\vspace{-7pt}$
(2)

where $\|\cdot\|_{1}$ denotes the $\mathcal{L}_{1}$ norm and SSIM is the
Structural Similarity Index Measure [Wang et al.(2004)Wang, Bovik, Sheikh, and
Simoncelli]. Note that $\mathcal{L}_{\text{PH}}$ is computed in a per-pixel
manner.

It is common to further include a smoothness regularization to prevent drastic
variations in the predicted depth map. Furthermore, in practice, not all the
3D points in $I_{t}$ can be found in $I_{s}$, due to occlusion and objects
(partially) moving out of the frame. Some objects can also be moving (e.g.,
cars), which is not considered in the geometric model of Eq. 1. In order to
correctly measure the photometric loss and train the networks, it is a common
practice to mask out the pixel points that violate the geometric model (see
[Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] for more details
on the masking techniques). Fig. 1 (gray block) illustrates the self-
supervised training scheme of a monocular depth network.

Figure 1: Overview of our proposed X-Distill approach. The gray block
describes the self-supervised training of a monocular depth network based on
single-view videos. The blue block illustrates our proposed cross-task
semantics-to-depth distillation. By utilizing a trainable depth-to-
segmentation network to translate predicted depth to segmentation, we enable
cross-task knowledge transfer from the pretrained segmentation teacher network
to the depth network during training. In addition, we regroup the semantic
classes such that they become compatible with the visual information in depth.

###  3.2 Cross-Task Distillation from Semantics to Depth

Consider a depth network, $f_{D}$, and a pretrained semantic segmentation
network, $f_{S}$. Our goal is to transfer the knowledge of the teacher
network, $f_{S}$, to the depth network, $f_{D}$. However, unlike conventional
knowledge distillation where teacher and student networks are used for the
same visual task, $f_{D}$ and $f_{S}$ are used for two different tasks, and
their outputs are not directly comparable. In other words, given an input, we
cannot directly measure the difference between the outputs of $f_{D}$ and
$f_{S}$ in order to generate a loss to train $f_{D}$.

As such, we utilize a Depth-to-Segmentation (D2S) neural network, $h_{D2S}$,
to translate depth to semantic segmentation. Given the segmentation map
generated from the predicted depth map, we are now able to construct a
segmentation loss to distill semantic knowledge from $f_{S}$ to $f_{D}$. More
formally, the new loss term is given as follows:

$\mathcal{L}_{D2S}(S_{t}^{\text{D}},S_{t})=\sum_{i=1}^{H}\sum_{j=1}^{W}\frac{\mathcal{\mathcal{L}}_{\text{CE}}(S^{\text{D}}_{t}(i,\,j),\,S_{t}(i,\,j))}{HW},\vspace{-7pt}$
(3)

where $S_{t}^{D}=h_{D2S}(f_{D}(I_{t}))$ is the semantic segmentation map
generated by $h_{D2S}$ based on the predicted depth map $D_{t}=f_{D}(I_{t})$,
$S_{t}$ is the semantic segmentation output generated by the semantic
segmentation teacher network, $\mathcal{L}_{\text{CE}}$ denotes the cross-
entropy loss, and $H$ and $W$ are the height and width of the input
image.111Note that we can include a “background” class for the pretrained
segmentation model (which is a common practice). This will allow us to ignore
pixels that are not of interest when computing the distillation loss of Eq. 3.

The total loss is then given by

$\mathcal{L}_{\text{Total}}=\sum_{k=1}^{N_{s}}\mathcal{L}_{PH,\,k}+\sum_{k=1}^{N_{s}}\lambda_{SM,\,k}\mathcal{L}_{SM,\,k}+\lambda_{D2S}\mathcal{L}_{D2S},\vspace{-5pt}$
(4)

where the self-supervised depth loss is computed over $N_{s}$ scales,
$\mathcal{L}_{PH,\,k}$ is the photometric loss at the $k^{\text{th}}$ scale,
$\lambda_{SM,\,k}$ and $\mathcal{L}_{SM,\,k}$ are the weight and loss for the
smoothness regularization at the $k^{\text{th}}$ scale, and $\lambda_{D2S}$ is
the weight of the cross-task distillation loss, $\mathcal{L}_{D2S}$.

It can be seen that during training, $h_{D2S}$ is jointly trained with the
depth network. This makes it possible for the pretrained teacher network to
provide semantic supervision to the depth network, by backpropagating through
$h_{D2S}$. Our proposed approach is illustrated in Fig. 1, with the semantics-
to-depth distillation module highlighted in the blue block.

For the depth-to-segmentation network, $h_{D2S}$, we adopt a small
architecture. More specifically, $h_{D2S}$ consists of two $3\\!\times\\!3$
convolutional layers, each followed by a BatchNorm layer and a ReLu layer, as
well as a pointwise convolutional layer at the end which outputs the
segmentation. Note that the $h_{D2S}$ should not be too complex, since a
deeper network would take over too much of the learning load and weaken the
knowledge flow to the depth network. As we shall see in our experiments in
Sec. 4, while using a deeper $h_{D2S}$ can still increase the accuracy of the
depth network, the improvement is not as significant as that by using our
proposed smaller $h_{D2S}$.

Once the training is finished, the depth network can then run in a standalone
manner, without requiring any extra computation of semantic information during
inference. Furthermore, our proposed distillation approach only adds a small
amount of computation to training. More specifically, the segmentation maps
from the teacher network only need to be computed once and the additional
forward/backward passes are cheap since $h_{D2S}$ is small.

###  3.3 Depth-Compatible Grouping of Semantic Classes

Semantic segmentation usually contains much more fine-grained visual
recognition information that is not present in the depth map. For instance,
road and sidewalk are typically treated as two different semantic classes.
However, the depth map does not contain such classification information as
both road and sidewalk are on the ground plane and have similar depth
variations. As a result, it is not necessary to differentiate them on the
depth map. On the other hand, the depth map does contain the information for
differentiating certain classes. For instance, a road participant (e.g.,
pedestrian, vehicle) can be easily separated from the background (e.g., road,
building) given the different patterns of their depth values.

As such, is it necessary to reconsider the grouping of semantic classes, such
that the key semantic information is preserved while the unnecessary
complexity is removed from the distillation. Table 1 summarizes our new
grouping, which results in four groups.222We focus on outdoor scenes in this
paper and will consider an extension to indoor scenes as part of future work.
In the first two groups, we have objects in the foreground. The respective
foreground objects in these two groups are then further differentiated based
on their shapes, where the first group contains thin structures, e.g., traffic
lights/signs (including the poles), and the second group consists of people
and vehicles which are of more general shapes. The third and fourth groups
then contain the background objects, such as buildings, vegetation, road, and
sidewalk. We further separate the ground plane (e.g., road and sidewalk) from
the other background objects.

Object Groups | Foreground | Shape of | Location in
---|---|---|---
vs. Background | Bounding Box | 3D Space
Thin objects | Foreground | Thin rectangular | Above ground plane
People and vehicles | Foreground | Rectangular | Above ground plane
Background objects | Background | Not boundable | Above ground plane
Ground | Background | Not boundable | On ground plane
Table 1: Depth-compatible semantic class grouping for outdoor scenes. Figure
2: Depth estimation on three sample images. The second row shows the estimated
depth maps by Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and
Brostow] and the third row shows the depth maps by our proposed X-Distill
approach. It can be seen that our method provides more accurate depth
estimation. The green boxes indicate sample regions where our method
considerably improves the estimation quality.

##  4 Experiments

In this section (and also in the supplementary file), we present a
comprehensive performance analysis on our proposed X-Distill approach and
compare with the current state of the art. We furthermore conduct in-depth
ablation studies on various aspects of our method.

###  4.1 Experiment Setup

Datasets: We evaluate depth estimation on KITTI [Geiger et al.(2013)Geiger,
Lenz, Stiller, and Urtasun] using the standard Eigen split [Eigen and
Fergus(2015)], with two input resolutions, $640\\!\times\\!192$ and
$1024\\!\times\\!320$. Following [Zhou et al.(2017)Zhou, Brown, Snavely, and
Lowe], we remove the static frames in the training set. There are 39,810,
4,424, and 697 samples for training, validation, and test. We use Cityscapes
[Cordts et al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson,
Franke, Roth, and Schiele] training and validation sets to train the
segmentation teacher network. We further use Make3D [Saxena et
al.(2005)Saxena, Chung, and Ng, Saxena et al.(2008)Saxena, Sun, and Ng] to
evaluate the generalizability of our KITTI-trained model.

Grouping of Semantic Classes: We group the Cityscapes classes according to our
proposed scheme in Table 1, such that they are compatible with the depth
information. More specifically, we have 1) thin objects: poles and traffic
lights/signs, 2) people and vehicles: persons, riders, cars, trucks, buses,
motorcycles, bicycles, and trains, 3) background objects: buildings, walls,
fences, vegetation, terrain, and sky, and 4) ground: road and sidewalk.

Networks: For the depth network and the pose network, we use the ResNet-50
(RN50)-based models in [Godard et al.(2019)Godard, Mac Aodha, Firman, and
Brostow]. The semantic segmentation teacher network is an HRNet [Wang et
al.(2020)Wang, Sun, Cheng, Jiang, Deng, Zhao, Liu, Mu, Tan, Wang, Liu, and
Xiao] with OCR [Yuan et al.(2020)Yuan, Chen, and Wang] and InverseForm [Borse
et al.(2021)Borse, Wang, Zhang, and Porikli]. It has an mIoU of 85.6% on
Cityscapes test set. During the self-supervised training of the depth network,
this segmentation network is frozen.

Hyperparameters: For the self-supervised part, we follow the hyperparameter
setting in [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow]. For
the semantics-to-depth distillation loss, $\mathcal{L}_{D2S}$, we linearly
increase its weight from 0 to 0.005 during training. As we shall see in the
ablation studies, this linear schedule can improve the training as compared to
using a constant weight.

Evaluation Metrics: We use the commonly used error metrics to evaluate the
depth estimation performance, including the Absolute Relative Error (Abs Rel),
Squared Relative Error (Sq Rel), Root Mean Squared Error (RMSE), and the RMSE
of the log of the depth values. In addition, we use the classification
metrics, $\delta_{1}$, $\delta_{2}$, and $\delta_{3}$, which measure whether
the ratio between the ground-truth and predicted depth values is within a
certain interval around 1. Mathematical definitions of these metrics can be
found in the supplementary file.

###  4.2 Results

We extensively compare our proposed approach with the latest state of the art
(SOTA) on KITTI, including methods that 1) use more complex architectures
[Guizilini et al.(2020a)Guizilini, Ambrus, Pillai, Raventos, and Gaidon,
Johnston and Carneiro(2020), Lyu et al.(2021)Lyu, Liu, Wang, Kong, Liu, Liu,
Chen, and Yuan], 2) require additional computation of semantic information
during inference [Guizilini et al.(2020b)Guizilini, Hou, Li, Ambrus, and
Gaidon], 3) utilize semantic information during training and do not incur
extra computation during test [Klingner et al.(2020)Klingner, Termöhlen,
Mikolajczyk, and Fingscheidt], 4) propose better photometric matching [Jiang
et al.(2020)Jiang, Ding, Sun, and Huang, Shu et al.(2020)Shu, Yu, Duan, and
Yang], 5) utilize multiple frames [Patil et al.(2020)Patil, Van Gansbeke, Dai,
and Van Gool], and 6) perform multi-task learning [Tosi et al.(2020)Tosi,
Aleotti, Ramirez, Poggi, Salti, Stefano, and Mattoccia]. Note that we do not
consider pretraining/online finetuning of the depth network or applying post-
processing on the predicted depth maps. We also analyze both the depth
estimation accuracy and computation efficiency of the methods. Furthermore, we
test our KITTI-trained model on Make3D and compare it with the related SOTA to
evaluate generalizability. Finally, we perform extensive ablation studies on
our proposed approach.

Method | Resolution | Lower is Better | Higher is Better
---|---|---|---
Abs Rel | Sq Rel | RMSE | $\text{RMSE}_{\text{Log}}$ | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$
Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] (RN18) | $640\times 192$ | 0.115 | 0.903 | 4.863 | 0.193 | 0.877 | 0.959 | 0.981
Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] (RN50) | $640\times 192$ | 0.110 | 0.831 | 4.642 | 0.187 | 0.883 | 0.962 | 0.982
Tosi et al. [Tosi et al.(2020)Tosi, Aleotti, Ramirez, Poggi, Salti, Stefano, and Mattoccia]† | $640\times 192$ | 0.120 | 0.792 | 4.750 | 0.191 | 0.856 | 0.958 | 0.984
PackNet-SfM [Guizilini et al.(2020a)Guizilini, Ambrus, Pillai, Raventos, and Gaidon] | $640\times 192$ | 0.111 | 0.785 | 4.601 | 0.189 | 0.878 | 0.960 | 0.982
Johnston et al. [Johnston and Carneiro(2020)] (RN18) | $640\times 192$ | 0.111 | 0.941 | 4.817 | 0.185 | 0.885 | 0.961 | 0.981
Johnston et al. [Johnston and Carneiro(2020)] (RN101) | $640\times 192$ | 0.106 | 0.861 | 4.699 | 0.185 | 0.889 | 0.962 | 0.982
HR-Depth [Lyu et al.(2021)Lyu, Liu, Wang, Kong, Liu, Liu, Chen, and Yuan] | $640\times 192$ | 0.109 | 0.792 | 4.632 | 0.185 | 0.884 | 0.962 | 0.983
Guizilini et al. [Guizilini et al.(2020b)Guizilini, Hou, Li, Ambrus, and Gaidon]† (RN18) | $640\times 192$ | 0.117 | 0.854 | 4.714 | 0.191 | 0.877 | 0.959 | 0.981
Guizilini et al. [Guizilini et al.(2020b)Guizilini, Hou, Li, Ambrus, and Gaidon]† (RN50) | $640\times 192$ | 0.113 | 0.831 | 4.663 | 0.189 | 0.878 | 0.971 | 0.982
Klingner et al. [Klingner et al.(2020)Klingner, Termöhlen, Mikolajczyk, and Fingscheidt]† (RN18) | $640\times 192$ | 0.113 | 0.835 | 4.693 | 0.191 | 0.879 | 0.961 | 0.981
Klingner et al. [Klingner et al.(2020)Klingner, Termöhlen, Mikolajczyk, and Fingscheidt]† (RN50) | $640\times 192$ | 0.112 | 0.833 | 4.688 | 0.190 | 0.884 | 0.961 | 0.981
DiPE [Jiang et al.(2020)Jiang, Ding, Sun, and Huang] | $640\times 192$ | 0.112 | 0.875 | 4.795 | 0.190 | 0.880 | 0.960 | 0.981
Patil et al. [Patil et al.(2020)Patil, Van Gansbeke, Dai, and Van Gool] | $640\times 192$ | 0.111 | 0.821 | 4.650 | 0.187 | 0.883 | 0.961 | 0.982
X-Distill (ours)† | $640\times 192$ | 0.106 | 0.777 | 4.580 | 0.184 | 0.888 | 0.963 | 0.982
Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] (RN18) | $1024\times 320$ | 0.115 | 0.882 | 4.701 | 0.190 | 0.879 | 0.961 | 0.982
Tosi et al. [Tosi et al.(2020)Tosi, Aleotti, Ramirez, Poggi, Salti, Stefano, and Mattoccia]† | $1024\times 320$ | 0.118 | 0.748 | 4.608 | 0.186 | 0.865 | 0.961 | 0.985
PackNet-SfM [Guizilini et al.(2020a)Guizilini, Ambrus, Pillai, Raventos, and Gaidon] | $1280\times 384$ | 0.107 | 0.802 | 4.538 | 0.186 | 0.886 | 0.962 | 0.981
HR-Depth [Lyu et al.(2021)Lyu, Liu, Wang, Kong, Liu, Liu, Chen, and Yuan] | $1024\times 320$ | 0.106 | 0.755 | 4.472 | 0.181 | 0.892 | 0.966 | 0.984
Klingner et al. [Klingner et al.(2020)Klingner, Termöhlen, Mikolajczyk, and Fingscheidt]† (RN18) | $1280\times 384$ | 0.107 | 0.768 | 4.468 | 0.186 | 0.891 | 0.963 | 0.982
Shu et al. [Shu et al.(2020)Shu, Yu, Duan, and Yang] | $1024\times 320$ | 0.104 | 0.729 | 4.481 | 0.179 | 0.893 | 0.965 | 0.987
X-Distill (ours)† | $1024\times 320$ | 0.102 | 0.698 | 4.439 | 0.180 | 0.895 | 0.965 | 0.983
Table 2: Performance evaluation on KITTI Eigen split. For methods that report
performance for multiple models, we use the encoder to differentiate them
(e.g., RN18 vs. RN50). Note that two architectures can be very different even
if they use the same encoder (e.g., Monodepth2 [Godard et al.(2019)Godard, Mac
Aodha, Firman, and Brostow] vs. Johnston et al. [Johnston and
Carneiro(2020)]). For each metric, the best (second best) results are in bold
(underlined). We use † to indicate methods that utilize semantic information
during training.

####  4.2.1 Performance Evaluation

Evaluation on KITTI: Table 2 shows the evaluation results on KITTI and
comparison with the latest SOTA methods. It can be seen that our proposed
X-Distill approach performs the best for most of the metrics. When our
approach does not achieve the top-1 result, it is very close to the best
number. Fig. 2 shows sample prediction results of our proposed approach as
compared to those by Monodepth2. It can be seen that Monodepth2 can predict
inconsistent depth values on an object, which visually appear as missing parts
on the depth map (e.g., see the missing upper part of the car in Fig. 2
(middle)). On the other hand, our approach provides more accurate and
semantically more structured depth maps, thanks to its ability to better
understand the semantics of the scene. For instance, it generates more
structurally complete depth estimations for the biker in Fig. 2 (left) and for
the cars in the middle and right figures (as indicated by the green boxes).
Moreover, our approach is also able to capture the thin structures better. For
instance, in Fig. 2 (right), our model is able to generate a more clear depth
estimation over the lamp post. This is because the thin objects are grouped as
a class in the semantics-to-depth distillation, which encourages the depth
network to learn to recognize these structures.

Accuracy vs. Computation Efficiency: Fig. 3 shows the accuracy, in terms of
squared relative error, and the efficiency, in terms of GMAC (Multiply-
Accumulate Operations in $10^{9}$), of our proposed approach and the SOTA
methods on KITTI.333For Johnston et al. [Johnston and Carneiro(2020)], the
GMAC shown in Fig. 3 is a lower bound which only includes the RN101 encoder’s
computation since their self-attention and discrete disparity volume
implementation is not publicly available. It can be seen that our trained
model is able to achieve smaller depth estimation errors while using the same
or less computation. We further show the performance of applying our cross-
task distillation to an RN18-based model from [Godard et al.(2019)Godard, Mac
Aodha, Firman, and Brostow]. It can be seen that our method allows this
smaller network to achieve an accuracy similar to PackNet (which uses
20$\times$ more computation).

Figure 3: Accuracy (in squared relative error) vs. computation efficiency (in
GMAC log-scale).

Depth Estimation on Center and Surrounding Areas: Since KITTI images are
acquired with a wide-angle lens, we further evaluate the depth estimation
performance on center and surrounding areas in the image. Specifically, we
horizontally divide each image into 3 equal sections. The middle part is
considered the center area and the left and right parts are surrounding areas.
It can be seen in Table 3 that the depth estimation is much more accurate in
the center area, for both Monodepth2 and X-Distill, since surrounding areas
suffer from lens distortion/rectification artifacts. We note that for both
areas, our proposed X-Distill consistently provides more accurate depth
estimation as compared to Monodepth2.

Method | Lower is Better | Higher is Better
---|---|---
Abs Rel | Sq Rel | RMSE | $\text{RMSE}_{\text{Log}}$ | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$
Over Center Areas
Monodepth2 (RN50) | 0.061 | 0.059 | 0.568 | 0.077 | 0.978 | 0.996 | 0.999
X-Distill (ours) | 0.056 | 0.048 | 0.526 | 0.072 | 0.982 | 0.997 | 0.999
Over Surrounding Areas
Monodepth2 (RN50) | 0.135 | 1.228 | 5.700 | 0.220 | 0.839 | 0.945 | 0.974
X-Distill (ours) | 0.125 | 1.054 | 5.368 | 0.210 | 0.852 | 0.950 | 0.976
Table 3: Performance evaluation on center and surrounding image areas.

Generalizability on Make3D: We evaluate the generalizability of our KITTI-
trained model on Make3D (following the test setup in [Godard et
al.(2019)Godard, Mac Aodha, Firman, and Brostow]). It can be seen in Table 4
that our model significantly outperforms other SOTA self-supervised methods on
this dataset.

Method | Supervision | Lower is Better
---|---|---
Abs Rel | Sq Rel | RMSE | $\text{RMSE}_{\text{Log}}$
Karsch [Karsch et al.(2014)Karsch, Liu, and Kang] | GT | 0.428 | 5.079 | 8.389 | 0.149
Liu [Liu et al.(2014)Liu, Salzmann, and He] | GT | 0.475 | 6.562 | 10.05 | 0.165
Laina [Laina et al.(2016)Laina, Rupprecht, Belagiannis, Tombari, and Navab] | GT | 0.204 | 1.840 | 5.683 | 0.084
Monodepth [Godard et al.(2017)Godard, Mac Aodha, and Brostow] | S | 0.544 | 10.94 | 11.760 | 0.193
Zhou et al. [Zhou et al.(2017)Zhou, Brown, Snavely, and Lowe] | M | 0.383 | 5.321 | 10.470 | 0.478
DDVO [Wang et al.(2018)Wang, Buenaposada, Zhu, and Lucey] | M | 0.387 | 4.720 | 8.090 | 0.204
Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] (RN18) | M | 0.322 | 3.589 | 7.417 | 0.163
X-Distill (ours)† | M | 0.308 | 3.122 | 7.015 | 0.158
Table 4: Performance evaluation on Make3D. GT indicates that the method is
trained with ground-truth Make3D depth annotations, S indicates self-
supervised training using KITTI stereo data, and M indicates self-supervised
training using KITTI single-view videos. We use † to indicate methods that
utilize semantic information during training.

####  4.2.2 Ablation Studies

Grouping Semantic Classes: In addition to our proposed grouping shown in Table
1, we further test the baseline of using the original 19 Cityscapes classes
without regrouping, as well as a more aggressive grouping method that only
considers foreground and background objects. As shown in Table 5, while the
other two grouping baselines can also improve the depth estimation, the
improvements are not as large as compared to our proposed method.

Categorization Scheme | Lower is Better | Higher is Better
---|---|---
Abs Rel | Sq Rel | RMSE | $\text{RMSE}_{\text{Log}}$ | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$
Monodepth2 (RN50) | 0.110 | 0.831 | 4.642 | 0.187 | 0.883 | 0.962 | 0.982
Fore/Back-ground (2) | 0.108 | 0.798 | 4.663 | 0.187 | 0.886 | 0.962 | 0.982
Proposed scheme (4) | 0.106 | 0.777 | 4.580 | 0.184 | 0.888 | 0.963 | 0.982
Cityscapes classes (19) | 0.110 | 0.806 | 4.619 | 0.184 | 0.882 | 0.963 | 0.983
Table 5: Performance of different ways of grouping the semantic classes.

Complexity of Depth-to-Segmentation Network: As discussed in Sec. 3.2, the D2S
network should be of a proper complexity such that it does not take away the
learning from the depth network. As shown in Table 6, by using a more complex
D2S network (about 2$\\!\times$ larger), the depth network gains a smaller
improvement. We further test a baseline using a simple D2S network with one-
layer pointwise convolution. This baseline does not perform well as the
corresponding D2S network is too simple to translate depth to segmentation.

Depth-to-Segmentation | Lower is Better | Higher is Better
---|---|---
Network | Abs Rel | Sq Rel | RMSE | $\text{RMSE}_{\text{Log}}$ | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$
Monodepth2 (RN50) | 0.110 | 0.831 | 4.642 | 0.187 | 0.883 | 0.962 | 0.982
2 3$\times$3-Conv + 1 Pointwise Conv | 0.106 | 0.777 | 4.580 | 0.184 | 0.888 | 0.963 | 0.982
4 3$\times$3-Conv + 1 Pointwise Conv | 0.108 | 0.786 | 4.615 | 0.185 | 0.886 | 0.963 | 0.982
1 Pointwise Conv | 0.110 | 0.840 | 4.683 | 0.188 | 0.885 | 0.961 | 0.981
Table 6: Depth-to-Segmentation network.

Weighting Segmentation Loss: In our proposed approach, we adopt a linear
weighting schedule to combine the segmentation distillation loss with the
self-supervised depth loss. It can be seen in Table 7, the linearly scheduled
weight allows the depth network to achieve a higher depth estimation accuracy
as compared to using a constant weight. We further vary the final weight by
$\pm$20% and the results show that our proposed method is not very sensitive
to the exact value of the weight.

Weighting of | Lower is Better | Higher is Better
---|---|---
Distillation Loss | Abs Rel | Sq Rel | RMSE | $\text{RMSE}_{\text{Log}}$ | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$
Monodepth2 (RN50) | 0.110 | 0.831 | 4.642 | 0.187 | 0.883 | 0.962 | 0.982
Constant: 0.0050 | 0.109 | 0.810 | 4.637 | 0.185 | 0.887 | 0.963 | 0.983
Linear: 0 - 0.0040 | 0.107 | 0.779 | 4.632 | 0.185 | 0.887 | 0.962 | 0.982
Linear: 0 - 0.0045 | 0.107 | 0.751 | 4.553 | 0.185 | 0.884 | 0.963 | 0.983
Linear: 0 - 0.0050 | 0.106 | 0.777 | 4.580 | 0.184 | 0.888 | 0.963 | 0.982
Linear: 0 - 0.0055 | 0.108 | 0.795 | 4.606 | 0.183 | 0.887 | 0.963 | 0.983
Linear: 0 - 0.0060 | 0.107 | 0.775 | 4.580 | 0.184 | 0.888 | 0.963 | 0.983
Table 7: Weighting of segmentation loss.

Applying X-Distill to Different Architectures: We apply our proposed approach
to different depth networks, e.g., Monodepth2 [Godard et al.(2019)Godard, Mac
Aodha, Firman, and Brostow] with different encoders and HR-Depth [Lyu et
al.(2021)Lyu, Liu, Wang, Kong, Liu, Liu, Chen, and Yuan]. Specifically, for
the encoder of Monodepth2, in addition to RN18 and RN50 that are used in the
original paper, we also employ a recent backbone, DONNA, which is optimized
for mobile processors via neural architecture search [Moons et al.(2021)Moons,
Noorzad, Skliar, Mariani, Mehta, Lott, and Blankevoort]. This will demonstrate
the efficacy of our method for practical mobile use cases. As can be seen in
Table 8, our proposed X-Distill considerably improves the depth estimation
accuracy for all these different depth networks.

Architectures | Lower is Better | Higher is Better
---|---|---
Abs Rel | Sq Rel | RMSE | $\text{RMSE}_{\text{Log}}$ | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$
Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] (RN18) | 0.115 | 0.903 | 4.863 | 0.193 | 0.877 | 0.959 | 0.981
\+ X-Distill | 0.111 | 0.791 | 4.772 | 0.188 | 0.874 | 0.960 | 0.983
Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] (RN50) | 0.110 | 0.831 | 4.642 | 0.187 | 0.883 | 0.962 | 0.982
\+ X-Distill | 0.106 | 0.777 | 4.580 | 0.184 | 0.888 | 0.963 | 0.982
Monodepth2 [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] (DONNA) | 0.115 | 0.916 | 4.827 | 0.193 | 0.879 | 0.960 | 0.981
\+ X-Distill | 0.109 | 0.772 | 4.678 | 0.188 | 0.884 | 0.962 | 0.982
HR-Depth [Lyu et al.(2021)Lyu, Liu, Wang, Kong, Liu, Liu, Chen, and Yuan] | 0.109 | 0.792 | 4.632 | 0.185 | 0.884 | 0.962 | 0.983
\+ X-Distill | 0.108 | 0.755 | 4.579 | 0.184 | 0.884 | 0.963 | 0.983
Table 8: Applying our semantics-to-depth distillation to different depth
networks. For each model, improved numbers by using X-Distill are highlighted
in bold.

##  5 Conclusions

In this paper, we presented a novel cross-task distillation approach,
X-Distill, to improve the self-supervised training of monocular depth by
transferring semantic knowledge from a teacher segmentation network to the
depth network. In order to enable such cross-task distillation, we utilized a
small, trainable network that translates the predicted depth map to a semantic
segmentation map, which the semantic teacher network can then supervise. This
enables the backpropagation from the semantic teacher’s supervision to the
depth network during training. We further studied the visual and geometric
characteristics of the objects and designed a new way of grouping them that
can be shared by both tasks. We evaluated our proposed approach on KITTI and
Make3D, and conducted extensive ablation studies. The results show that by
training with the proposed cross-task distillation, we can significantly
improve the depth estimation accuracy and outperform the state of the art
without incurring additional computation during inference.

## References

  * [Borse et al.(2021)Borse, Wang, Zhang, and Porikli] Shubhankar Borse, Ying Wang, Yizhe Zhang, and Fatih Porikli.  Inverseform: A loss function for structured boundary-aware segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * [Casser et al.(2019)Casser, Pirk, Mahjourian, and Angelova] Vincent Casser, Soeren Pirk, Reza Mahjourian, and Anelia Angelova.  Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2019.
  * [Chen et al.(2019a)Chen, Liu, Liu, and Wang] Po-Yi Chen, Alexander H Liu, Yen-Cheng Liu, and Yu-Chiang Frank Wang.  Towards scene understanding: Unsupervised monocular depth estimation with semantic-aware representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2624–2632, 2019a.
  * [Chen et al.(2019b)Chen, Schmid, and Sminchisescu] Yuhua Chen, Cordelia Schmid, and Cristian Sminchisescu.  Self-supervised learning with geometric constraints in monocular video: Connecting flow, depth, and camera.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7063–7072, 2019b.
  * [Cordts et al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson, Franke, Roth, and Schiele] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016.
  * [Dai et al.(2020)Dai, Patil, Hecker, Dai, Van Gool, and Schindler] Qi Dai, Vaishakh Patil, Simon Hecker, Dengxin Dai, Luc Van Gool, and Konrad Schindler.  Self-supervised object motion and depth estimation from video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 1004–1005, 2020.
  * [Eigen and Fergus(2015)] David Eigen and Rob Fergus.  Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2650–2658, 2015.
  * [Geiger et al.(2013)Geiger, Lenz, Stiller, and Urtasun] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun.  Vision meets robotics: The KITTI dataset.  _The International Journal of Robotics Research_ , 32(11):1231–1237, 2013.
  * [Godard et al.(2019)Godard, Mac Aodha, Firman, and Brostow] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow.  Digging into self-supervised monocular depth estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019.
  * [Godard et al.(2017)Godard, Mac Aodha, and Brostow] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 270–279, 2017.
  * [Gordon et al.(2019)Gordon, Li, Jonschkowski, and Angelova] Ariel Gordon, Hanhan Li, Rico Jonschkowski, and Anelia Angelova.  Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8977–8986, 2019.
  * [Gou et al.(2021)Gou, Yu, Maybank, and Tao] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao.  Knowledge distillation: A survey.  _International Journal of Computer Vision_ , pages 1–31, 2021.
  * [Guizilini et al.(2020a)Guizilini, Ambrus, Pillai, Raventos, and Gaidon] V. Guizilini, R. Ambrus, S. Pillai, A. Raventos, and A. Gaidon.  3D packing for self-supervised monocular depth estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020a.
  * [Guizilini et al.(2020b)Guizilini, Hou, Li, Ambrus, and Gaidon] V. Guizilini, R. Hou, J. Li, R. Ambrus, and A. Gaidon.  Semantically-guided representation learning for self-supervised monocular depth.  In _Proceedings of the International Conference on Learning Representations_ , 2020b.
  * [Jiang et al.(2020)Jiang, Ding, Sun, and Huang] Hualie Jiang, Laiyan Ding, Zhenglong Sun, and Rui Huang.  Dipe: Deeper into photometric errors for unsupervised learning of depth and ego-motion from monocular videos.  In _Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 10061–10067, 2020.
  * [Johnston and Carneiro(2020)] A. Johnston and G. Carneiro.  Self-supervised monocular trained depth estimation using self-attention and discrete disparity volume.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [Karsch et al.(2014)Karsch, Liu, and Kang] Kevin Karsch, Ce Liu, and Sing Bing Kang.  Depth transfer: Depth extraction from video using non-parametric sampling.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 36(11):2144–2158, 2014.
  * [Klingner et al.(2020)Klingner, Termöhlen, Mikolajczyk, and Fingscheidt] M. Klingner, J.-A. Termöhlen, J. Mikolajczyk, and T. Fingscheidt.  Self-supervised monocular depth estimation: Solving the dynamic object problem by semantic guidance.  In _Proceedings of the European Conference on Computer Vision_ , 2020\.
  * [Kumar et al.(2021)Kumar, Klingner, Yogamani, Milz, Fingscheidt, and Mader] Varun Ravi Kumar, Marvin Klingner, Senthil Yogamani, Stefan Milz, Tim Fingscheidt, and Patrick Mader.  SynDistNet: Self-supervised monocular fisheye camera distance estimation synergized with semantic segmentation for autonomous driving.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021.
  * [Laina et al.(2016)Laina, Rupprecht, Belagiannis, Tombari, and Navab] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In _Proceedings of the International Conference on 3D Vision_ , pages 239–248, 2016.
  * [Liu et al.(2014)Liu, Salzmann, and He] Miaomiao Liu, Mathieu Salzmann, and Xuming He.  Discrete-continuous depth estimation from a single image.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 716–723, 2014.
  * [Luo et al.(2019)Luo, Yang, Wang, Wang, Xu, Nevatia, and Yuille] Chenxu Luo, Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia, and Alan Yuille.  Every pixel counts++: Joint learning of geometry and motion with 3D holistic understanding.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 42(10):2624–2641, 2019.
  * [Lyu et al.(2021)Lyu, Liu, Wang, Kong, Liu, Liu, Chen, and Yuan] X. Lyu, L. Liu, M. Wang, X. Kong, L.a Liu, Y. Liu, X. Chen, and Y. Yuan.  HR-Depth: High resolution self-supervised monocular depth estimation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021.
  * [Meng et al.(2019)Meng, Lu, Raj, Sunarjo, Guo, Javidi, Bansal, and Bharadia] Yue Meng, Yongxi Lu, Aman Raj, Samuel Sunarjo, Rui Guo, Tara Javidi, Gaurav Bansal, and Dinesh Bharadia.  Signet: Semantic instance aided unsupervised 3D geometry perception.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [Ming et al.(2021)Ming, Meng, Fan, and Yu] Yue Ming, Xuyang Meng, Chunxiao Fan, and Hui Yu.  Deep learning for monocular depth estimation: A review.  _Neurocomputing_ , 2021.
  * [Moons et al.(2021)Moons, Noorzad, Skliar, Mariani, Mehta, Lott, and Blankevoort] Bert Moons, Parham Noorzad, Andrii Skliar, Giovanni Mariani, Dushyant Mehta, Chris Lott, and Tijmen Blankevoort.  Distilling optimal neural networks: Rapid search in diverse spaces.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2021.
  * [Patil et al.(2020)Patil, Van Gansbeke, Dai, and Van Gool] Vaishakh Patil, Wouter Van Gansbeke, Dengxin Dai, and Luc Van Gool.  Don’t forget the past: Recurrent depth estimation from monocular video.  _IEEE Robotics and Automation Letters_ , 5(4):6813–6820, 2020.
  * [Ramirez et al.(2018)Ramirez, Poggi, Tosi, Mattoccia, and Di Stefano] Pierluigi Zama Ramirez, Matteo Poggi, Fabio Tosi, Stefano Mattoccia, and Luigi Di Stefano.  Geometry meets semantics for semi-supervised monocular depth estimation.  In _Asian Conference on Computer Vision_ , pages 298–313. Springer, 2018.
  * [Ranjan et al.(2019)Ranjan, Jampani, Balles, Kim, Sun, Wulff, and Black] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black.  Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12240–12249, 2019.
  * [Saeedan and Roth(2021)] Faraz Saeedan and Stefan Roth.  Boosting monocular depth with panoptic segmentation maps.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 3853–3862, 2021.
  * [Saxena et al.(2005)Saxena, Chung, and Ng] Ashutosh Saxena, Sung H Chung, and Andrew Y Ng.  Learning depth from single monocular images.  In _Advances on Neural Information Processing Systems_ , pages 1161–1168, 2005.
  * [Saxena et al.(2008)Saxena, Sun, and Ng] Ashutosh Saxena, Min Sun, and Andrew Y Ng.  Make3D: Learning 3D scene structure from a single still image.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 31(5):824–840, 2008.
  * [Shu et al.(2020)Shu, Yu, Duan, and Yang] C. Shu, K. Yu, Z. Duan, and K. Yang.  Feature-metric loss for self-supervised learning of depth and egomotion.  In _Proceedings of the European Conference on Computer Vision_ , 2020\.
  * [Sun et al.(2021)Sun, Ye, Li, Li, Wang, and Xu] Baoli Sun, Xinchen Ye, Baopu Li, Haojie Li, Zhihui Wang, and Rui Xu.  Learning scene structure guidance via cross-task knowledge transfer for single depth super-resolution.  _arXiv preprint arXiv:2103.12955_ , 2021.
  * [Tosi et al.(2020)Tosi, Aleotti, Ramirez, Poggi, Salti, Stefano, and Mattoccia] Fabio Tosi, Filippo Aleotti, Pierluigi Zama Ramirez, Matteo Poggi, Samuele Salti, Luigi Di Stefano, and Stefano Mattoccia.  Distilled semantics for comprehensive scene understanding from videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [Wang et al.(2018)Wang, Buenaposada, Zhu, and Lucey] Chaoyang Wang, José Miguel Buenaposada, Rui Zhu, and Simon Lucey.  Learning depth from monocular videos using direct methods.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2022–2030, 2018.
  * [Wang et al.(2020)Wang, Sun, Cheng, Jiang, Deng, Zhao, Liu, Mu, Tan, Wang, Liu, and Xiao] J. Wang, K. Sun, T. Cheng, B.i Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang, W. Liu, and B. Xiao.  Deep high-resolution representation learning for visual recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [Wang et al.(2004)Wang, Bovik, Sheikh, and Simoncelli] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: From error visibility to structural similarity.  _IEEE Transactions on Image Processing_ , 13(4):600–612, 2004.
  * [Yang et al.(2018)Yang, Zhao, Shi, Deng, and Jia] Guorun Yang, Hengshuang Zhao, Jianping Shi, Zhidong Deng, and Jiaya Jia.  Segstereo: Exploiting semantic information for disparity estimation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 636–651, 2018.
  * [Ye et al.(2020)Ye, Lu, and Zhan] Han-Jia Ye, Su Lu, and De-Chuan Zhan.  Distilling cross-task knowledge via relationship matching.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12396–12405, 2020.
  * [Yin and Shi(2018)] Zhichao Yin and Jianping Shi.  Geonet: Unsupervised learning of dense depth, optical flow and camera pose.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [Yuan and Peng(2019)] Mingkuan Yuan and Yuxin Peng.  Ckd: Cross-task knowledge distillation for text-to-image synthesis.  _IEEE Transactions on Multimedia_ , 22(8):1955–1968, 2019.
  * [Yuan et al.(2020)Yuan, Chen, and Wang] Yuhui Yuan, Xilin Chen, and Jingdong Wang.  Object-contextual representations for semantic segmentation.  In _Proceedings of the European Computer Vision Conference_ , 2020\.
  * [Yue et al.(2020)Yue, Fu, Wu, and Wang] Min Yue, Guangyuan Fu, Ming Wu, and Hongqiao Wang.  Semi-supervised monocular depth estimation based on semantic supervision.  _Journal of Intelligent & Robotic Systems_, 100:455–463, 2020.
  * [Zhou et al.(2017)Zhou, Brown, Snavely, and Lowe] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe.  Unsupervised learning of depth and ego-motion from video.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * [Zhu et al.(2020)Zhu, Brazil, and Liu] Shengjie Zhu, Garrick Brazil, and Xiaoming Liu.  The edge of depth: Explicit constraints between segmentation and depth.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13116–13125, 2020.


# A $1$-D Special Relativistic
Shock Tube where Stellar Fluid
undergoes Neutrino Heating

Gregory Mohammed

Department of Physics
University of British Columbia - Okanagan Campus
Kelowna, BC. Canada V1V 1V7
<EMAIL_ADDRESS>
The simulation code in Java

(October 27, 2023)

###### Abstract

To investigate the physical nature of neutrino-heating on the result of a
1-dimensional core-collapse supernova. (Colgate, 1966) were the first to
suggest that neutrinos may play a crucial role in core collapse supernova by
taking up gravitational binding energy from the core and depositing it in the
rest of the star. The fluid is contained in a shock tube in a region extending
from the neutrino-sphere of a star out $1000$ km, and irradiated with a
neutrino flux emanating from the surface of the neutrino-sphere into the shock
tube. Interaction due to electron-neutrino scattering off of fluid nucleons is
used to account for neutrino heating of the fluid. The numerical method used
is the Godunov method implementing an exact Riemann solver (Rezzolla, 2001)
when needed. The simulated core-collapse produces an energy output of up to
$10^{36}$ J on a timescale of $3.33$ s. It was found that the Godunov method
performs very well using (Sod, 1978) data, with standard deviations of between
$0.22-0.04$ %. Thus, there is high confidence that the exact Riemann solver is
working correctly and that the Godunov method is also reproducing the exact
solutions to the Sod data with high confidence. A runtime of $3.33$ s was
used. The objective was to determine what would happen to the shock system
over longer timescales, but the simulations showed an explosion in a much
shorter time frame. It was found that the shock not only continues to move
outward, but is also driven by large energy and pressure behind the shock. The
density profiles show that fluid mass is pushed outward, but the fluid
velocity actually reversed on timescales close to $3.33$ s. In this paper, net
mass movement occurs at $500-600$ km and on a longer timescale of $3.33$ s
mass has moved out to $700$ km. This supports the _delayed post bounce shock
mechanism_.

_Keywords_ core-collapse supernova – neutrino heating in supernova – delayed
post bounce shock mechanism – prompt post bounce shock mechanism

##  1 Introduction

The discovery of weak neutral currents and the corresponding importance of
neutrino scattering off nucleons lead to the realization that the forming
neutron star is highly opaque to neutrinos. Thus, the neutrino luminosities
were too low and the energy transfer rate was not large enough to invert the
infall of the surrounding gas into an explosion (Janka, 2001). One approach to
understanding the role of the neutrinos and the particulars of the mechanisms
of the energy transfers is to consider neutrino cross-sections for interaction
with the stellar fluid. (Tubbs, 1975) and (Bowers, 1982) are excellent
resources for the neutrino cross-sections of the three flavors of neutrino
type, and for the development of the particulars of the neutrino interactions.

For a number of years efforts were concentrated on the _prompt post bounce
shock mechanism_ , which is the process whereby the energy given to the
hydrodynamical shockwave in the moment of core bounce was thought to lead
directly to the ejection of the stellar mantle and envelope (Janka, 2001).
More realistic models showed that, due to severe energy loss experienced by
the shock, the shock’s outward propagation stops well inside the iron core
(Bruenn, 2001).

This paper’s model is based on the work done by (Kuroda, 2012), and also used
restricted versions of the equations provided by (Matteo, 2002), (Liu, 2007)
and (Zhang, 2009). The simulation is that of the behavior of stellar fluid
undergoing neutrino heating in a shock tube. The model is planar, using the
usual special relativistic hydrodynamics equations.

Only electron-neutrinos are considered, since in gravitational collapse
problems the electron-neutrino emission outnumber the production of other
types by the dominance of $e^{{}-}+e^{{}+}\rightarrow\nu_{e}+\bar{\nu}_{e}$
(Tubbs, 1975). The numerical method used is the Godunov Finite Volume Method,
utilizing an exact Riemann solver on the nodes (boundaries of cells). The
exact Riemann solver is a relativistic one, written by (Rezzolla, 2001), which
was an improvement on work done by (Pons, 1999).

The physical model is that of stellar fluid undergoing neutrino heating within
a 1-dimensional shock tube placed against the surface of the neutrino-sphere.
The tube extends $1000$ km out from this location. The coordinate system is
Cartesian. The isolation of the system from its surroundings make it an
excellent test tube. The results of (Kuroda, 2012) were used to set up initial
conditions for the simulation when the neutrino model was activated and runs
done for each of three cases: the fluid under heating provided by only a
neutrino energy flux, the fluid under heating provided by only a neutrino
momentum flux and the fluid under heating provided by both neutrino energy
flux and neutrino momentum flux.

A runtime of $3.33$ s was used. This is well outside the $0.1$ s quoted by
(Janka, 2001). The objective was to determine what would happen to the shock
system over longer timescales, but the simulations showed an explosion in a
much shorter time frame. It was found that the shock not only continues to
move outward, but is also driven by large energy and pressure behind the
shock. The density profiles show that fluid mass is pushed outward, but the
fluid velocity actually reversed on timescales of $3.33$ s. According to
(Bruenn, 2001), shocks stall at $100-200$ km. In this paper, such a stall was
observed; in fact, net mass movement occurs at $500-600$ km and on a longer
timescale of $3.33$ s mass has moved out to $700$ km. This supports the
delayed post bounce shock mechanism.

The Godunov method was thoroughly tested against this solver using shocks
moving left to right and right to left, and also at velocities close to the
speed of light. It was found that the Godunov method performs very well using
Sod data, with standard deviations of between $0.22-0.04$ %. Thus, there is
high confidence that the exact Riemann solver is working correctly and that
the Godunov method is also reproducing the exact solutions to the Sod data
with high confidence.

It is worthwhile to note that this trim model and code has reproduced the
findings of much more complicated 3-dimensional codes running much more
realistic neutrino equations of state and detailed microphysics. This paper
has confirmed that on longer timescales there is no stalling of the shock
system. Instead there is an explosion which reach energies in the $10^{36}$ J
range.

##  2 1-D Special Relativistic Neutrino Hydrodynamics

(Kuroda, 2012) begin with the conservation of total energy-momentum, comprised
of the fluid and neutrino tensors.

$T^{\alpha\beta}=T^{\alpha\beta}_{(fluid)}+T^{\alpha\beta}_{(\nu)}$ (1)

Conservation of energy-momentum gives:

$\nabla_{\alpha}T^{\alpha\beta}_{(total)}=\nabla_{\alpha}T^{\alpha\beta}_{(fluid)}+\nabla_{\alpha}T^{\alpha\beta}_{(\nu)}=0$
(2)

This implied that Equation (2) can be written as,

$\nabla_{\alpha}T^{\alpha\beta}_{(fluid)}=-Q^{\beta}$ (3)

$\nabla_{\alpha}T^{\alpha\beta}_{(\nu)}=Q^{\beta}$ (4)

where $Q^{\beta}$ represents the source terms describing the exchange of
energy and momentum between the fluid and the neutrino radiation.

The contribution of neutrino mass to the fluid is ignored. Similarly the fluid
does not create neutrinos and therefore the fluid does not lose mass. These
approximations are reasonable in the outer regions of a supernova where
neutrinos are free-streaming.

##  3 The Neutrino Model

The neutrino-sphere is the region where $\tau_{(\nu)}\geq\frac{2}{3}$. Using
the graphs in the work done by (Kuroda, 2012), it was determined that the best
position to place the Riemann shock tube for the purposes of this paper was on
the surface of the neutrino-sphere. This is the only location which, according
to (Kuroda, 2012) data, would capture a shock in the arbitrarily chosen time
of $40$ ms.

The time of $40$ ms was chosen as it was the longest interval for which good
data from (Kuroda, 2012) graphs could be extracted. The neutrino-sphere’s
surface corresponds to $x=800$ km. The length of the shock tube is $1.8\times
10^{2}$ km. The data of (Kuroda, 2012) is used to obtain the values for the
velocity profile (Figure 3), the density profile (Figure 2) and the neutrino
specific energy (Figure 1) at time = $10$ ms to time = $40$ ms. The time of
$40$ ms was chosen because it intersects the energy profile for a
1-dimensional special relativistic simulation at an energy of $15$ MeV which,
according to (Burrows, 2002) is in the range of energies for electron-
neutrinos in the core collapse. Also, the velocity profile only extends to
$37$ ms so no greater time data could be utilized.

Knowing the time constraint and the location of the neutrino-sphere as
detailed in (Kuroda, 2012) then using the velocity profile allowed the
determination of the length of the shock tube and the location of the initial
shock (as midway between the neutrino-sphere and the location at a time of
$37$ ms).

These considerations can be viewed graphically, as above, by using the work of
(Kuroda, 2012) and imposing the beginning of the surface of the neutrino-
sphere at $10^{5}$ m from the center of the star.

Figure 1: Electron-neutrino energy at $40$ ms. Figure 2: Electron-neutrino
density at the surface of the neutrino-sphere. Figure 3: Electron-neutrino
velocity at the surface of the neutrino-sphere.

The 1-dimensional versions of (Kuroda, 2012) results, with
$W=\frac{1}{\sqrt{1-v^{2}}}$, are;

$Q^{t}=e^{-\beta_{\nu}\tau_{\nu}}\left(\epsilon_{s_{\nu}}\right)^{2}\tilde{\kappa}\left(WE_{(\nu)}-F_{(\nu)}v\right)$
(5)

$-Q^{x}=e^{-\beta_{\nu}\tau_{\nu}}\left(\epsilon_{s_{\nu}}\right)^{2}\tilde{\kappa}\left(-WF_{(\nu)}+P_{(\nu)}v\right)$
(6)

where $E_{(\nu)},F_{(\nu)}$ and $P_{(\nu)}$ are the radiation energy, flux and
pressure. Equations (5, 6) follow directly from the work of (Kuroda, 2012),
where they split the source term, $Q^{\mu}$ as:

$Q^{\mu}\equiv Q^{\mu,C}-Q^{\mu,H},$ (7)

where “C” implies a cooling term and “H” implies a heating term. This paper is
only concerned with the heating term, and neglects any cooling which may
occur. This is a constraint on this paper’s model.

The heating term is written, by (Kuroda, 2012), as:

$Q^{\mu,H}=\sum_{\nu\in\nu_{e},\overline{\nu}_{e}}e^{-\beta_{\nu}\tau_{\nu}}\left(\epsilon_{s_{\nu}}\right)^{2}\tilde{\kappa}_{\nu}\left(-\mathcal{J}_{\nu}u^{\mu}-\mathcal{H}_{\nu}^{\mu}\right).$
(8)

and, also from (Kuroda, 2012), where $\mathcal{J,H}$ denote the Eddington
moments in the comoving frame:

$\mathcal{J}=E_{(\nu)}W^{2}-2WF_{(\nu)}^{i}u_{i}+P_{(\nu)}^{ij}u_{i}u_{j},$
(9)

$\mathcal{H}^{\alpha}=\left(E_{(\nu)}W-F_{(\nu)}^{i}u_{i}\right)\left(n^{\alpha}-Wu^{\alpha}\right)+Wh_{\beta}^{\alpha}F_{(\nu)}^{\beta}-h_{i}^{\alpha}u_{j}P_{(\nu)}^{ij},$
(10)

where the Lorentz factor is $W=\alpha u^{0}$ and $h_{\alpha\beta}\equiv
g_{\alpha\beta}+u_{\alpha}u_{\beta}$.

Here, $e^{-\beta_{\nu}\tau_{\nu}}$ is taken as $1$, since there is no need to
progress smoothly from one region to the next. This paper deals only with the
“atmosphere” outside the neutrino-sphere, and simplifies that atmosphere to a
region which is optically thin to electron-neutrinos. This means that the
neutrino stream is decoupled from the stellar fluid, and so does not adopt the
fluid temperature. The neutrino cross section $\sigma_{0}$ means that the
neutrino stream presents some opacity to the fluid, and so there can be some
neutrino heating.

$\kappa_{sc}$ is defined to be
$\kappa_{sc}\equiv\left(\epsilon_{s_{\nu}}\right)^{2}\tilde{\kappa}$ which
simplifies the equations needed to be evolved in order to compute
$E_{(\nu)},F_{(\nu)}$ and therefore $P_{(\nu)}$. So,

$Q^{t}=\kappa_{sc}\left(WE_{(\nu)}-F_{(\nu)}v\right)$ (11)

$-Q^{x}=\kappa_{sc}\left(-WF_{(\nu)}+P_{(\nu)}v\right)$ (12)

$\kappa_{sc}=1.69\times 10^{4}\rho_{0}$. Thus, using (Janka, 2001) equations
and relevant quantity values,

$Q^{t}=1.69\times 10^{4}\rho_{0}\left(WE_{(\nu)}-F_{(\nu)}v\right)$ (13)

$-Q^{x}=1.69\times 10^{4}\rho_{0}\left(-WF_{(\nu)}+P_{(\nu)}v\right)$ (14)

To implement the evolution of $E_{(\nu)},F_{(\nu)}$ and $P_{(\nu)}$ is not
trivial, and would involve just as much work as the implementation of the
fluid evolution. For the purposes of this paper, a simpler model has been
developed using the works of (Matteo, 2002), (Liu, 2007) and (Zhang, 2009).
These works are concerned with gamma-ray bursts, which utilize the same
neutrino physics in core-collapse supernova. As such, these works have been
adapted to produce the neutrino model for the neutrino heating in this paper.

The primary goal of the neutrino model is an estimation of the neutrino flux.
This paper uses the approaches developed by (Matteo, 2002), (Liu, 2007) and
(Zhang, 2009). The neutrino pressure is given by,

$P_{(\nu)}=\frac{u_{(\nu)}}{3}$ (15)

The neutrino energy density, $E_{(\nu)}=u_{(\nu)}$, is given by (Matteo,
2002):

$E_{(\nu)}=\frac{\frac{7}{8aT_{(\nu)}^{4}}\left[\frac{\tau_{(\nu)}}{2}+\frac{1}{\sqrt{3}}\right]}{\frac{\tau_{(\nu)}}{2}+\frac{1}{\sqrt{3}}+\frac{1}{3\tau_{(\nu)}}}$
(16)

With the optical depth defined to be $\tau_{(\nu)}=\frac{2}{3}$ at the surface
of the neutrino-sphere:

$E_{(\nu)}=\frac{aT_{(\nu)}^{4}\left[7+7\sqrt{3}\right]}{20+8\sqrt{3}}$ (17)

where $a$ is the radiation constant and $\sigma$ is the Stefan-Boltzmann
constant. The neutrino flux, $F_{(\nu)}$ is given by (Matteo, 2002):

$F_{(\nu)}=\frac{\frac{7}{8}\sigma
T_{(\nu)}^{4}}{\frac{3}{4}\left(\frac{\tau_{(\nu)}}{2}+\frac{1}{\sqrt{3}}+\frac{1}{3\tau_{(\nu)}}\right)}$
(18)

We evaluate the neutrino flux at $\tau_{(\nu)}=\frac{2}{3}$ since this is
where the neutrinos are decoupled from the stellar fluid:

$F_{(\nu)}=\frac{7\sigma T_{(\nu)}^{4}}{5+2\sqrt{3}}$ (19)

The temperature of the neutrinos, $T_{(\nu)}$, is defined by,

$T_{(\nu)}=\varepsilon_{(\nu)}/k_{B}$ (20)

since the neutrinos are decoupled (this paper assumes that there is total
decoupling immediately at $\tau_{\nu}<\frac{2}{3}$) from the stellar fluid
outside the neutrino-sphere the temperature is constant. Using the values
found earlier, then $T_{(\nu)}=1.74\times 10^{11}$ K.

The relevant equations which need to be evolved and used in the calculations
at each time step are summarized below. The fluid equations are given with the
associated source terms.

$D_{,t}+(Dv^{x})_{,x}=0$ (21)

$E_{,t}+\left((E+p)v^{x}\right)_{,x}=-Q^{t}$ (22)

$S^{x}_{,t}+\left(S^{x}v^{x}\right)_{,x}=-Q^{x}$ (23)

The source terms are, in geometrized units, and $W=\frac{1}{\sqrt{1-v^{2}}}$,

$Q^{t}=1.69\times 10^{4}\rho_{0}\left(WE_{(\nu)}-F_{(\nu)}v\right)$ (24)

$-Q^{x}=1.69\times 10^{4}\rho_{0}\left(-WF_{(\nu)}+P_{(\nu)}v\right)$ (25)

The quantities $P_{(\nu)},E_{(\nu)}$ and $F_{(\nu)}$ are given by the
equations,

$E_{(\nu)}=\frac{aT_{(\nu)}^{4}\left[7+7\sqrt{3}\right]}{20+8\sqrt{3}}$ (26)

$P_{(\nu)}=\frac{E_{(\nu)}}{3}$ (27)

$F_{(\nu)}=\frac{7\sigma T_{(\nu)}^{4}}{5+2\sqrt{3}}$ (28)

$T_{(\nu)}=\frac{\varepsilon_{(\nu)}}{k_{B}}$ (29)

This is the complete set of fluid and neutrino equations implemented in the
code in order to simulate a 1-D special relativistic core collapse supernova.

##  4 Simulation Results and Analysis

The work here presented approached the model in three steps. First, only the
neutrino energy flux was applied, then only the neutrino momentum flux and
finally both fluxes as would be the real case. With only the neutrino energy
flux applied, more mass was moved out to the surface of the collapsing star,
indicating an impending explosion with some mass imploding onto the neutrino-
sphere. When only the neutrino momentum flux was applied, it was observed that
mass was driven both outward and inward with a large rarefaction between the
two. This indicated a large amount of interaction between the electron-
neutrinos and the stellar material.

When both neutrino fluxes were activated, a reversal of the fluid pressure
occurred at the left edge of the pressure profile, while at the right edge the
pressure is still very high. This indicated a settling of material onto the
neutrino-sphere and an explosive effect at the outgoing wave to the right. So
there is both an implosion and an explosion. This is shown in (Figures 4, 5 &
6).

When both fluxes were used, the results showed that the effects from the
neutrino momentum flux summed with the effects of neutrino energy flux,
resulting in greater fluid pressure and density. This is expected because the
source term in the energy-momentum tensor for the neutrinos was split as the
sum of the source provided by the neutrino momentum and the source provided by
the neutrino energy.

The result of a mixture of implosion and explosion when both the neutrino
fluxes were applied and the simulation run for $3.33$ s is intuitively
expected. In this scenario, the fluid pressure reverses but the outer shock is
still accelerating outward. The inner shock moves inward very little, and a
pressure spike is observed. It is conceivable that neutrinos within the
neutrino-sphere can be reheated and another burst would be produced into the
shock tube, thereby bolstering the outgoing shock and powering the supernova
explosion.

Figure 4: Rest density data with both neutrino fluxes on at times of $2.5$ s &
$3.33$ s. Figure 5: Pressure data with both neutrino fluxes on at times of
$2.5$ s & $3.33$ s. Figure 6: Internal energy data with both neutrino fluxes
on at times of $2.5$ s & $3.33$ s. Figure 7: Velocity data with both neutrino
fluxes on at times of $2.5$ s & $3.33$ s.

##  5 Conclusions

This paper supports the _delayed post bounce shock mechanism_ and employs a
number of ad-hoc terms. The model is a toy model, being in 1-dimension and not
taking into account a $\frac{1}{R^{2}}$ term even though the shock tube is
very long, extending from the surface of the neutrino-sphere at 8000km to a
region in the “atmosphere” at 100000km. The other assumption is that outside
of the neutrino-sphere, it can be assumed that the optical depth drops to $0$
and remains at $0$ out to the actual atmosphere of the dying star.

This eliminates a e-τ (where $\tau$ is the optical depth) term in the source
term, where the source is the neutrino stream produced from a hot spot in the
left side of the shock tube. That source term involves two assumptions. One is
that the neutrino momentum flux produced at the hot spot is that value
throughout the length of the shock tube, that is, a constant. The second is
that the neutrino energy flux is also a constant along the shock tube, due to
decoupling of the neutrino stream from the stellar fluid outside of the
neutrino-sphere. This made that source term very easy to implement. These
assumptions were done in special relativity.

The optical depth needs to be considered carefully, with a full integral of
d$\tau=\kappa\rho_{0}$ dx carried out to find $\tau$ along the shock tube.
This can then be used to determine e-τ which would scale the neutrino energy
and neutrino flux properly along the shock tube. Finally, the tube itself
needs to be changed to a frustum, to better model the spherical symmetry in a
1-dimensional model, which would introduce a $\frac{1}{R^{2}}$ term.

The neutrino momentum and energy fluxes need to be calculated at each time
slice using (Shibata, 2011) evolution equations for each of these fluxes. This
would accurately model the neutrino fluxes along the shock tube, instead of
assuming them to be constant once produced at the neutrino-sphere hot spot.
The work of (Kuroda, 2012) implement these equations. It would be interesting
to determine how closely this thesis’ results agree with evolved neutrino
fluxes. One effect that is known to be resolved in this case is that of the
spike at the inner shock. This occurs because of the inclusion in the fluid
evolution of a constant neutrino flux. The neutrino evolution equations will
correctly control this value, and lead to more stable results.

Realistic neutrino and fluid equations of state need to be used. This thesis
used the ideal gas equation of state, both for the fluid and neutrino gas. The
work of (Janka, 2001) makes it clear that more realistic equations of state
produce better neutrino evolution results. This may lead to higher explosion
energies; it is not clear if this may actually be the case, but integrating
the more realistic equations of state into this paper’s code will be a good
test ground.

## References

  * Rezzolla (2001) Rezzolla, L.,& Zanotti, O. 2001, An Improved Exact Riemann Solver for Relativistic Hydrodynamics (arXiv:gr-qc/0103005v2)
  * Sod (1978) Sod, G. A. 1978, A Survey of Several Finite Difference Methods for Systems of Nonlinear Hyperbolic Conservation Laws (Journal of Computational Physics) 27, 1–31.
  * Pons (1999) Pons, J. A., Martí, J. M.,& Müller, E. 1999, An Exact Riemann Solver for Multidimensional Special Relativistic Hydrodynamics (arXiv:astro-ph/9910462v1)
  * Leveque (2002) Leveque, R. J. 2002, Finite Volume Methods for Hyperbolic Problems (Cambridge University Press, New York)
  * Matzner (1999) Matzner, C. D.,& McKee, C. F. 1999, The Expulsion of Stellar Envelopes in Core-Collapse Supernovae (Astrophysics Journal) 510, 379
  * Wilson (2003) Wilson, J. R.,& Mathews, G. J. 2003, Relativistic Numerical Hydrodynamics (Cambridge University Press, Cambridge, United Kingdom)
  * May (1967) May, M.,& White, R. H. 1967, Stellar Dynamics and Gravitational Collapse (Methods Comput. Phys.) 73, 219
  * Bowers (1982) Bowers, R. L.,& Wilson, J. R. 1982, A Numerical Model for Stellar Core Collapse Calculations (Astrophysical Journal Supplement Series) 50, 115–159
  * Tubbs (1975) Tubbs, D. L.,& Schramm, D. N. 1975, Neutrino Opacities at High Temperatures and Densities (The Astrophysical Journal) 201, 467–488
  * Kuroda (2012) Kuroda, T., Kotake, K.,& Takiwaki, T. 2012, Fully General Relativistic Simulations of Core-Collapse Supernovae with An Approximate Neutrino Transport (arXiv:astro-ph/1202.2487v1)
  * Burrows (2002) Burrows, A.,& Thompson, T. A. 2002, Neutrino-Matter Interaction Rates in Supernovae (arXiv:astro-ph/0211404v1)
  * Shibata (2011) Shibata, M., Kiuchi, K., Sekiguchi, Y.,& Suwa, Y. 2011, Truncated Moment Formalism for Radiation Hydrodynamics in Numerical Relativity (Progress of Theoretical Physics) 125, 1255–1287
  * Matteo (2002) Di Matteo, T., Perna, R.,& Narayan, R. 2002, Neutrino Trapping and Accretion Models for Gamma-Ray Bursts (The Astrophysical Journal) 579, 706–715
  * Liu (2007) Liu, T., Gu, W., Xue, L.,& Lu, J. 2007, Structure and Luminosity of Neutrino-cooled Accretion Disks (The Astrophysical Journal) 661, 1025–1033
  * Zhang (2009) Zhang, D.,& Dai, Z. G. 2009, Hyperaccreting Neutron-Star Disks and Neutrino Annihilation (The Astrophysical Journal) 703, 461
  * Janka (2001) Janka, H.-Th. 2001 Conditions for Shock Revival by Neutrino Heating in Core-Collapse Supernovae (Astronomy and Astrophysics) 368, 527–560
  * Colgate (1966) Colgate, S. A.,& White, R. H. 1966 The Hydrodynamic Behavior of Supernovae Explosions (The Astrophysical Journal) 143, 626
  * Bruenn (2001) Bruenn, S. W., Nisco, K. R.,& Mezzacappa, A. 2001 General Relativistic Effects in the Core Collapse Supernova Mechanism (The Astrophysical Journal) 560, 326–338