in the 3D space. Compared with pairwise CRFs, VN encodes triplet based relations, thus being of high order.

**Virtual Normal Loss.** We can sample a large number of triplets and compute corresponding VNs. With the sampled VNs, we compute the divergence as the Virtual Normal Loss (VNL): $$    \def\bn{ { \boldsymbol n  } }
\label{eq:vnl}
    \ell_{VN} = \frac{1}{N} (
    \sum_{i=0}^{N}
        \| { \bn_{i}^{pred}} -{\bn_{i}^{gt}}\|_{1})$$ where the $N$ is the number of valid sampling groups satisfying $\mathscr{R}_{1}, \mathscr{R}_{2}$. In experiments we have employed online hard example mining.

**Pixel-wise Depth Supervision.** We also use a standard pixel-wise depth map loss. We quantize the real-valued depth and formulate the depth prediction as a classification problem instead of regression, and employ the cross-entropy loss. In particular we follow [@cao2017estimating] to use the weighted cross-entropy loss (WCEL), with the weight being the information gain. See [@cao2017estimating] for details.

To obtain the accurate depth map and recover high-quality 3D information, we combine WCEL and VNL together to supervise the network output. The overall loss is: $$\ell=\ell_{WCE} + \lambda \ell_{VN},
\label{eq: total loss}$$ where $\lambda$ is a trade-off parameter, which is set to $5$ in all experiments to make the two terms roughly of the same scale.

