with the i.i.d. assumption, the overall loss is summing over all pixels. To improve the performance, some endeavours have been made to employ other constraints besides the pixel-wise term. For example, a continuous conditional random field (CRF) [@liu2016learning] is used for depth prediction, which takes pair-wise information into account. Other high-order geometric relations [@fei2018geo; @qi2018geonet] are also exploited, such as designing a gravity constraint for local regions [@fei2018geo] or incorporating the depth-to-surface-normal mutual transformation inside the optimization pipeline [@qi2018geonet]. Note that, for the above methods, almost all the geometric constraints are ‘local’ in the sense that they are extracted from a small neighborhood in either 2D or 3D. [Surface normal is ‘local’ by nature as it is defined by the local tangent plane. ]{} As the ground truth depth maps of most datasets are captured by consumer-level sensors, such as the Kinect, depth values can fluctuate considerably. Such noisy measurement would adversely affect the precision and subsequently the effectiveness of those local constraints inevitably. Moreover, local constraints calculated over a small neighborhood have not fully exploited the structure information of the scene geometry that may be possibly used to boost the performance.

To address these limitations, here we propose a 