# PatchFusion: An End-to-End Tile-Based Framework
for High-Resolution Monocular Metric Depth Estimation

Zhenyu Li, Shariq Farooq Bhat, Peter Wonka
King Abdullah University of Science and Technology (KAUST)
https://zhyever.github.io/patchfusion/
<EMAIL_ADDRESS>
###### Abstract

Single image depth estimation is a foundational task in computer vision and
generative modeling. However, prevailing depth estimation models grapple with
accommodating the increasing resolutions commonplace in today’s consumer
cameras and devices. Existing high-resolution strategies show promise, but
they often face limitations, ranging from error propagation to the loss of
high-frequency details. We present PatchFusion, a novel tile-based framework
with three key components to improve the current state of the art: (1) A
patch-wise fusion network that fuses a globally-consistent coarse prediction
with finer, inconsistent tiled predictions via high-level feature guidance,
(2) A Global-to-Local (G2L) module that adds vital context to the fusion
network, discarding the need for patch selection heuristics, and (3) A
Consistency-Aware Training (CAT) and Inference (CAI) approach, emphasizing
patch overlap consistency and thereby eradicating the necessity for post-
processing. Experiments on UnrealStereo4K, MVS-Synth, and Middleburry 2014
demonstrate that our framework can generate high-resolution depth maps with
intricate details. PatchFusion is independent of the base model for depth
estimation. Notably, our framework built on top of SOTA ZoeDepth brings
improvements for a total of 17.3% and 29.4% in terms of the root mean squared
error (RMSE) on UnrealStereo4K and MVS-Synth, respectively.

Figure 1: High-Resolution Depth Estimation. Our tile-based monocular metric
depth estimation model processes high-resolution images and delivers high-
quality depth estimations with intricate details on test images corresponding
to the synthetic training dataset UnrealStereo4K [37] as well as for zero-shot
generalization to other types of real images. Top: In-domain sample from
UnrealStereo4K. Middle: Out-of-domain sample from Middleburry 2014 [35].
Bottom: Out-of-domain sample from the internet.

##  1 Introduction

This paper addresses the challenge of metric single image depth estimation for
high-resolution inputs. Single image depth estimation remains a cornerstone in
computer vision and generative modeling [11, 3, 22, 42]. Yet, most state-of-
the-art (SOTA) depth estimation architectures are bottlenecked by the
resolution capabilities of their backbone. For instance, ZoeDepth [3]
processes an input resolution of 384$\times$512, VPD [43] manages
480$\times$480, and AiT [29] is designed for 384$\times$512\. These figures
pale in comparison to the resolutions offered by modern consumer cameras, such
as the 45 Megapixel Canon EOS R5, the widely available 8K televisions, and
even mobile devices like the iPhone 15, which boasts a 12MP Ultra Wide lens.

Several methods have attempted to bridge this resolution gap: (1) Guided Depth
Super-Resolution (GDSR) techniques [44, 26, 20, 45] aim to refine high-
resolution depth maps from their low-resolution counterparts using high-
resolution color images as reference. (2) Implicit Function approaches, such
as SMD-Net [37], leverage algorithms like [28, 5] to estimate disparities
continuously across image locations, essentially performing on-the-fly super-
resolution. However, due to the low-resolution nature of many models, these
techniques have their limitations: GDSR can propagate errors, and the implicit
function still strips away crucial high-frequency details during input
downsampling. Lastly, the (3) Tile-Based Method proposed in BoostingDepth
[27], emphasizes relative depth estimation by processing image patches
independently before merging them to form a unified depth map.

Our approach refines the concept of tile-based depth estimation. While the
BoostingDepth already has promising results, we identified some shortcomings
to improve. First, we discover that BoostingDepth suffers from scale
inconsistencies, especially when transposed to metric depth estimation. These
inconsistencies then mandate rigorous post-process corrections, such as scale
optimization and Gaussian blending. Second, the fusion network in
BoostingDepth often stumbles due to the lack of guidance and its inability to
grasp a more holistic view of the input image, leading to local optima and
compelling the use of complex heuristic patch selections.

In light of these challenges, we introduce PatchFusion that stands on three
pillars: Firstly, we augment the fusion network with high-level feature
guidance, streamlining its training. Secondly, our proposed Global-to-Local
(G2L) module empowers the fusion network to stay context-aware, eliminating
complex patch selection heuristics. Thirdly, our Consistency-Aware Training
(CAT) and Inference (CAI) strategy places a special emphasis on patch overlap
consistency, facilitating consistency-aware training and inference. As a
result, we achieve an end-to-end tile-based framework without any necessity
for pre-processing such as patch selection, and post-processing like scale
optimization and Gaussian blending. In Fig. 1 we show selected examples to
illustrate how PatchFusion is able to significantly improve the results
compared to the previous SOTA. In summary, our key contributions include:

  * •

The introduction of a novel tile-based network architecture and training
strategy for metric monocular depth estimation called PatchFusion. PatchFusion
is adept at handling high-resolution images and is the first tile-based metric
depth estimations approach that can be trained in an end-to-end manner without
the need for additional post-processing or pre-processing steps.

  * •

We conducted exhaustive empirical validations using the UnrealStereo4K [37],
MVS-Synth [19], and Middleburry 2014 [35] datasets. Our framework further
improves current SOTA by 17.3% and 29.4% on UnrealStereo4K and MVS-Synth,
respectively.

##  2 Related Work

###  2.1 Monocular Depth Estimation

Tremendous progress in monocular depth estimation has been achieved by
publicly available large-scale datasets [36, 14, 7, 6], network design [11, 2,
22, 38, 41, 23], loss supervision [21, 24, 40], refined problem formulations
[13, 9, 2, 40], and training strategies [30, 15, 12]. Recently, the best-
performing frameworks have been built on the transformer architecture [25, 1,
10]. While current SOTA frameworks demonstrate exceptional performance, they
still use low-resolution images as input. For example, the SOTA ZoeDepth [3]
uses BEIT384-L [1] with 307M parameters and only infers 384$\times$512 (about
0.2 megapixel) images. This stands in stark contrast to the advancements in
modern imaging devices that capture images at increasing resolutions, and the
growing demand among users for high-resolution depth estimation. In this work,
we aim to utilize these large-scale models on high-resolution inputs and
achieve high-resolution depth estimation.

(a) Overall Pipeline

(b) Guided Fusion Network

Figure 2: Framework Illustration. (a) Overall Pipeline: The framework consists
of three phases, including a coarse network $\mathcal{N}_{c}$ providing
globally consistent yet coarse depth estimation $\mathbf{D}_{c}$, a fine
network $\mathcal{N}_{f}$ presenting finer details for an input tile (but all
tiles together would lack consistency), and a guided fusion network
$\mathcal{N}_{g}$ combining the best of both worlds. (b) Architecture of
Guided Fusion Network: The lightweight network includes (A) successive encoder
layers, (B) skip-connection modules, and (C) upsampling layers.

###  2.2 High-Resolution Depth

The pursuit of high-resolution depth estimation has typically converged on
three prevalent strategies: (1) Guided Depth Super-Resolution (GDSR): This
approach employs methods [44, 26, 20, 45] to reconstruct high-resolution depth
maps using low-resolution depth observations. These reconstructions are
facilitated by paired high-resolution color images. A notable limitation of
GDSR is its reliance on training samples derived from the direct downsampling
of high-resolution depth maps. When applied to monocular depth
estimation—where low-resolution depth maps stem from models limited by input
resolution constraints—this approach can introduce cascading errors during the
super-resolution process. (2) Implicit Function: Melding the implicit function
[28, 5] with stereo mixture density has enabled architectures like SMD-Net
[37] to deliver precise disparity estimations across continuous image
locations, thereby addressing the stereo super-resolution challenge.
Nevertheless, this method hasn’t fully confronted the constraints posed by
image resolution. Models are still vulnerable to losing intricate high-
frequency details during the input downsampling phase. (3) Tile-Based Method:
This paper aligns itself with the tile-based strategy, pioneered in works such
as BoostingDepth [27] and [33]. The underlying principle here is to segment
depth estimation into patches. Subsequently, these patches are merged to
construct a holistic image depth estimation. A chief advantage of this method
is its ability to sidestep the severe downsampling often mandated by the
restrictions on input resolution.

###  2.3 Depth Map Blending

Given coarse whole-image and fine patch-wise depth maps, blending them
together is a crucial part of tile-based methods. This process aims to keep
the correct global scale in coarse maps while maintaining the fine details in
the tile maps. BoostingDepth [27] applies a linear polynomial, whereas [33]
utilizes a deformable depth field proposed in [18] to achieve the blending.
The former optimizes one patch-wise scale and shift in the least squares
manner and the latter calculates pixel-wise deformation. Unlike previous
approaches, the output of our method can be seamlessly stitched without the
necessity for any post-optimization, resulting in an end-to-end framework for
high-resolution depth estimation.

##  3 Method

In this section, we present the overall framework in Sec. 3.1, Consistency-
Aware Training and Inference in Sec 3.2, and implementation details in Sec.
3.3.

###  3.1 Overall Framework

Our primary objective is to harness the capabilities of a pre-trained base
depth estimation model trained on low resolution and employ it for high-
resolution depth estimation, targeting high resolutions, e.g. 4K.
Unfortunately, a straightforward scaling of current base models to high
resolutions such as 4K far exceeds the memory and compute capabilities of
current hardware. Therefore, we adopt a patch-wise approach, breaking down the
high-resolution depth estimation task into three distinct steps (see Fig.
2(a)): (i) Global Scale-Aware Estimation, (ii) Local Fine-Depth Estimation,
and (iii) Fusion. We train a dedicated network for each step as described
below.

(i) Global Scale-Aware Estimation: In the initial step, we predict a coarse
depth map by first downsampling the input image to the native resolution of
the depth model. This downsampling reduces the memory requirements and
computational load while providing an initial estimation of the depth,
$\mathbf{D}_{c}$. For this step, we fine-tune our base depth model on the
downsampled data, resulting in the coarse network $\mathcal{N}_{c}$. Due to
downsampling, the output from this step is coarse in nature and fine, high-
frequency details are lost at the cost of global consistency.

(ii) Patch-Wise Depth Prediction: In this step, we divide the input images
into smaller manageable patches and feed the cropped patches as input to our
base model. We use a fixed patch size that is equal to or similar to the
native resolution of the base depth model. This results in a fine prediction
$\mathbf{D}_{f}$ containing rich details, particularly at boundaries and
intricate structures. Nevertheless, this detailed depth map, being confined to
only a segment of the original scene, remains oblivious to the global context,
making its scale potentially inconsistent with the actual scene. This could
result in a scale-shifted prediction and fluctuations across patches leading
to obvious patch artifacts as shown in Fig. 3.

Figure 3: Illustration of Inconsistent Depth Prediction. Patch-wise depth
prediction by itself suffers from the lack of global information, leading to
inconsistent depth predictions especially visible at the patch boundaries.

(iii) Fusion and Guided Fusion Network: To resolve the inconsistency issues
with patch-wise prediction, our goal in this step is to transfer the global
information from the coarse depth obtained in (i) to the patch-wise
predictions without compromising their higher details. While the transfer
could be implemented by simply learning a pix2pix U-Net [34, 27], our key idea
is to exploit the multi-scale features from $\mathcal{N}_{c}$ and
$\mathcal{N}_{f}$. We use two main components to achieve this transfer - the
Global-to-Local Module (G2L) and the Guided Fusion Network $\mathcal{N}_{g}$.

Global-to-Local (G2L) Module: Our empirical evaluations (in Tab. 2) indicate
that directly adopting the global guidance feature
$\mathcal{F}_{c}=\\{f_{c}^{i}\\}_{i=1}^{L}$ from $\mathcal{N}_{c}$ (where $L$
is the number of layers) still suffers from the scale-shift issue. Even though
$\mathcal{F}_{c}$ is derived from the entirety of the image, the necessary
information needed for accurate scale inference during fusion is lost post the
cropping operation. Addressing this, we present our G2L module designed to
retain global context.

While the key insight of G2L is to apply the global-wise self attention for
each-level feature in $\mathcal{F}_{c}$ to ensemble crucial information for
patch-wise scale-consistent prediction, we adopt the Swin Transformer Layer
(STL) [25] to preserve the global context while simultaneously alleviating GPU
memory concerns. The main ideas are the local attention and the shifted window
mechanism. Given each feature map $f_{c}^{i}$, STL subdivides it into
localized windows for self-attention (W-SA), which is then followed by
shifted-window attention for inter-window interactions (SW-SA). The operation
can be formulated as:

$f_{g2l}^{i}=\texttt{G2L}(f_{c}^{i})=\texttt{SW-
SA}(\texttt{W-SA}(f_{c}^{i})),$ (1)

where the superscript $i$ iterates through the multi-layer features. The
output set $\mathcal{F}_{g2l}=\\{f_{g2l}^{i}\\}_{i=1}^{L}$ is then sent to the
Guided Fusion Network $\mathcal{N}_{g}$ for further fusion.

Guided Fusion Network: The guided fusion network follows the U-Net [34] design
as shown in Fig. 2(b). The input comprises a concatenated ensemble of the
cropped original image $I$, the corresponding cropped coarse depth estimations
$\mathbf{D}_{c}$ from $\mathcal{N}_{c}$, and fine depth estimations
$\mathbf{D}_{f}$ from $\mathcal{N}_{f}$. The key idea of the design is to only
use image and depth values in the encoder and to delay the injection of
network features to the skip connections and decoder layers.

We use a lightweight encoder consisting of successive convolutional and max-
pooling layers to extract multi-level features. During the skip-connection, we
inject the scale-aware feature $\mathcal{F}_{g2l}$ with a fusion block (FB)
consisting of two $3\times 3$ convolutional layers with ReLU activation
functions as

$\tilde{f}_{g}^{i}=\texttt{FB}(\texttt{roi}(f_{g2l}^{i}),\hat{f}_{g}^{i}),$
(2)

where we apply the roi [17] operation to fetch features of the corresponding
cropped area. $\hat{f}_{g}^{i}$ denotes the initial output feature from the
$i$-th encoder layer. As for the decoder, we again harness the guidance
features $\mathcal{F}_{c}=\\{f_{c}^{i}\\}_{i=1}^{L}$ and
$\mathcal{F}_{f}=\\{f_{f}^{i}\\}_{i=1}^{L}$ from $\mathcal{N}_{c}$ and
$\mathcal{N}_{f}$, respectively, integrating them into the decoder of our
fusion model. The operation can be formulated as:

$f_{g}^{i}=\texttt{FB}(\tilde{f}_{g}^{i},\texttt{roi}(f_{c}^{i}),f_{f}^{i},\texttt{Upsample}(f_{g}^{i-1}))$
(3)

where $\tilde{f}_{g}^{i}$ is obtained from Eq. 2. The Upsample function
2$\times$ rescales the features from the previous level in the decoder. The
output features $\mathcal{F}_{g}=\\{f_{g}^{i}\\}_{i=1}^{L}$ are then sent to a
depth head [3] for final depth estimation.

Figure 4: Consistency-Aware Training Illustration. We apply our consistency
loss on the overlap area of intermediate features and depth predictions from
two patches.

###  3.2 Consistency-Aware Training and Inference

The effectiveness of the fusion network hinges on not just the accuracy of
predictions, but also their consistency across patch boundaries. While our
Guided Fusion Network makes scale-aware predictions, boundary inconsistencies
still exist. Recognizing this gap, we introduce Consistency-Aware Training
(CAT) and Inference (CAI) to ensure patch-wise depth prediction consistency.

Training: Our methodology, see Fig. 4, is based on the intuitive idea that
overlapping regions between cropped patches from the same image should ideally
produce consistent feature representations and depth predictions. We begin by
cropping an image patch, denoted as $I_{1}$. By shifting the cropping window,
we obtain another cropped patch $I_{2}$, such that there exists an overlap
region $\Omega$.

|  |  |  |
---|---|---|---|---
Input |  BoostingDepth [27] |  Graph-GDSR [8] |  Ours |  GT
Figure 5: Qualitative Results on UnrealStereo4K and MVS-Synth. The qualitative
comparisons showcased here are derived from the UnrealStereo4K and MVS-Synth
datasets, depicted in the first two and last two rows, respectively. Zoom in
to better perceive details near boundaries. Our framework outperforms
counterparts [27, 8] with much sharper edges.

Both these patches, $I_{1}$ and $I_{2}$, are independently processed through
our framework, yielding image features $\mathcal{F}_{g}^{1}$ and
$\mathcal{F}_{g}^{2}$, and depth predictions $\mathbf{D}_{1}$ and
$\mathbf{D}_{2}$, respectively. To enforce consistency, we impose an $L_{2}$
loss on the overlapping regions of both the extracted image features and the
depth predictions. This loss penalizes discrepancies in the feature
representations and depth predictions of the overlapping regions from the two
cropped patches. The consistency-aware loss can be mathematically expressed
as:

$\mathcal{L}_{c}=||\mathcal{F}_{g}^{1}-\mathcal{F}_{g}^{2}||_{2}+\mu_{1}||\mathbf{D}_{1}-\mathbf{D}_{2}||_{2},~{}x\in\Omega,$
(4)

where $c$ is short for consistency, $\mu_{1}$ is a hyperparameter empirically
set to 0.1 to balance the loss. While the idea of constraining the depth
values is quite intuitive, the good results mainly stem from constraining the
features.

Inference: One of the distinct advantages of our method, especially when
compared with BoostingDepth [27], is its freedom from heuristic patch
selection and post-processing strategies during inference. In our standard
inference pipeline, we slice the image into $\textsc{P}=16$ non-overlapping
patches, spanning its entirety. These patches are processed, and the depth
maps are seamlessly stitched together. This standard pipeline is called
PatchFusion${}_{\textsc{P=16}}$ in Tab. 1.

To unlock the full power of the network, we complement consistency-aware
inference in the following manner. Our model is further amplified with the
inclusion of an extra 33 shifted, tidily arranged patches, as illustrated in
the supplementary material, as PatchFusion${}_{\textsc{P=49}}$. An additional
improvement is to use extra randomly sampled patches, resulting in
PatchFusion${}_{\textsc{R}}$. Unless otherwise specified, we use $R=128$
random patches in our experiments.

During the processing of patches, the updated depth $\mathbf{D}_{g}$ is
concatenated with the cropped image $I$ and coarse depth map $\mathbf{D}_{c}$,
supplanting the $\mathbf{D}_{f}$, as the input to our guided fusion network.
This dynamic updating, coupled with a running mean, engenders a local
ensemble111Ensemble is combining several different predictions from different
models to make the final prediction. Here, we abuse this term to represent
combining different predictions from different patches but using the same
model approach, incrementally refining the depth estimations on the fly.

###  3.3 Implementation Details

Training: Both networks, $\mathcal{N}_{c}$ and $\mathcal{N}_{f}$, are trained
utilizing the scale-invariant loss $\mathcal{L}_{si}$ [11, 3, 22]. We use
weights from pretraining on the NYU-v2 dataset [36] as our initialization.
$\mathcal{N}_{c}$ and $\mathcal{N}_{f}$ are then fine-tuned on the target
dataset for 16 epochs. The training of the guided fusion network involves a
combination of the scale-invariant loss $\mathcal{L}_{si}$ and our specially
designed consistency loss $\mathcal{L}_{c}$ as

$\mathcal{L}=\mathcal{L}_{si}+\mu_{2}\mathcal{L}_{c},$ (5)

where $\mu_{2}$ is a hyperparameter empirically set to 0.1 to balance the
loss. The fusion network $\mathcal{N}_{g}$ is trained for 12 epochs. Data
augmentation for $\mathcal{N}_{f}$ and $\mathcal{N}_{g}$ includes random
cropping. Beyond this, we use the default augmentation strategies adopted in
the baseline depth model.

##  4 Experimental Results

###  4.1 Datasets and Metrics

UnrealStereo4K: The UnrealStereo4K dataset [37] offers synthetic stereo images
in 4K resolution (2160$\times$3840), complete with accurate, pixel-wise ground
truth. Since the dataset has some incorrectly labeled images, we use the
Structural Similarity Index (SSIM) [39] for quality assurance, comparing the
original and reconstructed left images with the given disparity maps. Entries
with SSIM below 0.7 are excluded (we filter 131 out of 7860 images total).
Utilizing the provided camera parameters, we convert the disparity maps to
metric depth maps. Our experiments follow the prescribed dataset splits in
[37], and we select a patch size of 540$\times$960 by default.

MVS-Synth: MVS-Synth [19] is a synthetic dataset designed for training Multi-
View Stereo (MVS) algorithms. It contains 120 unique sequences, each with 100
frames depicting urban scenes from the virtual environment of Grand Theft Auto
V. Each frame provides a 2K (1080$\times$1920) RGB image, along with a
corresponding ground truth depth map and camera parameters. We divide the
dataset into 11,160 training samples and 240 validation pairs, cropping images
into 270$\times$480 patches for inputs to $\mathcal{N}_{f}$ and
$\mathcal{N}_{g}$.

Middlebury 2014: The Middlebury 2014 dataset [35] comprises a set of high-
resolution stereo images (almost 4K), showcasing indoor scenes in controlled
lighting settings. It includes 23 images paired with corresponding ground-
truth disparity maps. Direct training of a monocular metric depth estimation
model on this dataset is not advised due to the risk of overfitting. We use
the dataset to test our zero-shot generalization capability, particularly from
synthetic to real-world scenarios.

Metrics: We use the standard metric depth evaluation metrics proposed in [11,
31, 3] and present details in the supplementary material. Furthermore, we
introduce two additional metrics to specifically evaluate the precision at
object boundaries and the consistency across patches: (1) Soft Edge Error
(SEE): As per the recommendations in [37, 4], SEE assesses the fidelity of
boundary estimation. It measures the discrepancy between predicted disparity
and the ground truth within a 3×3 local patch around object edges, penalizing
any over-smoothing tendencies and underlining the model’s ability to capture
high-fidelity depth contours. (2) Consistency Error (CE): To ascertain patch
coherence, CE is calculated as the mean absolute error among patches designed
with half-resolution overlaps. This metric evaluates the uniformity of the
depth estimation across different patches, ensuring a seamless transition and
consistency in the reconstructed depth map.

Method | UnrealStereo4K | MVS-Synth
---|---|---
$\delta_{1}(\%)$$\uparrow$ |  REL$\downarrow$ |  RMS$\downarrow$ |  SiLog$\downarrow$ |  SEE$\downarrow$ |  $\delta_{1}(\%)$$\uparrow$ |  REL$\downarrow$ |  RMS$\downarrow$ |  SiLog$\downarrow$ |  SEE$\downarrow$
iDisc [31] |  96.940 |  0.0534 |  1.4035 |  8.5022 |  1.0697 |  93.010 |  0.0866 |  1.4386 |  15.4157 |  1.5624
BoostingDepth* [27] |  75.483 |  0.1890 |  4.7310 |  56.3251 |  3.3204 |  71.393 |  0.2731 |  4.6859 |  85.8841 |  4.1082
BoostingDepth [27] |  98.104 |  0.0437 |  1.1233 |  6.6623 |  0.9390 |  95.409 |  0.0694 |  0.9535 |  11.5144 |  1.2694
Graph-GDSR* [8] |  97.757 |  0.0454 |  1.3012 |  7.6316 |  0.8734 |  94.075 |  0.0760 |  1.2735 |  14.8825 |  1.2386
Graph-GDSR [8] |  97.932 |  0.0438 |  1.2642 |  7.4691 |  0.8718 |  94.195 |  0.0748 |  1.2435 |  14.0723 |  1.2106
SMD-Net [37] |  97.774 |  0.0439 |  1.2817 |  7.3888 |  0.8828 |  93.842 |  0.0776 |  1.2563 |  14.1074 |  1.2747
ZoeDepth${}_{\textsc{coarse}}$ [3] |  97.717 |  0.0455 |  1.2887 |  9.1227 |  0.9144 |  93.978 |  0.0769 |  1.2676 |  14.1236 |  1.3036
ZoeDepth${}_{\textsc{fine}}$ [3] |  97.027 |  0.0627 |  1.2058 |  7.4483 |  0.9546 |  95.113 |  0.0715 |  0.9454 |  11.3844 |  1.1142
ZoeDepth+PF${}_{\textsc{P=16}}$ |  98.419 |  0.0399 |  1.0878 |  6.2122 |  0.8382 |  95.991 |  0.0613 |  0.9213 |  10.1511 |  1.0759
ZoeDepth+PF${}_{\textsc{P=49}}$ |  98.450 |  0.0392 |  1.0747 |  6.1311 |  0.8462 |  96.069 |  0.0599 |  0.9050 |  9.9524 |  1.0700
ZoeDepth+PF${}_{\textsc{R=128}}$ |  98.469 |  0.0388 |  1.0655 |  6.0846 |  0.8488 |  96.172 |  0.0589 |  0.8944 |  9.7696 |  1.0833

Table 1: Quantitative comparison on UnrealStereo4K and MVS-Synth. Best results
are in bold, second best are underlined. * indicates out of bbox inference
without training on the target datasets. PF is short for PatchFusion.
---
Input, Base Depth [32] |  Our Depth Map |  Prompt 1 + Our Depth |  Prompt 2 + Our Depth
Figure 6: Qualitative Results of Depth-Guided Text-to-Image Generation. We
show two cases each with two different text prompts. Given the image, we use
the default depth estimation model [32] in ControlNet [42] and our PatchFusion
to predict depth maps, respectively. Along with the same prompt, the depth
maps are processed by the text-to-image model in ControlNet to generate
images. The depth resolution is aligned by rescaling. We show some images in a
smaller size to save space. Zoom in for details like the leaves and handlebar
marked by red boxes. The images are from Middlebury 2014 [35].

| Method |  REL$\downarrow$ |  SEE$\downarrow$ |  CE$\downarrow$ |  T(s) $\downarrow$
---|---|---|---|---|---
| Coarse Baseline |  0.0455 |  0.9144 |  - |  0.080
| Fine Baseline |  0.0627 |  0.9546 |  0.2546 |  1.132
① | Patch Global Scale&Shift opt. |  0.0544 |  1.0060 |  0.3703 |  2.256
② | Pixel Deformation opt. |  0.0437 |  0.9016 |  0.2213 |  9.729
③ | Poisson Image Editing opt. |  0.0467 |  1.0313 |  0.2499 |  31.875
|  G |  G2L |  CAT-P |  CAT-F |  |  |  |
④ |  |  |  |  |  0.0450 |  0.8932 |  0.2155 |  1.556
⑤ |  ✓ |  |  |  |  0.0423 |  0.8588 |  0.2318 |  1.642
⑥ |  ✓ |  |  ✓ |  |  0.0419 |  0.8415 |  0.1700 |
⑦ |  ✓ |  ✓ |  |  |  0.0414 |  0.8473 |  0.1714 |  2.782
⑧ |  ✓ |  ✓ |  ✓ |  |  0.0411 |  0.8624 |  0.1215 |
⑨ |  ✓ |  ✓ |  |  ✓ |  0.0403 |  0.8451 |  0.1624 |
⑩ |  ✓ |  ✓ |  ✓ |  ✓ |  0.0399 |  0.8382 |  0.1441 |
| PatchFusion${}_{\textsc{P}=49}$ (w/o CAI) |  0.0398 |  0.8397 |  0.1441 |  7.818
★ | PatchFusion${}_{\textsc{P}=49}$ (full method) |  0.0392 |  0.8462 |  0.0464 |

Table 2: Ablation study on UnrealStereo4K dataset. G denotes the feature
guidance. CAT-P and CAT-F denote adopting consistency-aware training on
predictions and features, respectively. Best results are in bold, second best
are underlined. Time: average inference time for one image.

###  4.2 Main Results

Our framework can enhance any metric depth estimation model. For our
experiments, we select ZoeDepth [3] as our baseline due to its SOTA
performance. We compare our framework against various other methods, including
(1) traditional monocular metric depth estimation methods such as ZoeDepth [3]
and iDisc [31], (2) the guided depth super-resolution approach Graph-GDSR [8],
(3) SMD-Net [37] which utilizes an implicit function head for enhanced depth
estimation, and (4) the tile-based technique BoostingDepth [27]. To ensure
fair comparisons, we standardize the depth models across these strategies to
ZoeDepth or we fine-tune the models using their provided NYU-v2 [36]
pretrained versions on both the UnrealStereo4K [37] and MVS-Synth [19]
datasets, setting this as our default experimental protocol.

UnrealStereo4K: The performance comparison on the UnrealStereo4K test set is
presented on the left of Tab. 1. While all compared methods enhance prediction
quality to some degree, our PatchFusion framework significantly surpasses the
baseline ZoeDepth model by 17.3% in RMSE and 14.7% in REL. This indicates a
considerable performance gain afforded by our approach. Our results also
showcase the lowest Soft Edge Error (SEE), evidencing superior boundary
delineation. Qualitative comparisons in Fig. 5 further underscore the high-
quality depth maps generated by our framework. We capture intricate details,
especially in boundary areas. More cases are shown in our supplementary
material.

MVS-Synth: The right of Tab. 1 provides the quantitative comparison results
for experiments on the MVS-Synth dataset [19]. Our framework outperforms all
alternative approaches in every evaluated metric. Notably, our technique
boosts the RMS score by approximately 29.4% and reduces the scale-invariant
logarithmic error by 30.8%. We present qualitative results in Fig. 5 and the
supplementary material.

Middlebury 2014: We first evaluate the zero-shot transfer capability of our
framework. The accompanying supplementary material provides both quantitative
and qualitative comparisons to illustrate the efficacy of our approach.
Building on this foundation, we explored our framework’s application to the
field of text-to-image generation. We substituted the conventional depth
estimation model [32] employed by ControlNet [42] with our PatchFusion. To
ensure compatibility with the depth range required by ControlNet, we
normalized the metric depth output of PatchFusion to the interval $[0,256]$
and then applied the transformation $256-\hat{d}$ to invert it. The results
are illustrated in Fig. 6. Armed with the high-fidelity depth maps generated
by PatchFusion, which capture intricate details, the text-to-image generation
model yields coherent structures, textures, and details of significantly
higher quality than the baseline.

###  4.3 Ablation Studies and Discussion

In this section, we ablate and discuss the contributions of individual
components within our framework. Unless stated otherwise, we utilize the
UnrealStereo4K dataset and the PatchFusion variant with $P=16$ patches for
clarity and ease of comparison. Inference time benchmarks are performed on a
single NVIDIA A100 GPU.

Comparison with other Blending Strategies: The primary role of the fusion
network in our framework is to act as an adaptive blending module, integrating
the fine detail from $\mathbf{D}_{f}$ with the scale information retained in
$\mathbf{D}_{c}$. We benchmark our network against other blending strategies
prevalent in various tile-based depth estimation methods [27, 33, 18]. As
illustrated in Tab. 2 (①, ②, ③), our approach yields superior results in terms
of estimation accuracy and satisfactory computational efficiency. We highlight
that traditional blending strategies often grapple with the challenge of
specifying an optimal handcrafted target for optimization, which can result in
suboptimal outcomes. Our network circumvents this issue by learning to blend
the input patches effectively through a data-driven process. This is evidenced
by the substantial gains in performance metrics, with an 8.7% improvement in
REL and a 34.8% reduction in CE during inference.

Guided Fusion Network with G2L Module: Our initial focus is to evaluate the
impact of the Guided Fusion Network equipped with the G2L (Global to Local)
Module. We incrementally introduce the guidance feature and the G2L module to
a baseline encoder-decoder structure, similar to the MergeNet in BoostingDepth
[27], and observe the changes in performance. As indicated in Tab. 2, the
standalone model without the guidance feature and the G2L module (④) yields
the least impressive results. With RMSE and SEE metrics closely aligning with
the coarse baseline, and only a marginal improvement in the consistency error,
it becomes evident that this variant struggles to integrate the finer details
from $\mathbf{D}_{f}$ into the coarse depth map $\mathbf{D}_{c}$. This
underlines the crucial role our framework plays in surpassing the capabilities
of BoostingDepth [27].

Introducing the guidance feature (⑤) marks an uptick in performance with
improvements of 6.0% in REL and 3.8% in SEE, confirming the feature’s role in
enhancing training and the assimilation of fine details. The integration of
the G2L module (⑦) leads to a 26.0% reduction in consistency error, achieved
without imposing explicit constraints. The concomitant reduction in 2.1% REL
underscores the G2L module’s effectiveness in harnessing global information,
thus improving both accuracy and consistency.

Consistency-Aware Training and Inference: Integrating the consistency-aware
training (CAT) loss, results in enhanced REL and CE metrics for models, both
with and without the G2L module (⑥ and ⑧, respectively). Notably, the
introduction of the G2L module and consistency loss leads to a marginal
reduction in SEE. This suggests that the model may be excessively penalized,
potentially neglecting some fine details in the process. When the consistency
loss is applied to the intermediate features (⑨), we observe a more pronounced
improvement in REL, while SEE is preserved, indicating a balanced detail
capture. Nevertheless, this approach yields a diminished CE, suggesting a
slight trade-off. Upon concurrently enforcing consistency constraints on both
predictions and intermediate features, we note a further refinement in both
REL and SEE metrics, along with an acceptable level of CE (⑩). This finding
underscores the balanced improvements in standard metrics and patch-wise
consistency, which is pivotal for a tile-based framework. Collectively, these
results affirm the CAT’s role in bolstering high-quality depth estimation.
Finally, with the use of additional patches and our consistency-aware
inference (CAI), we obtain another significant boost in REL and CE (★) due to
the local ensemble, but at the expense of SEE and time complexity.

###  4.4 Limitations and Future Work

One limitation is the computational efficiency of our framework. While
incorporating an increasing number of randomly selected patches does improve
depth prediction, it also results in a proportional increase in processing
time. We therefore suggest an efficient patch selection strategy as an avenue
for future work. Another limitation is the lack of high-resolution real-world
training data. We can observe this gap in our results from synthetic to real
transfer. While the results showcase sharp edges, the scale of the results
could be improved. Also, we noticed that UnrealStereo4K [37] does not contain
enough images with large homogeneous foreground objects. We believe that the
collection of large real-world high-resolution depth datasets will be a great
contribution, as depth estimation, in general, seems to be a highly valuable
pre-training task [16], and therefore recommend this as future work.

##  5 Conclusions

We presented PatchFusion, an end-to-end tile-based framework tailored to high-
resolution monocular metric depth estimation. We introduced a novel tile-based
network architecture together with a consistency-aware training and inference
strategy. This combination yields a framework that only relies on forward
passes through networks and obviates the need for additional pre-processing
and post-processing. Our proposed framework decisively improves upon the
baseline model for UnrealStereo4K (17.3% in RMSE) and MVS-Synth (29.4% in
RMSE), while demonstrating satisfactory performance in zero-shot transfer.

|  |  |  |
---|---|---|---|---
GT, Input |  Graph-DGSR [8] |  SMD-Net [37] |  BoostingDepth [27] |  Ours
Figure 7: Qualitative Results on MVS-Synth. The first row shows the predicted depth maps while the second row depicts the corresponding error maps. Zoom in to better perceive details near boundaries. As we claim in Sec.1, due to the low-resolution nature of base models, DGSR can propagate errors, and the implicit function still strips away crucial high-frequency details during downsampling. Compared with the SOTA tile-based framework BoostingDepth [27], our framework achieves better estimation quality at boundaries and shows better scale consistency. |  |  |
---|---|---|---
GT, Input |  ZeoDepth${}_{\textsc{COARSE}}$ |  BoostingDepth [27] |  Ours
Figure 8: Qualitative Results on UnrealStereo4K. The first row shows the
predicted depth maps while the second row depicts the corresponding error
maps. Zoom in to better perceive details near boundaries. Our framework
outperforms BoostingDepth [27] not only in details but also scale-consistency.

##  Appendix A Evaluation Metrics

We follow the standard evaluation protocol proposed in previous monocular
metric depth estimation works to evaluate the effectiveness of our framework.
We utilize the root mean squared error (RMSE)
$=|\frac{1}{M}\sum_{i=1}^{M}|d_{i}-\hat{d}_{i}|^{2}|^{\frac{1}{2}}$, mean
absolute relative error (AbsRel)
$=\frac{1}{M}\sum_{i=1}^{M}|d_{i}-\hat{d}_{i}|/d_{i}$, scale-invariant
logarithmic error (SILog)
$=|\frac{1}{M}\sum_{i=1}^{M}e^{2}-|\frac{1}{M}\sum_{i=1}^{M}e|^{2}|^{1/2}\times
100$ where $e=\log{\hat{d}_{i}}-\log{d_{i}}$, and the accuracy under the
threshold ($\delta_{i}<1.25^{i},i=1$), where $d_{i}$ and $\hat{d}_{i}$ refer
to ground truth and predicted depth at pixel $i$, respectively, and $M$ is the
total number of pixels in the image. Since UnrealStereo4K [37] contains both
indoor and outdoor images, we cap the evaluation depth at the range of
$(1e^{-3}\text{m},80\text{m})$. Final model outputs are evaluated at ground
truth resolution.

##  Appendix B More Qualitative Results

We present qualitative comparisons with all three types of strategies for
depth super-resolution in Fig. 7, 8: (1) Depth-Guided Super-Resolution Graph-
GDSR [8], (2) Baseline depth model ZoeDepth [3] with Implicit Function [37],
and (3) Tile-Based Framework BoostingDepth [27] finetuning on the target
dataset. Due to the low-resolution nature of base models, (1) DGSR propagates
errors, and the implicit function (2) still strips away crucial high-frequency
details during input downsampling. Compared with (3) the SOTA tile-based
framework BoostingDepth [27], our framework achieves better estimation quality
at boundaries and shows better scale consistency.

|  |  |
---|---|---|---
Input |  BoostingDepth [27] |  Ours |  GT
Figure 9: Qualitative Comparison on Middlebury 2014. Zoom in to observe more
details. We use BoostingDepth [27] out of the box and align the prediction of
BoostingDepth with ground-truth scales with post-process optimization for
visualization. Note that BoostingDepth is trained on this dataset, while we
adopt PatchFusion in a synthetic-to-real zero-shot transfer manner.

##  Appendix C Zero-Shot Transfer on Middlebury 2014

Method |  RMS$\downarrow$ |  SEE$\downarrow$
---|---|---
ZoeDepth${}_{\textsc{COARSE}}$ |  1.0777 |  0.8326
ZoeDepth + PatchFusion${}_{\textsc{P=16}}$ |  1.0743 |  0.8284
ZoeDepth + PatchFusion${}_{\textsc{P=40}}$ |  1.0678 |  0.8219
ZoeDepth + PatchFusion${}_{\textsc{r=128}}$ |  1.0620 |  0.8195
ZoeDepth + PatchFusion${}_{\textsc{r=256}}$ |  1.0580 |  0.8194
ZoeDepth + PatchFusion${}_{\textsc{r=1024}}$ |  1.0536 |  0.8178

Table 3: Results of models trained on the UnrealStereo4K dataset and tested on
the Middleburry 2014 dataset without fine-tuning. Best results are in bold,
second best are underlined.

In this set of experiments, we delve into the zero-shot transfer capability of
our framework, which is an essential quality for deployment in diverse real-
world scenarios. The outcomes, as depicted in Tab. 3, confirm that each
configuration of our framework successfully elevates the zero-shot transfer
potential beyond the baseline model. The comparisons also substantiate our
claim from Sec. 3: incorporating an incremental number of randomly selected
patches within our framework would systematically refine the depth
estimations. Visual results are provided in Fig. 9. Remarkably, our method
achieves enhanced results on the Middlebury 2014 dataset [35], delineating
clearer object boundaries, despite the fact that it has not been trained on
it. This is in contrast to the BoostingDepth [27], which includes training on
this specific dataset.

##  Appendix D Inference Setup

As shown in Fig. 10, we slice the image into $\textsc{P}=16$ non-overlapping
patches, spanning its entirety. These patches are processed, and the depth
maps are seamlessly stitched together. This standard pipeline is called
PatchFusion${}_{\textsc{P=16}}$. To unlock the full power of the network, our
model is further amplified with the inclusion of an extra 33 shifted, tidily
arranged patches, as PatchFusion${}_{\textsc{P=49}}$. An additional
improvement is to use extra randomly sampled patches, resulting in
PatchFusion${}_{\textsc{R}}$.

Figure 10: Tile Placement. We slice the image into $\textsc{P}=16$ non-
overlapping patches, leading to PatchFusion${}_{\textsc{P}=16}$. Then, we
introduce an extra 33 shifted, tidily arranged patches, corresponding to
PatchFusion${}_{\textsc{P}=49}$. In PatchFusion${}_{\textsc{R}=128}$, the
extra 128 patches are then randomly placed on the image.

##  Appendix E Qualitative Comparison of CAI

In our study, we introduce a Consistency-Aware Inference (CAI) technique aimed
at improving the consistency of patch-wise depth estimations during the
inference phase. The significance of this approach is highlighted in Fig. 11.
Although our Guided Fusion Network, supplemented by the Global-to-Local module
and Consistency-Aware Training, facilitates scale-aware and consistency-aware
predictions, discontinuities, particularly around prediction edges, remain a
challenge. This problem persists even when increasing the number of randomly
selected patches, as newer patch estimations tend to overwrite the earlier
ones.

To address this issue, the CAI strategy updates predictions in a running
average manner. It calculates the prediction for each pixel as an average of
all previous predictions from overlapping patches. This method not only
ensures robustness but also enhances the continuity of predictions across
different patches, resulting in a more seamless and integrated depth
estimation.

Figure 11: Qualitative Comparison of Consistency-Aware Inference (CAI). Left:
PatchFusion${}_{\textsc{R=128}}$ without CAI. Right:
PatchFusion${}_{\textsc{R=128}}$ with CAI. Without CAI, newer patch
estimations tend to overwrite the earlier ones, leading to inevitable
artifacts on patch boundaries. Our CAI successfully alleviate this issue by
the running average strategy.

## References

  * Bao et al. [2021] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_ , 2021.
  * Bhat et al. [2021] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka.  Adabins: Depth estimation using adaptive bins.  In _CVPR_ , pages 4009–4018, 2021.
  * Bhat et al. [2023] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller.  Zoedepth: Zero-shot transfer by combining relative and metric depth.  _arXiv preprint arXiv:2302.12288_ , 2023.
  * Chen et al. [2019] Chuangrong Chen, Xiaozhi Chen, and Hui Cheng.  On the over-smoothing problem of cnn based disparity estimation.  In _ICCV_ , pages 8997–9005, 2019.
  * Chen et al. [2021] Yinbo Chen, Sifei Liu, and Xiaolong Wang.  Learning continuous image representation with local implicit image function.  In _CVPR_ , pages 8628–8638, 2021.
  * Cordts et al. [2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _CVPR_ , pages 3213–3223, 2016.
  * Dai et al. [2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _CVPR_ , pages 5828–5839, 2017.
  * De Lutio et al. [2022] Riccardo De Lutio, Alexander Becker, Stefano D’Aronco, Stefania Russo, Jan D Wegner, and Konrad Schindler.  Learning graph regularisation for guided super-resolution.  In _CVPR_ , pages 1979–1988, 2022.
  * Diaz and Marathe [2019] Raul Diaz and Amit Marathe.  Soft labels for ordinal regression.  In _CVPR_ , pages 4738–4747, 2019.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Eigen et al. [2014] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  _NeurIPS_ , 27, 2014.
  * Fan et al. [2023] Rizhao Fan, Matteo Poggi, and Stefano Mattoccia.  Contrastive learning for depth prediction.  In _CVPRW_ , pages 3225–3236, 2023.
  * Fu et al. [2018] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep ordinal regression network for monocular depth estimation.  In _CVPR_ , pages 2002–2011, 2018.
  * Geiger et al. [2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  _IJRR_ , 32(11):1231–1237, 2013.
  * Godard et al. [2019] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow.  Digging into self-supervised monocular depth estimation.  In _ICCV_ , pages 3828–3838, 2019.
  * Goldblum et al. [2023] Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, et al.  Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks.  _arXiv preprint arXiv:2310.19909_ , 2023.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * Hedman and Kopf [2018] Peter Hedman and Johannes Kopf.  Instant 3d photography.  _ACM Transactions on Graphics (TOG)_ , 37(4):1–12, 2018.
  * Huang et al. [2018] Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang.  Deepmvs: Learning multi-view stereopsis.  In _CVPR_ , 2018.
  * Hui et al. [2016] Tak-Wai Hui, Chen Change Loy, and Xiaoou Tang.  Depth map super-resolution by deep multi-scale guidance.  In _ECCV_ , pages 353–369. Springer, 2016.
  * Lee and Kim [2020] Jae-Han Lee and Chang-Su Kim.  Multi-loss rebalancing algorithm for monocular depth estimation.  In _ECCV_ , pages 785–801. Springer, 2020.
  * Li et al. [2022] Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang.  Binsformer: Revisiting adaptive bins for monocular depth estimation.  _arXiv preprint arXiv:2204.00987_ , 2022.
  * Li et al. [2023] Zhenyu Li, Zehui Chen, Xianming Liu, and Junjun Jiang.  Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation.  _Machine Intelligence Research_ , pages 1–18, 2023.
  * Liu et al. [2023] Ce Liu, Suryansh Kumar, Shuhang Gu, Radu Timofte, and Luc Van Gool.  Single image depth prediction made better: A multivariate gaussian take.  In _CVPR_ , pages 17346–17356, 2023.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_ , pages 10012–10022, 2021.
  * Metzger et al. [2023] Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler.  Guided depth super-resolution by deep anisotropic diffusion.  In _CVPR_ , pages 18237–18246, 2023.
  * Miangoleh et al. [2021] S Mahdi H Miangoleh, Sebastian Dille, Long Mai, Sylvain Paris, and Yagiz Aksoy.  Boosting monocular depth estimation models to high-resolution via content-adaptive multi-resolution merging.  In _CVPR_ , pages 9685–9694, 2021.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021.
  * Ning et al. [2023] Jia Ning, Chen Li, Zheng Zhang, Zigang Geng, Qi Dai, Kun He, and Han Hu.  All in tokens: Unifying output space of visual tasks via soft token.  _arXiv preprint arXiv:2301.02229_ , 2023.
  * Petrovai and Nedevschi [2022] Andra Petrovai and Sergiu Nedevschi.  Exploiting pseudo labels in a self-supervised learning framework for improved monocular depth estimation.  In _CVPR_ , pages 1578–1588, 2022.
  * Piccinelli et al. [2023] Luigi Piccinelli, Christos Sakaridis, and Fisher Yu.  idisc: Internal discretization for monocular depth estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21477–21487, 2023.
  * Ranftl et al. [2022] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  _IEEE TPAMI_ , 44(3), 2022.
  * Rey-Area et al. [2022] Manuel Rey-Area, Mingze Yuan, and Christian Richardt.  360monodepth: High-resolution 360deg monocular depth estimation.  In _CVPR_ , pages 3762–3772, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015.
  * Scharstein et al. [2014] Daniel Scharstein, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nešić, Xi Wang, and Porter Westling.  High-resolution stereo datasets with subpixel-accurate ground truth.  In _Pattern Recognition: 36th German Conference, GCPR 2014, Münster, Germany, September 2-5, 2014, Proceedings 36_ , pages 31–42. Springer, 2014.
  * Silberman et al. [2012] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _ECCV_ , pages 746–760. Springer, 2012.
  * Tosi et al. [2021] Fabio Tosi, Yiyi Liao, Carolin Schmitt, and Andreas Geiger.  Smd-nets: Stereo mixture density networks.  In _CVPR_ , pages 8942–8952, 2021.
  * Wang et al. [2020] Lijun Wang, Jianming Zhang, Yifan Wang, Huchuan Lu, and Xiang Ruan.  Cliffnet for monocular depth estimation with hierarchical embedding loss.  In _ECCV_ , pages 316–331. Springer, 2020.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE TIP_ , 13(4):600–612, 2004.
  * Xian et al. [2020] Ke Xian, Jianming Zhang, Oliver Wang, Long Mai, Zhe Lin, and Zhiguo Cao.  Structure-guided ranking loss for single image depth prediction.  In _CVPR_ , pages 611–620, 2020.
  * Yang et al. [2021] Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe, and Elisa Ricci.  Transformer-based attention networks for continuous pixel-wise prediction.  In _ICCV_ , pages 16269–16279, 2021.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _ICCV_ , pages 3836–3847, 2023.
  * Zhao et al. [2023] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu.  Unleashing text-to-image diffusion models for visual perception.  _ICCV_ , 2023.
  * Zhao et al. [2022] Zixiang Zhao, Jiangshe Zhang, Shuang Xu, Zudi Lin, and Hanspeter Pfister.  Discrete cosine transform network for guided depth map super-resolution.  In _CVPR_ , pages 5697–5707, 2022.
  * Zhong et al. [2023] Zhiwei Zhong, Xianming Liu, Junjun Jiang, Debin Zhao, and Xiangyang Ji.  Guided depth map super-resolution: A survey.  _ACM Computing Surveys_ , 2023.


# A Survey of Local Differential Privacy and Its Variants

Likun Qin, Nan Wang, Tianshuo Qiu

Department of Electrical and Computer Engineering  Shandong University, Jinan,
China

###### Abstract

The introduction and advancements in Local Differential Privacy (LDP) variants
have become a cornerstone in addressing the privacy concerns associated with
the vast data produced by smart devices, which forms the foundation for data-
driven decision-making in crowdsensing. While harnessing the power of these
immense data sets can offer valuable insights, it simultaneously poses
significant privacy risks for the users involved. LDP, a distinguished privacy
model with a decentralized architecture, stands out for its capability to
offer robust privacy assurances for individual users during data collection
and analysis. The essence of LDP is its method of locally perturbing each
user’s data on the client-side before transmission to the server-side,
safeguarding against potential privacy breaches at both ends. This article
offers an in-depth exploration of LDP, emphasizing its models, its myriad
variants, and the foundational structure of LDP algorithms.

##  I Introduction

Collecting and analyzing data introduces significant privacy concerns because
it often includes sensitive user information. With the advent of sophisticated
data fusion and analysis methods, user data becomes even more susceptible to
breaches and exposure in this era of big data. For instance, by studying
appliance usage, adversaries can deduce daily routines or behaviors of
individuals, like when they are home or their specific activities such as
watching TV or cooking. It’s crucial to prioritize the protection of personal
data when gathering information from diverse devices. Currently, the European
Union (EU) has released the GDPR[1], which oversees EU data protection laws
for its citizens and outlines the specifics related to the handling of
personal data. Similarly, the U.S. National Institute of Standards and
Technology (NIST) is in the process of crafting privacy frameworks. These
frameworks aim to more effectively recognize, evaluate, and address privacy
risks, enabling individuals to embrace innovative technologies with increased
trust and confidence[2, 3].

From a privacy-protection standpoint, differential privacy (DP) has been
introduced over a decade ago [4, 5]. Recognized as a robust framework for
safeguarding privacy, it’s often termed as global DP or centralized DP. DP’s
strength lies in its mathematical rigor; it operates independent of an
adversary’s background knowledge and assures potent privacy protection for
users. It has found applications across various domains[6]. However, DP
assumes the presence of a trustworthy server, which can be a challenge since
many online platforms or crowdsourcing systems might have untrustworthy
servers keen on user data statistics[7, 8].

Emerging from the concept of DP, local differential privacy (LDP) was
introduced [9]. LDP stands as a decentralized version of DP, offering
individualized privacy assurances and making no assumptions about third-party
server trustworthiness. LDP has become a focal point in privacy research due
to its theoretical significance and practical implications[10]. Numerous
corporations, including Apple’s iOS[11], Google Chrome, and the Windows
operating system, have integrated LDP-driven algorithms into their systems.
Owing to its robust capabilities, LDP has become a preferred choice to address
individual privacy concerns during various statistical and analytical
operations. This includes tasks like frequency and mean value estimation[12],
the identification of heavy hitters[13], k-way marginal release, empirical
risk minimization (ERM), federated learning, and deep learning.

While LDP is powerful, it’s not without its challenges, notably in striking an
optimal balance between utility and privacy[14]. To address this, there are
two primary approaches. Firstly, by devising improved mechanisms - leading to
the introduction of numerous LDP-based protocols and sophisticated mechanisms
in academic circles. Secondly, by revisiting the definition of LDP itself,
with researchers suggesting more flexible privacy concepts to better cater to
the utility-privacy balance required for real-world applications. Given the
growing significance of LDP, a thorough survey of the topic is both timely and
essential. While there exists some literature reviewing LDP, the focus has
often been narrow. They either focus on specific applications or certain types
of mechanisms.

In this paper, we delve deep into the world of LDP and its various offshoots,
meticulously studying their recent advancements and associated mechanisms. We
embark on a thorough exploration of the foundational principles that drive LDP
and the evolutionary trajectories of its multiple variants. We aim to identify
the cutting-edge developments, shedding light on the innovations that have
shaped these privacy tools and the challenges they aim to address in our
contemporary digital landscape. Furthermore, we analyze the specific
mechanisms that support and enhance the capabilities of LDP, understanding
their technical intricacies and the real-world applications they cater to.
Through this comprehensive study, we aspire to provide readers with a
panoramic view of the current state of LDP research, setting the stage for
future inquiries and innovations in this critical domain.

##  II Local Differential Privacy, Properties and Mechanisms

In this section, we study LDP and its properties, and LDP based mechanisms. We
start from the definition of LDP.

Definition 1 ($\varepsilon$-Local Differential Privacy ($\varepsilon$-LDP) .
A randomized mechanism $M$ satisfies $\varepsilon$-LDP if and only if for any
pairs of input values $v$, $v^{\prime}$ in the domain of $M$, and for any
possible output $y\in Y$, it holds

$P[M(v)=y]\leq e^{\varepsilon}\cdot P[M(v^{\prime})=y],$ (1)

where $P[\cdot]$ denotes probability and $\varepsilon$ is the privacy budget.
A smaller $\varepsilon$ means stronger privacy protection, and vice versa.

The basic properties of LDP include the followings:

Composition: [15] Given two mechanisms $M_{1}$ and $M_{2}$ that provide
$\varepsilon_{1}$-LDP and $\varepsilon_{2}$-LDP respectively, their sequential
composition provides $(\varepsilon_{1}+\varepsilon_{2})$-LDP.

$M(v)=(M_{1}(v),M_{2}(v))\implies M\text{ is
}(\varepsilon_{1}+\varepsilon_{2})\text{-LDP}$ (2)

Post-processing: Any function applied to the output of an $\varepsilon$-LDP
mechanism retains the $\varepsilon$-LDP guarantee.

$\text{If }M(v)\text{ is }\varepsilon\text{-LDP, then }f(M(v))\text{ is also
}\varepsilon\text{-LDP}.$ (3)

Robustness to Side Information: LDP guarantees hold even if an adversary has
access to auxiliary or side information.

Utility-Privacy Tradeoff: Generally, a lower value of $\varepsilon$ implies
stronger privacy but might result in reduced utility of the perturbed data.

Independence of Background Knowledge: The privacy guarantees of LDP mechanisms
are designed to hold regardless of any background knowledge an adversary might
have.

Next, we study mechanisms that satisfy LDP:

Randomize Response[16]

The Randomized Response Mechanism is a simple yet effective approach to
achieving LDP. It’s particularly used for binary data, i.e., when a user’s
data item is either $0$ or $1$. The mechanism operates as follows:

  1. 1.

With probability $\frac{1}{2}$, the user truthfully answers a question.

  2. 2.

With probability $\frac{1}{2}$, the user randomly answers the question.

Mathematically, given a user’s true data item $v\in\\{0,1\\}$, the mechanism
outputs $v$ with probability $\frac{1}{2}$ and outputs $1-v$ (i.e., the
opposite of $v$) with probability $\frac{1}{2}$.

The probability mass function (pmf) is given by:

$\displaystyle P[M(v)=1]$
$\displaystyle=\frac{1}{2}v+\frac{1}{2}(1-v)=\frac{1}{2}$ (4) $\displaystyle
P[M(v)=0]$ $\displaystyle=\frac{1}{2}(1-v)+\frac{1}{2}v=\frac{1}{2}$ (5)

This mechanism ensures $\varepsilon$-LDP with $\varepsilon=\ln(2)$.

Laplace Mechanism[17]

The Laplace Mechanism adds noise drawn from the Laplace distribution to the
true value of the data. For LDP, this mechanism can be adjusted as:

Given a data item $v$, the mechanism outputs:

$M(v)=v+\text{Lap}(\frac{\Delta f}{\varepsilon})$

where $\Delta f$ is the sensitivity of the function $f$ and
$\text{Lap}(\cdot)$ represents the Laplace distribution.

Gaussian Mechanism

Similar to the Laplace Mechanism, the Gaussian Mechanism adds noise but from
the Gaussian distribution:

Given a data item $v$, the mechanism outputs:

$M(v)=v+\mathcal{N}(0,\sigma^{2})$

where $\sigma^{2}$ determines the amount of noise based on the desired
$\varepsilon$ and function sensitivity $\Delta f$.

Exponential Mechanism

The Exponential Mechanism selects an output based on a scoring function and
weights outputs with the exponential of their score. Given a set of possible
outputs $R$, a data item $v$, and a scoring function $q(v,r)$, the probability
of selecting output $r$ is proportional to:

$\exp\left(\frac{\varepsilon q(v,r)}{2\Delta q}\right)$

where $\Delta q$ is the sensitivity of $q$.

Perturbed Histogram Mechanism

For a set of items, instead of perturbing each item, this mechanism perturbs
the histogram of the data items. Given a data item set $V$, the mechanism
constructs a histogram $H$ and then outputs:

$M(H)=H+\text{Lap}(\frac{\Delta H}{\varepsilon})$

where $\Delta H$ is the sensitivity of the histogram construction.

Observe that each mechanism’s efficacy is closely tied to the sensitivity of
the query, denoted as $\Delta f$. In the realm of Local Differential Privacy
(LDP), this sensitivity can often grow significantly, especially when the
input domain is vast. The larger the sensitivity, the more noise needs to be
introduced by the mechanism to ensure the desired privacy level. This can lead
to significant distortion in the data, compromising its utility.

Furthermore, as the input support size increases, maintaining the desired
privacy guarantee becomes a challenge. Noise calibrated to a high sensitivity
can sometimes overshadow the actual data, rendering the results almost
meaningless or leading to misinterpretations.

The consequence of this is a pronounced tradeoff between utility and privacy.
Achieving stronger privacy often means accepting reduced accuracy and utility
in the results, and vice versa. For applications that require high precision,
this can be problematic. It implies that while these mechanisms provide a
robust privacy guarantee in theory, their practical applicability can be
constrained, especially in scenarios where fine-grained insights from data are
crucial.

Hence, while the promise of LDP is enticing, its real-world implementation
requires careful consideration of the utility-privacy balance, pushing
researchers to seek more efficient mechanisms or modified privacy models to
better cater to practical needs.

##  III Advanced LDP mechanisms

As we mentioned in the introduction, to improve the utility-privacy tradeoff
provided by LDP, there are typically two manners. One is to design dedicated
mechanism or advanced protocols. The other is to relax the definition of LDP
to enhance the data utility. In this section, we summarized several advanced
LDP algorithms, aiming to improve the general utility-privacy tradeoff.

RAPPOR [10] (Randomized Aggregatable Privacy-Preserving Ordinal Response):

Introduced by Google. RAPPOR enhances the randomized response mechanism
through the incorporation of Bloom filters. Each user’s value is hashed
multiple times into a Bloom filter, which is then perturbed using the RR
technique. This allows multiple string values to be encoded before
randomization. Advantage: Its main strength lies in collecting statistics
about low-frequency items in the user population. It can provide meaningful
insights even when items are not commonly observed.

Local Hashing[12]:

Addressing the problem of efficiency in the RR technique when dealing with a
large domain of inputs, local hashing maps the original vast domain into a
smaller domain using hash functions. This condensed domain can then be
analyzed using traditional RR techniques. Advantage: It substantially reduces
the noise introduced in the randomization process, enabling accurate
estimation of frequencies for individual items in the domain. This mechanism
improves the utility, especially when the original domain is considerably
large.

Piecewise RR:

Instead of applying the same randomization mechanism across the entire input
domain, the Piecewise RR technique divides the domain into multiple segments
or pieces. Each segment then gets its own randomization mechanism tailored to
its characteristics. Advantage: This method achieves a more granular utility-
privacy tradeoff. It can offer enhanced privacy in sensitive segments while
improving utility in less-sensitive ones. Optimized RR:

The protocol doesn’t just use a fixed randomization parameter; instead, it
optimizes the parameters of the RR. This optimization is often based on real
data distribution or some auxiliary information, ensuring that the
randomization provides the best possible utility. Advantage: By adjusting the
randomization according to data distribution, it achieves better accuracy in
aggregate statistics.

Fourier Perturbation Algorithm (FPA) [18]:

Instead of perturbing the raw data directly, FPA operates in the frequency
domain. The data undergoes a Fourier transformation, after which the
perturbation is applied. This allows for randomization in a different space
that might be more conducive to certain types of analyses. Advantage: Provides
enhanced utility for specific query types, especially those that are
frequency-based or need insights from periodic patterns in data.

##  IV LDP variants and mechanisms

In this section, we introduce LDP variants that aim to provide better utility-
privacy tradeoff in different applications.

###  IV-A Variants and Mechanisms of LDP

####  IV-A1 $(\varepsilon,\delta)$-LDP

Drawing parallels with how $(\varepsilon,\delta)$-DP [19] extends
$\varepsilon$-DP, $(\varepsilon,\delta)$-LDP (sometimes termed as approximate
LDP) serves as a more flexible counterpart to $\varepsilon$-LDP (or pure LDP).

######  Definition 1 (Approximate Local Differential Privacy).

A randomized process $M$ complies with $(\varepsilon,\delta)$-LDP if, for all
input pairs $v$ and $v^{\prime}$ within $M$’s domain and any probable output
$y\in Y$, the following holds:

$P[M(v)=y]\leq e^{\varepsilon}\cdot P[M(v^{\prime})=y]+\delta.$

Here, $\delta$ is customarily a small value.

In essence, $(\varepsilon,\delta)$-LDP implies that $M$ achieves
$\varepsilon$-LDP with a likelihood not less than $1-\delta$. If $\delta=0$,
$(\varepsilon,\delta)$-LDP converges to $\varepsilon$-LDP.

####  IV-A2 BLENDER Model

BLENDER [20], a fusion of global DP and LDP, optimizes data utility while
retaining privacy. It classifies users based on their trust in the aggregator
into two categories: the opt-in group and clients. BLENDER enhances utility by
balancing data from both. Its privacy measure mirrors that of
$(\varepsilon,\delta)$-DP [21].

####  IV-A3 Geo-indistinguishability

Originally tailored for location privacy with global DP, Geo-
indistinguishability [22] uses the data’s geographical distance. Alvim et al.
[23] argued for metric-based LDP’s advantages in specific contexts.

######  Definition 2 (Geo-indistinguishability).

A randomized function $M$ adheres to Geo-indistinguishability if, for any
input pairs $v$ and $v^{\prime}$ and any output $y\in Y$, the subsequent
relation is met:

$P[M(v)=y]\leq e^{\varepsilon\cdot d(v,v^{\prime})}\cdot P[M(v^{\prime})=y],$

where $d(.,.)$ designates a distance metric.

This model adjusts privacy depending on data distance, augmenting utility for
datasets like location or smart meter consumption that are sensitive to
distance.

####  IV-A4 Local Information Privacy

Local Information Privacy (LIP) was originally proposed in [24] as a prior-
aware version of LDP, and then, in [25], Jiang et al relax the prior-aware
assumption to partial prior-aware (Bounded Prior in their version). The
definition of LIP is shown as follows:

######  Definition 3.

$(\epsilon,\delta)$-Local Information Privacy[26] A mechanism ${M}$ satisfies
$(\epsilon,\delta)$-LIP, if $\forall{x\in{\mathcal{X}}}$,
$y\in{\textit{Range}(\mathcal{M})}$:

$\displaystyle P(Y=y)\geq e^{-\epsilon}P(Y=y|X=x)-\delta,$ (6) $\displaystyle
P(Y=y)\leq{e^{\epsilon}P(Y=y|X=x)}+\delta.$

####  IV-A5 Sequential Information Privacy

Sequential Information Privacy (SIP), built upon LIP, measures the privacy
leakage for a data sequence, or time series data. SIP naturally decomposes
using chain rule-similar techniques and is comparable to that of LDP.

######  Definition 4.

$[(\epsilon)$-Sequence Information Privacy][27] A mechanism $\mathcal{M}$
satisfies $(\epsilon)$-SIP for some $\epsilon\in{\mathds{R}^{+}}$, if
$\forall{X_{1}^{T}\in{\mathcal{X}}}$,
$Y_{1}^{T}\in{\textit{Range}(\mathcal{M})}$:

$\displaystyle
e^{-\epsilon}\leq\frac{P[M(x_{1}^{T})=y_{1}^{T}]}{P[X_{1}^{T}=x_{1}^{T}]}\leq{e^{\epsilon}}$
(7)

The operational meaning of LIP is, the output $Y$ provides limited additional
information about any possible input $X$, and the amount of the additional
information is measured by the privacy budget $\epsilon$ and failure
probability $\delta$.

In [28], multiple LIP mechanisms were proposed and testified, showing that
even though $\epsilon$-LIP is stronger than $2\epsilon$-LDP in terms of
privacy protection. The mechanisms achieve more than 2 times of utility gain.

####  IV-A6 CLDP

Recognizing LDP’s diminished utility with fewer users, Gursoy et al. [29]
introduced the metric-based model of condensed local differential privacy
(CLDP).

######  Definition 5 ($\alpha$-CLDP).

For all input pairs $v$ and $v^{\prime}$ in $M$’s domain and any potential
output $y\in Y$, a randomized function $M$ satisfies $\alpha$-CLDP if:

$P[M(v)=y]\leq e^{\alpha\cdot d(v,v^{\prime})}\cdot P[M(v^{\prime})=y],$

where $\alpha>0$.

In CLDP, a decline in $\alpha$ compensates for a growth in distance $d$.
Gursoy et al. employed an Exponential Mechanism variant to devise protocols,
particularly benefitting scenarios with limited users.

####  IV-A7 PLDP

PLDP [30] offers user-specific privacy levels. Here, users can modify their
privacy settings, denoted by $\varepsilon$.

######  Definition 6 ($\varepsilon$-PLDP).

For a user $U$, and all input pairs $v$ and $v^{\prime}$ in $M$’s domain and
any potential output $y\in Y$, a randomized function $M$ meets
$\varepsilon_{U}$-PLDP if:

$P[M(v)=y]\leq e^{\varepsilon_{U}}\cdot P[M(v^{\prime})=y].$

Approaches like the personalized count estimation protocol and advanced
combination strategy cater to users with varying privacy inclinations.

####  IV-A8 Utility-optimized LDP (ULDP)

Traditional LDP assumes all data points have uniform sensitivity, often
causing excessive noise addition. Recognizing that not all personal data have
equivalent sensitivity, the Utility-optimized LDP (ULDP) model was proposed.
In this model, let $KS\subseteq K$ be the sensitive dataset and $KN=K\setminus
KS$ be the non-sensitive dataset. Let $Y_{P}\subseteq Y$ be the protected
output set and $Y_{I}=Y\setminus Y_{P}$ be the invertible output set. The
formal definition of ULDP is:

######  Definition 7.

Given $KS\subseteq K$, $Y_{P}\subseteq Y$, a mechanism $M$ adheres to
$(KS,Y_{P},\epsilon)$-ULDP if:

  * •

For every $y\in Y_{I}$, there is a $v\in KN$ with $P[M(v)=y]>0$ and
$P[M(v^{\prime})=y]=0$ for any $v^{\prime}\neq v$.

  * •

For all $v,v^{\prime}\in K$ and $y\in Y_{P}$, $P[M(v)=y]\leq e^{\epsilon}\cdot
P[M(v^{\prime})=y]$.

In simpler terms, $(KS,Y_{P},\epsilon)$-ULDP ensures that sensitive inputs are
mapped only to the protected output set.

####  IV-A9 Input-Discriminative LDP (ID-LDP)

While ULDP classifies data as either sensitive or non-sensitive, the ID-LDP
model offers a more nuanced approach by acknowledging varying sensitivity
levels among data. It is defined as:

######  Definition 8.

Given a set of privacy budgets $E=\\{\epsilon_{v}\\}_{v\in K}$, a mechanism
$M$ adheres to $E$-ID-LDP if for all input pairs $v$ and $v^{\prime}$, and any
output $y\in Y$:

$P[M(v)=y]\leq e^{r(\epsilon_{v},\epsilon_{v^{\prime}})}\cdot
P[M(v^{\prime})=y]$

where $r(\cdot,\cdot)$ is a function of two privacy budgets.

The study in [31] primarily utilizes the minimum function between
$\epsilon_{v}$ and $\epsilon_{v^{\prime}}$ and introduces the MinID-LDP as a
specialized case.

####  IV-A10 Parameter Blending Privacy (PBP)

PBP was proposed as a more flexible LDP variant [32]. In PBP, let $\Theta$
represent the domain of privacy parameters. Given a privacy budget
$\theta\in\Theta$, let $P(\theta)$ denote the frequency with which $\theta$ is
selected. PBP is defined as:

######  Definition 9.

A mechanism $M$ adheres to $r$-PBP if, for all
$\theta\in\Theta,v,v^{\prime}\in K,y\in Y$, there exists a
$\theta^{\prime}\in\Theta$ such that:

$P(\theta)P[M(v;\theta)=y]\leq e^{r(\theta)}\cdot
P(\theta^{\prime})P[M(v^{\prime};\theta^{\prime})=y]$

###  IV-B A Summary of LDP variants

Local Differential Privacy (LDP) is a foundational approach tailored for all
data types and operates using the randomized response (RR) technique. Its
primary advantage is its broad applicability, but it may add more noise than
necessary, especially when not all data attributes have the same sensitivity
levels. To address this, approximate LDP, which allows for minor violations in
privacy guarantees, introduces flexibility. However, this relaxation can be a
double-edged sword, potentially compromising privacy in highly sensitive
scenarios.

BLENDER, on the other hand, is crafted explicitly for categorical data. By
synergizing aspects of both global Differential Privacy and LDP, it aims to
improve data utility. Yet, its reliance on grouping user data might introduce
challenges in dynamic or constantly changing environments. Local d-privacy is
another variant, designed with metric spaces in mind. It’s particularly
beneficial for data like location points, but may not be the first choice for
other data structures due to its specific metric-based method.

CLDP stands out for its unique approach to address challenges that arise with
smaller user counts, an often overlooked but crucial aspect in privacy.
However, while it addresses issues in smaller datasets, it might introduce
complexities when the user base grows, making scalability a potential concern.
PLDP, meanwhile, strives to provide a more granular level of privacy. While
this granularity is its strength, the trade-off might be a more significant
computational overhead and intricate implementation details.

ULDP takes a novel stance by focusing on optimizing utility through an
emphasis on sensitive data. The premise here is that not all data pieces hold
equal sensitivity. However, the challenge and responsibility of correctly
categorizing which data is sensitive can be daunting. ID-LDP further refines
this concept by providing protection based on the actual sensitivity of the
input, using unary encoding to achieve this. Its main challenge is the
intricate parameter setting required to ensure optimal performance. Lastly,
PBP is distinct in its pursuit of robust privacy. By maintaining the secrecy
of provider parameters, it bolsters privacy assurances. Yet, this added layer
of secrecy might introduce complexities in implementation and understanding.

##  V Conclusion

In the realm of data privacy, Local Differential Privacy (LDP) stands out as a
vital tool for preserving user data. This research delves into various LDP
mechanisms, protocols and variants in definition, each addressing unique
challenges. From the foundational LDP to specialized versions like BLENDER for
categorical data and Local d-privacy for metrics, the spectrum of solutions is
vast. Techniques like CLDP tackle smaller datasets, while PLDP, ULDP, and ID-
LDP optimize data utility and privacy levels. The introduction of PBP
emphasizes secrecy in privacy parameters. Ultimately, this paper underscores
the importance of selecting the right LDP variant, given the specific nature
of data and privacy needs.

## References

  * [1] European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council. [Online]. Available: https://data.europa.eu/eli/reg/2016/679/oj
  * [2] R. MEYER. (2014) Facebook’s mood manipulation experiment might have been illegal. [Online]. Available: https://www.theatlantic.com/technology/archive/2014/09/facebooks-mood-manipulation-experiment-might-be-illegal/380717/
  * [3] “No free lunch in data privacy,” in _Proceedings of SIGMOD 2011 and PODS 2011_ , ser. Proceedings of the ACM SIGMOD International Conference on Management of Data. Association for Computing Machinery, 2011, pp. 193–204.
  * [4] C. Dwork, F. McSherry, and K. Nissim, “Calibrating noise to sensitivity in private data analysis,” in _Theory of Cryptography: Third Theory of Cryptography Conference_ , 2006, pp. 265–284. [Online]. Available: https://doi.org/10.1007/11681878_14
  * [5] C. Dwork, “Differential privacy,” in _Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Part II_ , M. Bugliesi, B. Preneel, V. Sassone, and I. Wegener, Eds., 2006, pp. 1–12. [Online]. Available: https://doi.org/10.1007/11787006_1
  * [6] Y. Xiao and L. Xiong, “Protecting locations with differential privacy under temporal correlations,” in _Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’15, New York, NY, USA, 2015, p. 1298–1309.
  * [7] W. Primoff and S. Kess, “The equifax data breach: What cpas and firms need to know now,” _The CPA Journal_ , vol. 87, no. 12, pp. 14–17, 2017.
  * [8] J. Lu, “Assessing the cost, legal fallout of capital one data breach,” _Legal Fallout Of Capital One Data Breach (August 15, 2019)_ , 2019.
  * [9] C. Dwork, “Differential privacy: A survey of results,” in _Theory and Applications of Models of Computation: 5th International Conference, TAMC_ , M. Agrawal, D. Du, and Z. Duan, Eds., 2008, pp. 1–19. [Online]. Available: https://doi.org/10.1007/978-3-540-79228-4_1
  * [10] Úlfar Erlingsson, V. Pihur, and A. Korolova, “Rappor: Randomized aggregatable privacy-preserving ordinal response,” in _Proceedings of the 21st ACM CCS_ , 2014. [Online]. Available: https://arxiv.org/abs/1407.6981
  * [11] A. Greenberg, “Apple’s ‘differential privacy’ is about collecting your data—but not your data,” 2016. [Online]. Available: https://www.wired.com/2016/06/apples-differential-privacy-collecting-data/
  * [12] T. Wang, J. Blocki, N. Li, and S. Jha, “Locally differentially private protocols for frequency estimation,” in _26th USENIX Security 17_. USENIX Association, 2017, pp. 729–745. [Online]. Available: https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/wang-tianhao
  * [13] Z. Qin, Y. Yang, and T. Yu, “Heavy hitter estimation over set-valued data with local differential privacy,” in _Proceedings of the 2016 ACM SIGSAC_ , ser. CCS ’16, 2016, pp. 192–203. [Online]. Available: http://doi.acm.org/10.1145/2976749.2978409
  * [14] C. Huang, P. Kairouz, X. Chen, L. Sankar, and R. Rajagopal, “Context-aware generative adversarial privacy,” _CoRR_ , vol. abs/1710.09549, 2017. [Online]. Available: http://arxiv.org/abs/1710.09549
  * [15] P. Kairouz, S. Oh, and P. Viswanath, “The composition theorem for differential privacy,” _IEEE Transactions on Information Theory_ , vol. 63, no. 6, pp. 4037–4049, 2017.
  * [16] S. L. Warner, “Randomized response: A survey technique for eliminating evasive answer bias,” _Journal of the American Statistical Association_ , vol. 60, no. 309, pp. 63–69, 1965. [Online]. Available: http://www.jstor.org/stable/2283137
  * [17] C. Dwork, M. Naor, T. Pitassi, and G. N. Rothblum, “Differential privacy under continual observation,” in _Proceedings of the Forty-Second ACM Symposium on Theory of Computing_ , ser. STOC ’10, New York, NY, USA, 2010, p. 715–724.
  * [18] E. Bozkir, O. Günlü, W. Fuhl, R. Schaefer, and E. Kasneci, “Differential privacy for eye tracking with temporal correlations,” _PLOS ONE_ , vol. 16, p. e0255979, 08 2021.
  * [19] R. Bassily, “Linear queries estimation with local differential privacy,” _CoRR_ , vol. abs/1810.02810, 2018. [Online]. Available: http://arxiv.org/abs/1810.02810
  * [20] B. Avent, A. Korolova, D. Zeber, T. Hovden, and B. Livshits, “BLENDER: Enabling local search with a hybrid differential privacy model,” in _26th USENIX Security Symposium (USENIX Security 17)_. Vancouver, BC: USENIX Association, Aug. 2017, pp. 747–764. [Online]. Available: https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/avent
  * [21] C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor, “Our data, ourselves: Privacy via distributed noise generation,” in _Advances in Cryptology - EUROCRYPT 2006, 25th Annual International Conference on the Theory and Applications of Cryptographic Techniques, St. Petersburg, Russia, May 28 - June 1, 2006, Proceedings_ , ser. Lecture Notes in Computer Science, S. Vaudenay, Ed., vol. 4004. Springer, 2006, pp. 486–503. [Online]. Available: https://doi.org/10.1007/11761679\\_29
  * [22] M. E. Andrés, N. E. Bordenabe, and K. Chatzikokolakis, “Geo-indistinguishability: Differential privacy for location-based systems,” _CoRR_ , vol. abs/1212.1984, 2012. [Online]. Available: http://arxiv.org/abs/1212.1984
  * [23] M. S. Alvim, K. Chatzikokolakis, C. Palamidessi, and A. Pazii, “Metric-based local differential privacy for statistical applications,” _CoRR_ , vol. abs/1805.01456, 2018. [Online]. Available: http://arxiv.org/abs/1805.01456
  * [24] B. Jiang, M. Li, and R. Tandon, “Context-Aware data aggregation with localized information privacy,” in _2018 IEEE Conference on Communications and Network Security (CNS)_ , May 2018.
  * [25] ——, “Local information privacy with bounded prior,” in _ICC 2019-2019 IEEE International Conference on Communications (ICC)_ , May 2019, pp. 1–7.
  * [26] ——, “Local information privacy and its application to privacy-preserving data aggregation,” _IEEE Transactions on Dependable and Secure Computing_ , vol. 19, no. 3, pp. 1918–1935, 2022.
  * [27] ——, “Online context-aware data release with sequence information privacy,” 2023.
  * [28] B. Jiang, M. Seif, R. Tandon, and M. Li, “Context-aware local information privacy,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 3694–3708, 2021.
  * [29] M. E. Gursoy, A. Tamersoy, S. Truex, W. Wei, and L. Liu, “Secure and utility-aware data collection with condensed local differential privacy,” _CoRR_ , vol. abs/1905.06361, 2019. [Online]. Available: http://arxiv.org/abs/1905.06361
  * [30] Y. NIE, W. Yang, L. Huang, X. Xie, Z. Zhao, and S. Wang, “A utility-optimized framework for personalized private histogram estimation,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 31, no. 4, pp. 655–669, 2019.
  * [31] T. Murakami and Y. Kawamoto, “Utility-optimized local differential privacy mechanisms for distribution estimation,” in _Proceedings of the 28th USENIX Conference on Security Symposium_ , ser. SEC’19. USA: USENIX Association, 2019, p. 1877–1894.
  * [32] S. Takagi, Y. Cao, and M. Yoshikawa, “Poster: Data collection via local differential privacy with secret parameters,” in _Proceedings of the 15th ACM Asia Conference on Computer and Communications Security_ , ser. ASIA CCS ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 910–912. [Online]. Available: https://doi.org/10.1145/3320269.3405441