Paper Title: Explainable and Controllable Motion Curve Guided Cardiac Ultrasound Video Generation.







Cardiac Ultrasound Video Generation




 


1National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Medical School,
Shenzhen University, China

xinyang@szu.edu.cn; nidong@szu.edu.cn 

2Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, China

3Marshall Laboratory of Biomedical Engineering, Shenzhen University, China

4Faculty of Applied Sciences Macao Polytechnic University, Macao, China 

5Shenzhen RayShape Medical Technology Co., Ltd, China




J.Yu and R.Chen et al.


Explainable and Controllable Motion Curve Guided Cardiac Ultrasound Video Generation
    Junxuan Yu1,2,3Junxuan Yu and Rusi Chen contribute equally to this work. Rusi Chen1,2,3⋆ Yongsong Zhou1,2,3 Yanlin Chen1,2,3 Yaofei Duan4 Yuhao Huang1,2,3 Han Zhou1,2,3,5 Tan Tao4 Xin Yang1,2,3() Dong Ni1,2,3()
    August 5, 2024
======================================================================================================================================================================================================================



§ ABSTRACT 
Echocardiography video is a primary modality for diagnosing heart diseases, but the limited data poses challenges for both clinical teaching and machine learning training.
Recently, video generative models have emerged as a promising strategy to alleviate this issue.
However, previous methods often relied on holistic conditions during generation, hindering the flexible movement control over specific cardiac structures.
In this context, we propose an explainable and controllable method for echocardiography video generation, taking an initial frame and a motion curve as guidance. 
Our contributions are three-fold.
First, we extract motion information from each heart substructure to construct motion curves, enabling the diffusion model to synthesize customized echocardiography videos by modifying these curves.
Second, we propose the structure-to-motion alignment module, which can map semantic features onto motion curves across cardiac structures.
Third, The position-aware attention mechanism is designed to enhance video consistency utilizing Gaussian masks with structural position information.
Extensive experiments on three echocardiography datasets show that our method outperforms others regarding fidelity and consistency. The full code will be released at https://github.com/mlmi-2024-72/ECMhttps://github.com/mlmi-2024-72/ECM.







§ INTRODUCTION

Echocardiography is a primary method that relies on dynamic video to obtain structural information for clinical diagnoses <cit.>. 
However, training radiologists with diagnostic skills and establishing machine learning models both suffer from limitations on video resources.
Recently, video generation models have demonstrated a promising ability to solve this problem, owing to their powerful capability in modeling data distribution <cit.>.
Several studies about video generation based on specific conditions have been investigated.
The typical ones relied on canny edges or depth maps <cit.> extracted from additional videos as conditions. Nevertheless, these conditions are non-editable and lack motion-driving information.
In contrast, Shi et al. <cit.> proposed the Motion-I2V framework to predict dense optical flow and guide video generation, maintaining both spatial and motion consistency.
Wang et al. <cit.> then employed a self-tracking training method.
Specifically, they specified the box positions of the first and last frames along with motion paths, to control the movement of objects.
However, these paths are basic and coarse, inadequate for ultrasound video synthesis that demands an accurate representation of intricate structure motions.





In the field of echocardiography video generation, 
Zhou et al. <cit.> proposed the OnUVS framework to synthesize ultrasound videos by animating source images and leveraging motion information from driving video. 
Nevertheless, OnUVS faced challenges due to the anatomical structure gap between the source image and the driving video, making accurate motion control difficult.
In addition, some studies have mined the structural information of the heart to guide the movement of videos.
For instance, Reynaud et al. <cit.> developed a Generative Adversarial Network (GAN) capable of generating echocardiography videos corresponding to the left ventricular ejection fractions (LVEFs).
They further employed a cascade video diffusion model conditioned on randomly sampled frames, enhancing the synthesis quality of echocardiography video <cit.>.
Van et al. <cit.> utilized segmentation masks of end-diastolic (ED) frames as a condition to generate four-chamber heart videos.
However, both LVEFs and ED masks are relatively sparse conditions, leading to an imbalance of information between intricate motions and limited conditions. 
This sparsity poses challenges for effectively controlling fine-grained cardiac structures, thus limiting the ability to capture the full complexity of heart movements and dynamics.

To address the above issues, we propose an explainable and controllable motion curve guided video diffusion model (ECM) that can synthesize video guided by the initial frame and motion curves (Fig. <ref>). 
Our contributions are threefold: 
(1) We innovatively mine the motion of the echocardiography video to obtain motion curves, which fully reflect the movement of each cardiac structure. This easily controlled approach enables the customization of videos through the modification (scaling and replacing) of the initial motion curve.
(2) As the curve lacks category information of the structure, we propose a Structure-to-Motion alignment mechanism. This mechanism extracts the semantic features of each cardiac structure and maps them with motion curve features, aiming to align visual and motion information effectively.
(3) We design position-aware attention masks based on the position of the cardiac structure movement, effectively enhancing the motion consistency of each structure.
To the best of our knowledge, ECM is the first study to apply cardiac motion curve guidance in echocardiography video generation. Extensive experimental results show that the proposed ECM is a flexible, controllable, and reliable method.




§ METHODOLOGY




Fig. <ref> shows the framework of ECM. During the training process, ECM takes an original video as input. 
A pretrained variational auto-encoder (VAE) from Stable Diffusion (SD) <cit.> is then utilized to downsample the video into latent features. 
By gradually adding noise to the latent and then learning to denoise it, the model can obtain the generated video with a pretrained VAE decoder. 
Remarkably, we develop the structure-to-motion alignment module to match cardiac structures with motion, yielding aligned features that condition the ECM model. 
Additionally, we employ Gaussian masks for each structure using a Position-aware Attention Mechanism, incorporating these into the spatial layers of the diffusion model. 
During the testing process, ECM generates echocardiography videos using an initial frame and motion curves as input. Users can customize the videos by replacing the initial frame or adjusting (scaling or replacing) the motion curves, highlighting the controllability and interpretability of our method.




 §.§ Extraction of Motion Curves

Previous studies <cit.> faced challenges in controlling the motion of specific cardiac substructures since they relied solely on the single sparse condition (LVEFs), while our approach aims to provide fine-grained control over the motion of each cardiac substructure.
Therefore, we extract motion curves for the key structures (e.g., Left Ventricle, Left Atrium, Mitral Valve, etc.) as conditions.

As illustrated in Fig. <ref>, the process of extracting motion curves is as follows: 
(a) We employ a well-trained anatomy detector (average accuracy=85%) to identify each substructure in each frame of the echocardiogram video. 
(b) Subsequently, we utilize the pixel coordinates of each substructure's bounding box (bbox) as the basis for encoding the motion curves, which form the basis for encoding the motion curves, represented as 
f_c^m∈ℝ^B × N × C × ( 4 × 2),
where N represents the number of frames and C represents the categories of the substructure. Notably, any missed detected structures would be treated as learnable parameters, initialized by the network.
(c) Since cardiac motion is periodic, we employ Fourier transformation (FT) to transform the pixel coordinates into a high-dimensional feature representation, denoted as 
f_c^m∈ℝ^B × N × C × E, 
where E represents the dimensionality of the features obtained from FT. 
(d) Finally, the motion embedding is passed through several multi-layer perceptron (MLP) layers. Overall, these motion curve features can be formulated as f_c^m∈ℝ^B × N × C × 1024.



 


 §.§ Structure-to-Motion Alignment Module


Although the motion curves of the echocardiography videos are captured, it is challenging for the model to distinguish the relationship between the motion curves and the semantic information of each substructure.
Recently, GLIGEN <cit.> has demonstrated its effectiveness in combining caption and bbox information, enhancing visual-language understanding to enable fine-grained control over specific objects in natural images.
However, our preliminary experiments suggested that GLIGEN struggles to effectively represent and interpret texts related to cardiac structures in echocardiography.



Innovatively, we replace texts with the cardiac structure features, denoted as f_i^s. 
As shown in Fig. <ref>, to obtain f_i^s, 
the regions of interest (ROI) that correspond to the cardiac substructures in the initial frame are cropped by the well-pretrained detector as mentioned above. 
Then, a pretrained CLIP image encoder <cit.> is utilized to transform the ROIs into structural embedding features.
Then, these features pass through several MLP layers,
resulting in an output denoted as f_c^s∈ℝ^B × N × C × 1024, which has the same shape with the extracted motion curve features f_i^m mentioned in Sec. <ref>.
Consequently, structural features and their corresponding motion curve features from the same category are concatenated to create aligned features. The aligned features are formulated as:

    F_c^a = Concat(f_c^s, f_c^m).


Note that any undetected structures are replaced with general features from the dataset.
Next, the concatenated features pass through an additional MLP layer for further integration.
Finally, to introduce the aligned motion curve features to guide the generation of echocardiography video, we then mapped 
F_c^a to the intermediate spatial layers of the UNet<cit.> via a cross-attention layer implementing cross attention, formulated as:

    CrossAtten(Q, K, V) = softmax( QK^T/√(d)) · V,

where Q, K and V represent the query, key and value respectively in the attention mechanism. Here, we regard noise latent feature F_l^m∈ℝ^B × 64 × 64 as query and the structure-to-motion embedding features as key and value. 






 §.§ Position-aware Attention Mechanism

Enhancing the consistency of cardiac motion is crucial in the task of cardiac video generation. 
To address this, we aim to inject the positional information of the cardiac structure into the cross-attention mechanism. Specifically, we design Gaussian masks that are generated based on the position of the cardiac structures. 
The Gaussian masks are defined as:

    M_g(x, y) = 1/2πσ^2exp(-(x - μ_x)^2 + (y - μ_y)^2/2σ^2),

where x and y represent the spatial positions of the mask.
μ_x and μ_y denote the center positions of the Gaussian distribution, which are the coordinates of the four corner points of the detected bbox. 
σ is the standard deviation to control the distribution width. 
Here, we set σ to 10 pixels as default.
Subsequently, the Gaussian masks are resized to match the dimensions of the latent feature map. 
The Gaussian-weighted cross-attention mechanism is as follows:

    CrossAttenMask(Q, K, V, M_g) = softmax( QK^T/√(d_k)⊙M_g ) V,

where ⊙ is the element-wise multiplication. Overall, this position-aware attention mechanism effectively integrates positional information of cardiac structures, enhancing the consistency and realism of generated cardiac motion.





§ EXPERIMENTS


Datasets and Implementations. 
To assess the performance of ECM, we gathered data from three sources: two private dataset from multiple hospitals and the publicly available dataset named EchoNet-Dynamic <cit.>.
The in-house dataset includes 144 apical four-chamber (A4C) and 100 apical two-chamber (A2C) heart videos, and the public one comprises 10,030 labeled A4C echocardiography videos.
For both datasets, the videos were randomly split into training (90%) and testing (10%) sets. During training, 12-frame clips were randomly sampled from each video, with a sampling interval ranging from 1 to 4 frames. 
For testing, videos were truncated according to different sampling intervals. 
The input videos from the private datasets were resized to 256x256 pixels, while those from the EchoNet-Dynamic dataset were kept at their original resolution of 112x112 pixels.
All methods were implemented in PyTorch using an NVIDIA RTX 4090 GPU under same settings. The Adam optimizer was used with a learning rate of 5e-3 and 60,000 training steps.

Evaluation Metrics.
Our evaluation metrics cover both image-level and video-level evaluation.
The image-level assessment includes Structural Similarity Index (SSIM) <cit.>, Mean Absolute Error (MAE) <cit.>, Peak Signal-to-Noise Ratio (PSNR) <cit.>, Fréchet Inception Distance (FID) <cit.> and Learned Perceptual Image Patch Similarity (LPIPS) <cit.>.
For video-level assessment, we only considered the commonly-used Fréchet Video Distance (FVD) <cit.>.
Notably, to assess the consistency of cardiac structures between the generated and target videos, we introduce a new indicator calculated by the Intersection over Union (IoU) between the bboxes of the original and synthesized videos.




Method Comparison.
The quantitative comparison of ECM and other methods is reported in Table <ref>.
The ECM model achieves the best performance across all metrics, indicating superior image quality and high fidelity compared to other methods.
It can be seen that the SEG Diffusion method, generating echocardiography videos without any control conditions, achieves poor quality.
Notably, ECM model markedly outperforms the SD method under bbox/text conditions, achieving 67.8%↓ in FVD. 
This demonstrates that synthesizing 2D images with rich conditions and stitching them into videos does not yield satisfactory outcomes. 
The integration of a position-aware attention mechanism significantly contributes to our superior performance and enhanced video consistency.
Furthermore, the ECM model shows better performance than 3D ControlNet. 
This disparity arises from the fact that natural images typically have clearly defined control conditions, whereas the motion in echocardiography videos is inherently more complex.
Similarly, for the EchoNet-Dynamic dataset, guided by motion curves, our proposed ECM outperforms the strong competitor <cit.> across most metrics, indicating its superior generation capacity.










Ablation Study.
We conducted the ablation study to test the contribution of each component in Table <ref>. 
The ECM model consistently outperforms others across most metrics for both the A4C and A2C datasets, achieving outstanding FVD scores of 189.27 and 208.12, and impressive IoU scores of 0.766 and 0.828, respectively.
This highlights that ECM effectively enhances video generation consistency while maintaining strong image quality.
Additionally, incorporating motion curves instead of traditional textual control leads to notable improvements. 
Specifically, using motion curves, the FVD decreased from 307.50 to 259.77, and the IoU increased from 0.691 to 0.748.
This indicates that motion curves provide precise control over the motion in echocardiography videos.
It can also be observed that the Structure-to-Motion alignment module (`+S2M') and the position-aware attention mechanism (`+Att') enhance image quality and video motion consistency, respectively. Consequently, the final ECM model demonstrates excellent performance at both the image and video levels.



Qualitative Results. 
Fig. <ref> demonstrates that ECM generates videos closely matching the target in both the A4C and EchoNet-Dynamic datasets by inputting the initial frame and motion curves. It effectively mimics heart structure motions such as chamber dilation, contraction, and diastolic opening and systolic closing
Furthermore, Fig. <ref> (a) illustrates ECM's controllability. Starting from the same initial frame, the second-row video uses the original motion curves from target video A, while the third-row video uses replaced curves from target video B. The mitral valve's systole and diastole differ after replacing the motion curves (as shown by yellow arrows), indicating ECM's effective control over motion curves.
More visualization results, including generated videos with replaced and scaled motion curves, can be found at http://106.53.160.189:8000/anonymous link.




§ CONCLUSION

In this study, We have presented ECM for generating echocardiography videos guided by motion curves, which reflect the movement of each cardiac structure. ECM enables to customize the generated videos by adjusting (scaling and replacing) the initial motion curve. Besides, to link the structure features with corresponding movement information, we propose the structure-to-motion alignment mechanism. Moreover, attention masks based on the position of the anatomical structures are introduced to enhance the motion consistency of each structure. Overall, our proposed ECM achieves state-of-the-art performance for generating echocardiography videos in terms of fidelity and consistency.




  §.§.§ 

This work was supported by the grant from National Natural Science Foundation of China (12326619, 62101343, 62171290), Science and Technology Planning Project of Guangdong Province (2023A0505020002), Science and Technology Development Fund of Macao (0021/2022/AGJ), and Shenzhen-Hong Kong Joint Research Program (SGDX20201103095613036).



  §.§.§ 

The authors have no competing interests to declare that are relevant to the content of this article.


splncs04




 Paper Title: Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention.




Quantized electrical, thermal, and spin transports of non-Hermitian clean and dirty two-dimensional topological insulators and superconductors
    Bitan Roy
    August 5, 2024
==============================================================================================================================================




§ ABSTRACT 

Conditional diffusion models have shown remarkable success in visual content generation, producing high-quality samples across various domains, largely due to classifier-free guidance (CFG). Recent attempts to extend guidance to unconditional models have relied on heuristic techniques, resulting in suboptimal generation quality and unintended effects. In this work, we propose Smoothed Energy Guidance (SEG), a novel training- and condition-free approach that leverages the energy-based perspective of the self-attention mechanism to enhance image generation. By defining the energy of self-attention, we introduce a method to reduce the curvature of the energy landscape of attention and use the output as the unconditional prediction. Practically, we control the curvature of the energy landscape by adjusting the Gaussian kernel parameter while keeping the guidance scale parameter fixed. Additionally, we present a query blurring method that is equivalent to blurring the entire attention weights without incurring quadratic complexity in the number of tokens. In our experiments, SEG achieves a Pareto improvement in both quality and the reduction of side effects. The code is available at <https://github.com/SusungHong/SEG-SDXL>.






§ INTRODUCTION


Diffusion models <cit.> have emerged as a promising tool for visual content generation, producing high-quality and diverse samples across various domains, including image <cit.>, video <cit.>, and 3D generation <cit.>. The success of these models can be largely attributed to the use of classifier-free guidance (CFG) <cit.>, which enables sampling from a sharper distribution, resulting in improved sample quality. However, CFG is not applicable to unconditional image generation, where no specific conditions are provided, creating a disparity between the capabilities of text-conditioned sampling and sampling without text. This disparity results in a restriction in application, e.g., synthesizing images with ControlNet<cit.> without a text prompt (see the last two columns of Fig. <ref>).

Recent literature <cit.> has attempted to decouple CFG and image quality by extending guidance to general diffusion models, leveraging their inherent representations <cit.>. Self-attention guidance (SAG) <cit.> proposes leveraging the intermediate self-attention map of diffusion models to blur the input pixels and provide guidance, while perturbed attention guidance (PAG) <cit.> perturbs the attention map itself by replacing it with an identity attention map. Despite these efforts, these methods rely on heuristics to make perturbed predictions, resulting in unintended effects such as smoothed-out details, saturation, color shifts, and significant changes in the image structure when given a large guidance scale. Notably, the mathematical underpinnings of these unconditional guidance approaches are not well elucidated.




In this work, we approach the objective from an energy-based perspective of the self-attention mechanism, which has been previously explored based on its close connection to the Hopfield energy <cit.>. Specifically, we start from the definition of the energy of self-attention, where performing a self-attention operation is equivalent to taking a gradient step. In light of this, we propose a tuning- and condition-free method that reduces the curvature of the underlying energy function by directly blurring the attention weights, and then leverages the output as the negative prediction. We call this method Smoothed Energy Guidance (SEG).

SEG does not merely rely on the guidance scale parameter that cause side effects when its value becomes large. Instead, we can continuously control the original and maximally attenuated curvature of the energy landscape behind the self-attention by simply adjusting the parameter of the Gaussian kernel, with the guidance scale parameter fixed. Additionally, we introduce a novel query blurring technique, which is equivalent to blurring the entire attention weights without incurring quadratic cost in the number of tokens.

We validate the effectiveness of SEG throughout the various experiments without and with text conditions, and ControlNet <cit.> trained on canny and depth maps. Based on the attention modulation, SEG results in less structural change from the original prediction compared to previous approaches <cit.>, while achieving better sample quality.





§ PRELIMINARIES




 §.§ Diffusion models



Diffusion models <cit.> are a class of generative models that generate data through an iterative denoising process. The process of adding noise to an image 𝐱 over time t ∈ [0, T] is governed by the forward stochastic differential equation (SDE):

    d𝐱 = 𝐟(𝐱, t)dt + g(t)d𝐰,

where 𝐟 and g are predefined functions that determine the manner in which the noise is added, and d𝐰 denotes a standard Wiener process.

Correspondingly, the denoising process can be described by the reverse SDE:

    d𝐱 = [𝐟(𝐱, t) - g(t)^2 ∇_𝐱log p_t(𝐱)] dt + g(t)d𝐰̅,

where ∇_𝐱log p_t(𝐱) represents the score of the noisy data distribution and d𝐰̅ denotes the standard Wiener process for the reversed time.

Diffusion models are trained to approximate the score function with 𝐬_θ(𝐱, t) ≈∇_𝐱log p_t(𝐱). To generate an image based on a condition c, e.g., a class label or text, one simply needs to train diffusion models to approximate the conditional score function with 𝐬_θ(𝐱, t, c) ≈∇_𝐱log p_t(𝐱|c) and replace ∇_𝐱log p_t(𝐱) with it in the denoising process. To enhance the quality and faithfulness of the generated samples, classifier-free guidance (CFG) <cit.> is widely adopted. Accordingly, the reverse process becomes:

    d𝐱 = [𝐟(𝐱, t) - g(t)^2 (γ_cfg𝐬_θ(𝐱, t, c) - (γ_cfg-1) 𝐬_θ(𝐱, t))] dt + g(t)d𝐰̅.

Here, 𝐬_θ(𝐱, t) is learned by dropping the label by a certain proportion, and γ_cfg is a hyperparameter that controls the strength of the guidance. Intuitively, CFG helps us to sample from sharper distribution by conditioning on a class label or text.



 §.§ Energy-based view of attention mechanism



The attention mechanism <cit.>, which has been widely adopted in diffusion models <cit.>, has been interpreted through the lens of energy-based models (EBMs) <cit.>, especially through its close connection with the Hopfield energy <cit.>. In the modern (continuous) Hopfield network, the attention operation can be derived based on the concave-convex procedure (CCCP) from the following energy function <cit.>:

    E(ξ) = - lse(𝐗ξ^⊤) + 1/2ξξ^⊤,

where ξ∈ℝ^1× d, 𝐗∈ℝ^N× d, and lse stands for the log-sum-exp function, defined as lse(𝐯) := log(∑_i=1^N e^v_i). The quadratic term acts as a regularizer to prevent ξ from exploding <cit.>, while -lse(𝐗ξ^⊤) penalizes misalignment between 𝐗 and ξ.

Mathematically, it turns out that the attention mechanism is equivalent to the update rule of the modern Hopfield network <cit.>. Specifically, inspired by the Hopfield energy in (<ref>), and noticing that the first term depends on the attention weights, we propose the following energy function for entire self-attention weights in diffusion models:

Let 𝐐∈ℝ^(HW)× d be a matrix of query vectors and 𝐊∈ℝ^(HW)× d be a matrix of key vectors, where H, W, and d represent the height, width, and dimension, respectively. Let 𝐀∈ℝ^(HW)× (HW) := 𝐐𝐊^⊤. The energy function with respect to entire self-attention weights in diffusion models is defined as:

    E(𝐀) := ∑_i=1^H∑_j=1^W E'(𝐚_:(i,j)),
       E'(𝐚) := - lse(𝐚) = - log(∑_k=1^H ∑_l=1^W e^a_(k,l)).


Note that to explicitly denote the spatial dimension, we use the subscript (x, y) to represent the index of a row or column of the matrices. Despite using the definition in (<ref>) for the rest of the paper for simplicity, we additionally discuss the dual case, where we use the swapped indexing, in Appendix <ref>.

This view leads us to an important intuition: the attention operation can be seen as a minimization step on the energy landscape, considering that the first derivative represents the softmax operation which also appears in the attention operation. Building upon this intuition, we argue that Gaussian blurring on the attention weights modulates the underlying landscape to have less curvature, and we demonstrate this in the following sections by analyzing the second derivatives.



§ METHOD


Our aim is to theoretically derive the effect of Gaussian blur applied on the attention weights, which in the end attenuates the curvature of the underlying energy function. Then, utilizing this fact, we develop attention-based drop-in diffusion guidance that enhances the quality of the generated samples, regardless of whether an explicit condition is given. In Section <ref>, we claim some useful properties of Gaussian blur: that it preserves mean, reduces variance, and thus decreases the lse value. In Section <ref>, we find that the curvature of the energy landscape is attenuated by the attention blur operation, leading naturally to a blunter prediction for guidance. And finally, in Section <ref>, built upon this fact, we define Smoothed Energy Guidance (SEG) and propose the equivalent query blurring method, which can perform attention blurring while avoiding quadratic complexity in the number of tokens.



 §.§ Gaussian blur to attention weights



In this section, we derive some important properties of the Gaussian blur with the aim of figuring out the variation of the energy landscape. To this end, we start from some mathematical underpinnings on applying Gaussian blur to attention weights.

A 2D Gaussian filter is a convolution kernel that uses a 2D Gaussian function to assign weights to neighboring pixels. The 2D Gaussian function is defined as:

    G(x, y) = 1/2πσ^2 e^-(x-μ_x)^2 + (y-μ_y)^2/2σ^2

where μ_x and μ_y are the means in the x and y directions, and σ is the standard deviation.
The 2D Gaussian filter possesses symmetry, i.e., G(x, y) = G(-x, -y), and normalization, i.e., ∬ G(x, y)dxdy = 1. In practice, we use a discretized version of the Gaussian filter with a finite kernel size depending on σ, normalized to sum to 1.


Spatially applying a 2D Gaussian blur to the attention weights 𝐚 := 𝐐𝐤^⊤ preserves the average 𝔼_i, j[a_(i,j)]. In addition, the variance monotonically decreases every time we apply the Gaussian blur.



Proof sketch.
Applying a 2D Gaussian filter to the attention weights a_(i, j) yields the blurred values ã_(i, j):

    ã_(i,j) = ∑_m=-k^k∑_n=-k^k G(m, n) · a_(i+m, j+n)

where k is the filter size, G(m, n) is the Gaussian filter value at position (m, n), and a_(i+m, j+n) is the attention weight at position (i+m, j+n). Since the Gaussian filter is symmetric and normalized, it can be shown that the mean of the blurred attention weights is equal to the mean of the original attention weights. Similarly, we can show that the variance monotonically decreases when we apply a 2D Gaussian filter. See Appendix <ref> for the complete proof.

Note that this fact also implies that blurring with a Gaussian filter with a larger standard deviation causes a greater decrease in the variance of attention weights. This is because a Gaussian filter with a larger standard deviation can always be represented as a convolution of two filters with smaller standard deviations, due to the associativity of the convolution operation.

Finally, we show that applying a 2D Gaussian blur to attention weights increases the lse value in (<ref>), i.e., increases the energy in (<ref>). This provides a bit of intuition about the underlying energy landscape, yet it is more prominently utilized in the claims in the following sections.


Applying a 2D Gaussian blur to attention weights 𝐚 := 𝐐𝐤^⊤ increases the lse term when we consider the second-order Taylor series approximation of the exponential function around the mean μ := 𝔼_i,j[a_(i,j)]. Consequently, the maximum is achieved when the attention is uniform, i.e., a_(i,j) = a_(k,l) ∀ i,j,k,l. This corresponds to the case when we apply the Gaussian blur with σ→∞.



Proof sketch. Applying the second-order Taylor series approximation around the mean μ, and using Proposition <ref>, we show that the second-order approximation of lse(𝐚) is larger than or equal to that of lse(𝐚̃). Subsequently, we introduce Lagrange multipliers to find the maximum, which gives us the result, a_(i,j) = a_(k,l) ∀ i,j,k,l. We leave the full proof in Appendix <ref>.




 §.§ Analysis of the energy landscape



In this section, we demonstrate that applying a 2D Gaussian blur to the attention weights before the softmax operation results in computing the updated value with reduced curvature of the underlying energy function. To this end, we analyze the Gaussian curvature before and after blurring the attention weights. This is closely related to the Hessian of the energy function.


Let the attention weights be defined as 𝐚 := 𝐐𝐤^⊤. Consider the energy function in (<ref>). Then, applying a Gaussian blur to the attention weights 𝐚 before the softmax operation results in the attenuation of the Gaussian curvature of the underlying energy function where gradient descent is performed.



Proof sketch.
Let 𝐇 denote the Hessian of the original energy function, i.e., the derivative of the negative softmax, and 𝐇̃ denote the Hessian of the new energy function associated with blurred attention weights. Furthermore, let b_ij denote the i-th row, j-th column entry in the Toeplitz matrix 𝐁 representing the Gaussian blur. Calculating the derivatives, we have the elements of the Hessians, h_ij = (ξ(𝐚)_i - δ_ij)ξ(𝐚)_j and h̃_ij = (ξ(𝐚̃)_i - δ_ij)ξ(𝐚̃)_j b_ij. Using Lemmas <ref> and <ref> and under reasonable assumptions, we observe that |(𝐇)| > |(𝐇̃)|, which implies that the minimization step is performed on a smoother energy landscape with attenuated Gaussian curvature. The full proof is in Appendix <ref>.

To provide more intuition about what is actually happening and how we utilize this property in the later section, it is intriguing to consider the attenuating effect on the curvature in analogy to classifier-free guidance (CFG). CFG uses the difference between the prediction based on the sharper conditional distribution and the prediction based on the smoother unconditional distribution to guide the sampling process. By analogy, we propose a method to make the landscape of the energy function smoother to guide the sampling process, as opposed to the original (sharper) energy landscape.

From a probabilistic perspective, this process can be thought of as maximizing the likelihood of the attention weights in terms of the Boltzmann distribution conditioned on a given configuration, i.e., the feature map. Blurring the attention weights diminishes this likelihood as shown in Lemma <ref>, and also reduces the curvature of the distribution as shown in Theorem <ref>.




 §.§ Smoothed energy guidance for diffusion models



Based on the above observation that the Gaussian blur on attention weights attenuates the curvature of the energy function, we propose Smoothed Energy Guidance (SEG) in this section. For brevity, we redefine the unconditional score prediction as 𝐬_θ(𝐱, t), and the unconditional score prediction with the energy curvature reduced as 𝐬̃_θ(𝐱, t). Specifically, 𝐬̃_θ(𝐱, t) is the prediction with the attention weights blurred using a 2D Gaussian filter G with the standard deviation σ. We formulate the process as:

    (𝐐𝐊^⊤)_seg = G * (𝐐𝐊^⊤),

where * denotes the 2D convolution operator. Then, we replace the original attention weights with (𝐐𝐊^⊤)_seg and compute the final value as in ordinary self-attention.

For practical purposes when the number of tokens is large, we propose an efficient computation of (<ref>) using the property of a linear map, since the convolution operation is linear. Concretely, blurring queries is exactly the same as blurring the entire attention weights, and we propose the following proposition to justify our claim.


Let 𝐐 and 𝐊 be the query and key matrices in self-attention, and let G be a 2D Gaussian filter. Blurring the attention weights with G is equivalent to blurring the query matrix 𝐐 with G and then computing the attention weights.




Since the convolution operation is linear, we can always find a Toeplitz matrix 𝐁 such that:

    G ∗ (𝐐𝐊^⊤) = 𝐁(𝐐𝐊^⊤),

where ∗ denotes the 2D convolution operation. Using the properties of matrix multiplication, we can rewrite (<ref>) as:

    𝐁(𝐐𝐊^⊤) = (𝐁𝐐)𝐊^⊤ = (G ∗𝐐)𝐊^⊤.


Finally, SEG is formulated as follows:

    d𝐱 = [𝐟(𝐱, t) - g(t)^2 (γ_seg𝐬_θ(𝐱, t) - (γ_seg - 1) 𝐬̃_θ(𝐱, t))] dt + g(t)d𝐰̅,

where γ_seg denotes the guidance scale of SEG.

In a straightforward manner, as SEG does not rely on external conditions, it can be used for conditional sampling strategies such as CFG <cit.> and ControlNet <cit.>. For the combinatorial sampling with CFG, following <cit.>, we simply extend (<ref>) for improved conditional sampling with both SEG and CFG as follows:

    d𝐱 = [𝐟(𝐱, t) - g(t)^2 ((1-γ_cfg+γ_seg) 𝐬_θ(𝐱, t) + γ_cfg𝐬_θ(𝐱, t, c) - γ_seg𝐬̃_θ(𝐱, t))] dt + g(t)d𝐰̅,

which is an intuitive result, as the update rule moves x towards the conditional prediction while keeping it far from the prediction with blurred attention weights.

We are likely to get a result with saturation when using a large guidance scale, such as with classifier-free guidance (CFG) <cit.>, self-attention guidance (SAG) <cit.>, and perturbed attention guidance (PAG) <cit.>. This is a significant caveat since we need to increase the scale to achieve a maximum effect with these methods. Contrary to this, we can fix the scale of SEG as justified in Sec. <ref> and control its maximum effect through σ of the Gaussian blur, making the choice more flexible. For σ, two extreme cases are recognized. If σ→ 0, the blurred attention weights remain the same as the original, while when σ→∞, the attention weights merely adopt a single mean value across spatial axes. We find that even the latter extreme case results in a high-quality outcome, corroborating that we can control the quality to the limit without saturation.












§ DISCUSSION ON RELATED WORK




Classifier-free guidance (CFG) <cit.>, first proposed as a replacement for classifier guidance (CG) <cit.> is controlled by a scale parameter. The higher we set classifier-free guidance, the more we get faithful, high-quality images. However, it requires external labels, such as text <cit.> or class <cit.> labels, making it impossible to apply to unconditional diffusion models. Also, it requires specific traning procedure with label dropping and it is known that high CFG causes saturation <cit.>.

Tackling the caveats of CFG, unconditional approaches such as self-attention guidance (SAG) <cit.> and perturbed attention guidance (PAG) <cit.> have been proposed. SAG selectively blurs images with the mask obtained from the attention map and guides the generation process given the prediction. This indirect approach causes saturation and noisy images when given a large guidance scale, leading to the selection of a guidance scale less than or equal to 1. PAG guides images using prediction with identity attention, where the attention map is an identity matrix. However, the reliance on heuristics to make perturbed predictions results in unintended side effects. As an example of the side effects of replacing the attention map with identity attention, PAG changes the visual structure and color distribution of an image, as evidenced in Figs. <ref>, <ref>, and <ref>.

Contrary to these, we control the effect of SEG through the standard deviation of the Gaussian filter, σ. Moreover, while being theory-inspired, SEG is relatively free from unintended effects. In the following section, we corroborate our claim with extensive experiments.




§ EXPERIMENTS





 §.§ Implementation details


We build upon the current open-source state-of-the-art diffusion model, Stable Diffusion XL (SDXL) <cit.>, as our baseline, and do not change the configuration. To sample with SEG, we choose the same attention layers (mid-blocks) and guidance scale as PAG <cit.>. For SEG and PAG sampling, we use the Euler discrete scheduler <cit.>, while for SAG <cit.>, we instead use the DDIM scheduler <cit.> since the current implementation of SAG does not support the Euler discrete sampler. For SAG and PAG, we use the same configurations they used in the experiments with the previous version of Stable Diffusion, with guidance scales of 1.0 and 3.0, respectively. We set γ_seg to 3.0, except in the ablation study.















 §.§ Metrics


We use various metrics to evaluate quality (FID <cit.> and CLIP score <cit.>, calculated with 30k references from the MS-COCO 2014 validation set <cit.>) and to assess the extent of change due to applied guidance (LPIPS_vgg, alex <cit.>). The latter metric, calculated using the outputs of vanilla SDXL, measures the extent of side effects by comparing guided images to their unguided counterparts.



 §.§ Controlling image generation with the standard deviation


In this section, our aim is to demonstrate that with SEG, we can sample plausible images using vanilla SDXL <cit.> under various conditions and even without any conditions, as demonstrated in Fig. <ref>. Furthermore, without the risk of saturation, we can control the quality and plausibility of the samples. For the results, we use σ∈{1, 2, 5, 10}. Additionally, as mentioned in Sec. <ref>, we present two extreme cases, σ→ 0 (vanilla SDXL) and σ→∞ (uniform queries).



  
Unconditional generation

In this section, our aim is to demonstrate that with SEG, we can sample plausible images from the unconditional mode of the vanilla SDXL, which was originally trained on a large-scale text-to-image dataset. The results are presented in Fig. <ref>, Fig. <ref>, and Table <ref>. The results show a clear tendency to draw higher quality samples by utilizing the differences between the two energy landscapes with different curvatures derived from self-attention mechanisms.

In Fig. <ref> and Fig. <ref>, we show the effectiveness of generating more plausible images, while vanilla SDXL is unable to generate high-quality images without any conditions. The results show a clear tendency to draw higher quality samples by utilizing the differences between the two energy landscapes with different curvatures derived from self-attention mechanisms. When σ is larger, the definition and expression of the samples improve, as the difference in curvature becomes more pronounced.



  
Conditional generation

In Figs. <ref>, <ref>, <ref>, <ref>, and <ref>, we display sampling results conditioned on text, Canny, and depth map. Using text (Fig. <ref>), the vanilla SDXL without CFG is unable to generate high-quality images and produces noisy results. Canny and depth map conditioning on SDXL (Fig. <ref>, <ref>, and <ref>) is achieved through ControlNet <cit.>, trained on such maps. The results show that SEG enhances the quality and fidelity of the generated images while preserving the textual and structural information provided by the conditioning inputs. Notably, as σ increases, the generated images exhibit improved definition and quality without introducing significant artifacts or deviations from the original condition. The combination with higher CFG scales is shown in Figs. <ref>–<ref>.

In Table <ref>, we show the quantitative results for text-conditional generation in terms of σ. We observe a clear trade-off between image quality (represented by FID and CLIP score) and the deviation from the original sample (represented by LPIPS). We sample 30k images for each σ to compute the metrics.




 §.§ Comparison with previous methods

Since the results are visually favorable when we use σ = 10 and σ→∞, and they are the best in terms of CLIP score and FID, respectively, we adopt those configurations for comparison of unconditional guidance methods. The results are presented in Figs. <ref>, <ref>, <ref>, and Table <ref>. Notably, our method achieves better image quality in terms of FID, while remaining similar to the original output of vanilla SDXL as measured by LPIPS, implying a Pareto improvement.

r6.5cm



    < g r a p h i c s >



Ablation study on γ_seg and σ.








 §.§ Ablation study



In this section, we address two parameters, γ_seg and σ, and justify that fixing γ_seg is a reasonable choice. In Fig. <ref>, we present the results from our testing. The results reveal that increasing γ_seg does not generally lead to improved sample quality in terms of FID and CLIP score, due to various issues such as saturation. In contrast, increasing σ tends to improve sample quality and plausibility. This supports the claim that image quality should be controlled by σ, instead of the guidance scale parameter. We sample 30k images for each combination to calculate the metrics.







§ CONCLUSION, LIMITATIONS AND SOCIETAL IMPACTS




  
Conclusion

We introduce Smoothed Energy Guidance (SEG), a novel training- and condition-free guidance method for image generation with diffusion models. The key advantages of SEG lie in its flexibility and the theoretical foundation, allowing us to significantly enhance sample quality without side effects by adjusting the standard deviation of the Gaussian filter. We hope our method inspires further research on improving generative models, and extending the approach beyond image generation, for example, to video or natural language processing.



  
Limitations and societal impacts

The paper proposes guidance to enhance quality outcomes. Consequently, the attainable quality of our approach is contingent upon the baseline model employed. Furthermore, the application of SEG to temporal attention mechanisms in video or multi-view diffusion models is not addressed, remaining a promising avenue for future research. It is important to note that the improvements achieved through this method may potentially lead to unintended negative societal consequences by inadvertently amplifying existing stereotypes or harmful biases.


plain













§ FULL PROOFS




 §.§ Proof of Lemma <ref>



Let a_(i, j) denote the original attention weights and ã_(i, j) denote the blurred attention weights, as in the main paper. Assume that the original attention weights are properly padded to maintain consistent statistics. Then, the following shows that the mean of the blurred attention weights remains the same.

    𝔼_i,j[ã_(i,j)]    = 1/HW∑_i=1^H∑_j=1^Wã_(i,j) = 1/HW∑_i=1^H∑_j=1^W∑_m=-k^k∑_n=-k^k G(m, n) · a_(i+m,j+n)
       = ∑_m=-k^k∑_n=-k^k G(m, n) ·(1/HW∑_i=1^H∑_j=1^W a_(i+m, j+n)) 
       = ∑_m=-k^k∑_n=-k^k G(m, n) ·𝔼_i,j[a_(i,j)] = 𝔼_i,j[a_(i,j)] ·∑_m=-k^k∑_n=-k^k G(m, n) = 𝔼_i,j[a_(i,j)]


In addition, the variance of the blurred attention weights is smaller than or equal to the variance of the original attention weights.

    Var_i,j[ã_(i,j)]    = 1/HW∑_i=1^H∑_j=1^W (ã_(i,j) - 𝔼_i,j[ã_(i,j)])^2 
       = 1/HW∑_i=1^H∑_j=1^W(∑_m=-k^k∑_n=-k^k G(m, n) · (a_(i+m, j+n) - 𝔼_i,j[a_(i,j)]))^2 
       = ∑_m=-k^k∑_n=-k^k∑_r=-k^k∑_s=-k^k G(m, n) · G(r, s) ·Cov[a_(i+m, j+n), a_(i+r, j+s)]

Using the Cauchy-Schwarz inequality and the normalization property of the 2D Gaussian filter, we can show that the variance monotonically decreases when we apply Gaussian blur.

    Var_i,j[ã_(i,j)]    ≤∑_m=-k^k∑_n=-k^k∑_r=-k^k∑_s=-k^k G(m, n) · G(r, s) ·√(Var[a_(i+m, j+n)] ·Var[a_(i+r, j+s)])
       = (∑_m=-k^k∑_n=-k^k G(m, n) ·√(Var[a_(i+m, j+n)]))^2 
       ≤(∑_m=-k^k∑_n=-k^k G(m, n)) ·(∑_m=-k^k∑_n=-k^k G(m, n) ·Var[a_(i+m, j+n)]) 
       = ∑_m=-k^k∑_n=-k^k G(m, n) ·Var[a_(i+m, j+n)] 
       = Var_i,j[a_(i,j)]






 §.§ Proof of Lemma <ref>



Applying the second-order Taylor series approximation of e^x to our function f around the mean μ, we get:

    ∑_i=1^H∑_j=1^W e^a_(i,j)   ≈∑_i=1^H∑_j=1^W(e^μ + e^μ(a_(i,j) - μ) + 1/2e^μ(a_(i,j) - μ)^2) 
       = HW · e^μ + 1/2e^μ∑_i=1^H∑_j=1^W (a_(i,j) - μ)^2

In the last step, we used the fact that ∑_i=1^H∑_j=1^W (a_(i,j) - μ) = 0 because μ is the mean.

Similarly,

    ∑_i=1^H∑_j=1^W e^ã_(i,j)   ≈ HW · e^μ + 1/2e^μ∑_i=1^H∑_j=1^W (ã_(i,j) - μ)^2

Since Var[a] > Var[ã], we have:

    ∑_i=1^H∑_j=1^W (a_(i,j) - μ)^2 ≥∑_i=1^H∑_j=1^W (ã_(i,j) - μ)^2

Therefore, the second-order approximation of lse(𝐚) is larger than that of lse(𝐚̃).

Note that this fact also implies blurring with a Gaussian filter with a bigger variance causes more decrease in the variance of attention weights, because Gaussian filter with a larger variance can always be represented as a convolution of two filters with smaller variances, and the convolution operation is associative.

To find the maximum value subject to the constraint a_(1,1) + a_(1,2) + … + a_(H,W) = c for some constant c, we introduce Lagrange multipliers. Let g(a_(1,1), a_(1,2), …, a_(H,W)) = a_(1,1) + a_(1,2) + … + a_(H,W). The Lagrangian function is defined as:

    L(a_(1,1), a_(1,2), …, a_(H,W), λ) = e^a_(1,1) + e^a_(1,2) + … + e^a_(H,W) - λ(a_(1,1) + a_(1,2) + … + a_(H,W) - c)

Taking partial derivatives and setting them to zero yields:

    ∂ L/∂ a_(i,j) = e^a_(i,j) - λ = 0

Solving for a_(i,j), we obtain a_(i,j) = ln(λ) for all i = 1, 2, …, H and j = 1, 2, …, W
Summing these equations results in:

    λ = e^c/HW

Substituting λ back into a_(i,j) = ln(λ) gives a_(1,1) = a_(1,2) = … = a_(H,W) = c/HW.
Therefore, the minimum value of ∑_i=1^H∑_j=1^W e^a_(i,j) is achieved when a_(1,1) = a_(1,2) = … = a_(H,W).




 §.§ Proof of Theorem <ref>



Let 𝐚 = (a_1, …, a_n) denote the attention values before the softmax operation, and let 𝐚̃ = (ã_1, …, ã_n) denote the attention values after applying the 2D Gaussian blur. Let 𝐇 denote the Hessian of the original energy, i.e., the derivative of the negative softmax, and 𝐇̃ denote the Hessian of the underlying energy associated with the blurred weights.

The elements in the i-th row and j-th column of the Hessian matrices are given by:

    h_ij = (ξ(𝐚)_i - δ_ij)ξ(𝐚)_j,


    h̃_ij = (ξ(𝐚̃)_i - δ_ij)ξ(𝐚̃)_j b_ij,

respectively, where b_ij are the elements of the Toeplitz matrix corresponding to the Gaussian blur kernel, and δ_ij denotes the Kronecker delta.

Assuming ξ(𝐚̃)_i ξ(𝐚̃)_j ≈ 0 and ξ(𝐚)_i ξ(𝐚)_j ≈ 0 for all i and j, which is a reasonable assumption when the number of token is large, the non-diagonal elements of the Hessians approximate to 0 and the diagonal elements dominate. Therefore, the determinants of the Hessian matrices are approximated as the product of the dominant terms:

    |(𝐇)| ≈∏_i=1^n ξ(𝐚)_i,    |(𝐇̃)| ≈∏_i=1^n ξ(𝐚̃)_i b_ii


We have the following inequality:

    ∏_i=1^n ξ(𝐚̃)_i b_ii < ∏_i=1^n ξ(𝐚̃)_i    = e^∑_j=1^nã_j/(∑_j=1^n e^ã_j)^n
       ≤e^∑_j=1^na_j/(∑_j=1^n e^a_j)^n = ∏_i=1^n ξ(𝐚)_i,

where the first inequality follows from the property of the Gaussian blur kernel, 0 ≤ b_ii < 1, and the second inequality is derived from Lemmas <ref> and <ref>, which demonstrate the mean-preserving property and the decrease in the lse value when applying a blur. The monotonicity of the logarithm function implies that the denominator involving the blurred attention weights is smaller. Eventually, we obtain the following inequality:

    |(𝐇̃)| < |(𝐇)|.

This implies that the updated value is derived with attenuated curvature of the energy function underlying the blurred softmax operation compared to that of the original softmax operation.






§ DUAL DEFINITION



As we previously stated in Section <ref>, we have the dual definition regarding (<ref>), where we use swapped indexing. Importantly, the swapped indices can be interpreted as altering the definition of attention weights to 𝐀:=𝐊𝐐^⊤.

A similar conclusion can be drawn as in the main paper, except that query blurring becomes key blurring with this definition. To see this, Eq. <ref> changes slightly with this definition, using the symmetry of the Toeplitz matrix 𝐁:

    G ∗ (𝐊𝐐^⊤)    = 𝐁(𝐊𝐐^⊤) 
       = ((𝐊𝐐^⊤)^⊤𝐁^⊤)^⊤
       = (𝐐𝐊^⊤𝐁^⊤)^⊤
       = (𝐐(𝐁𝐊)^⊤)^⊤
       = (𝐐(G ∗𝐊)^⊤)^⊤
       = (G ∗𝐊)𝐐^⊤,

where ∗ denotes the 2D convolution operation. Empirically, this altered definition does not introduce a significant difference in the overall image quality, as shown in Fig. <ref>.




§ ADDITIONAL QUALITATIVE RESULTS


In this section, we present further qualitative results to demonstrate the effectiveness and versatility of our Smoothed Energy Guidance (SEG) method across various generation tasks and in comparison with other approaches.



  
Comparison with previous methods

Figs. <ref> and <ref> provide a qualitative comparison of SEG against vanilla SDXL <cit.>, Self-Attention Guidance (SAG) <cit.>, and Perturbed Attention Guidance (PAG) <cit.>. These comparisons highlight the superior performance of SEG in terms of image quality, coherence, and adherence to the given prompts. SEG consistently produces sharper details, more realistic textures, and better overall composition compared to the other methods.



  
Conditional generation with ControlNet

Figs. <ref> and <ref> showcase the application of SEG in conjunction with ControlNet <cit.> for conditional image generation. These results illustrate how SEG can enhance the quality and coherence of generated images while maintaining fidelity to the provided control signals. The images demonstrate improved detail, texture, and overall visual appeal compared to standard ControlNet outputs without prompts.



  
Unconditional and text-conditional generation

Fig. <ref> demonstrates the capability of SEG in unconditional image generation, showcasing its ability to produce high-quality, diverse images without text prompts. Fig. <ref> exhibits text-conditional generation results using SEG, illustrating its effectiveness in translating textual descriptions into visually appealing and accurate images.



  
Interaction with classifier-free guidance

Figs. <ref>–<ref> present a series of experiments exploring the combination of SEG with CFG. In these experiments, the SEG guidance scale (γ_seg) is fixed at 3.0, while the CFG scale is varied. The results demonstrate that SEG consistently improves image quality across different CFG scales without causing saturation or significant changes in the general structure of the images.














































 Paper Title: GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Models.










2676 















A]Ali Abdollahi
[Equal contribution.]

A]Mahdi 	Ghaznavi

A,D]Mohammad Reza Karimi Nejad
 
A]Arash Mari Oriyad
A]Reza Abbasi
A]Ali Salesi
B,C]Melika Behjati
A]Mohammad Hossein Rohban


A]Mahdieh Soleymani Baghshah




[A]Sharif University of Technology
[B]ecole polytechnique fédérale de lausanne (EPFL)
[C]IDIAP research institute
[D]Shamsipour Technical and Vocational College



§ ABSTRACT 
Vision-language models (VLMs) are intensively used in many downstream tasks, including those requiring assessments of individuals appearing in the images. While VLMs perform well in simple single-person scenarios, in real-world applications, we often face complex situations in which there are persons of different genders doing different activities. We show that in such cases, VLMs are biased towards identifying the individual with the expected gender (according to ingrained gender stereotypes in the model or other forms of sample selection bias) as the performer of the activity. We refer to this bias in associating an activity with the gender of its actual performer in an image or text as the Gender-Activity Binding (GAB) bias and analyze how this bias is internalized in VLMs. To assess this bias, we have introduced the GAB dataset with approximately 5500 AI-generated images that represent a variety of activities, addressing the scarcity of real-world images for some scenarios. To have extensive quality control, the generated images are evaluated for their diversity, quality, and realism. We have tested 12 renowned pre-trained VLMs on this dataset in the context of text-to-image and image-to-text retrieval to measure the effect of this bias on their predictions. Additionally, we have carried out supplementary experiments to quantify the bias in VLMs’ text encoders and to evaluate VLMs’ capability to recognize activities. Our experiments indicate that VLMs experience an average performance decline of about 13.2% when confronted with gender-activity binding bias.













§ INTRODUCTION



Nowadays large multi-modal models have shown tremendous potential in various tasks, from information retrieval systems to image captioning, visual question answering, image generation, and visual reasoning.
As a branch of multi-modal models, Vision-Language Models (VLMs) <cit.>
that provide a shared cross-modal embedding space between text and image modalities, 
have been intensively investigated by researchers recently and are used in many real-world applications <cit.>. 








Despite their wide range of success in downstream tasks, VLMs are subject to many critical biases, which can consequently affect their performance, especially in sensitive applications. For example, <cit.> provides a comprehensive analysis of biases and their implications in the CLIP Models, revealing how this model is exposed to gender biases and how challenging it is to trust it in real-world applications.
As <cit.> and <cit.> state, the bias is introduced in visual-linguistic pre-training due to unfairness in the training data, and additionally, at inference time, the visual and linguistic contexts that is used for few-shot applications can also promote bias. 





One important type of bias, which is highly discussed in the literature, is the gender bias. Gender bias demonstrates the undesirable association of a factor with a gender according to the model. For instance, due to the high occurrence of data in which men are repairing devices, this activity is usually associated with men by deep models, which leads to failure when faced with women repairing devices.
 <cit.> suggests that different VLMs do not perform equally well on determining gender for an occupation, and do not assign equal retrieval likelihood to images of male and female professionals. Also, <cit.> and <cit.> show that CLIP <cit.> shows biased behavior in retrieval based on gender-neutral queries. Please note that, in line with common practice in the literature, our focus is on two genders, masculine and feminine. This approach streamlines the process of generating the dataset and analyzing the experiments.



In this research, we delve into the issue of gender-activity binding in retrieval tasks. We aim to retrieve the corresponding caption or image from two given captions or images, each depicting an activity performed by a different gender, based on a provided image or caption that shows one of the genders performing the activity. We refer to this bias in associating gender and activity as the Gender-Activity Binding (GAB) bias. This bias arises due to ingrained gender stereotypes in the model or other types of sample selection biases <cit.>. Our research indicates that VLMs do not display a substantial bias in retrieval tasks when both text and image modalities depict only one gender. This is because the identification of the performer’s gender can directly lead to the correct output, eliminating the need to bind the performed activity with the performer’s gender. Though, text encoders have shown a considerable bias towards the expected gender. However, The bias becomes more pronounced when the scene is more complex, i.e., when two individuals of different genders are present in the image. We observe a drop in retrieval accuracy in scenarios where the activity performer is the unexpected gender. 
For example, in the context of the “repairing” activity, the retrieval accuracy of VLMs in identifying the performer decreases when a woman is repairing a device and a man is also present in the scene. This is in comparison with scenarios where the performer is a man, or there is no man present in the scene to associate the activity with him.

The significance of this bias escalates when we consider its underlying implications. Not only does it perpetuate societal biases and fairness concerns related to gender stereotypes, but it can also lead to serious complications if incorporated into judgement and decision-making systems. The potential for such bias to inadvertently influence outcomes underscores the importance of addressing it.

















































We have created a dataset known as the Gender-Activity Binding (GAB) dataset as shown in Figure <ref>(a). This dataset comprises images depicting various activities performed by men or women in different contexts (with or without the presence of an individual of the opposite gender), each accompanied by a descriptive caption.


Given that our scenarios are uncommon due to biases and stereotypes, suitable real-world images are scarce for evaluation purposes. To address this issue, all images are generated using DALL-E 3[https://openai.com/dall-e3]. We employ extensive prompt enhancement techniques to ensure the diversity and quality of the generated images. We also evaluated the generated images on diversity, quality, and realism. We performed multiple quantitative assessments consisting of Fréchet Inception Distance (FID) for both quality and diversity assessment, Structural Similarity Index (SSIM) 
and Learned Perceptual Image Patch Similarity (LPIPS) for diversity assessment, and human feedback for a final qualitative assessment to ensure the final results accurately represent the intended activities and their related contexts.

The activities included in GAB dataset are sourced from various places to encapsulate societal gender stereotypes and other activities that are statistically biased towards a specific gender, as observed in the LAION-400M dataset <cit.>.






We benchmark the performance of 12 renowned multi-modal foundation models and assess their performance in terms of text-to-image retrieval, image-to-text retrieval, and recognizing activities on GAB dataset as shown in Figure <ref>(b). 
Specifically, in the image-to-text retrieval task, we observed that while an unexpected gender is doing a stereotypically biased activity, the average performance of VLMs declined by approximately 33.2% due to the presence of the expected gender in the scene. Moreover, most VLMs undergo an average accuracy reduction of approximately 13.2% when encountering gender-activity binding bias. On the other hand, in the text-to-image retrieval task, most VLMs achieve an accuracy of approximately 50%, indicating that their performance is nearly random, and they fail to recognize the performer of the activity based on the text.


















As far as we are aware, this research is the first to delve into the origins of the association between genders and activities in vision-language models by conducting a thorough analysis both image-to-text and text-to-image retrieval accuracy of VLMs. It encompasses not only straightforward scenarios involving single individuals, where VLMs demonstrate high accuracy for both genders <cit.>, but also more complex situations with individuals of different genders <cit.>. This comprehensive approach allows us to isolate the impact of the complexity of the scenario on performance decline <cit.> and more accurately measure the Gender-Activity Binding bias. Additionally, the study carries out supplementary experiments to investigate the bias in VLMs’ text encoders and their proficiency in recognizing activities.



To summarize, our contributions are as follows:

    
    
  * We have created a novel dataset, known as the gender-activity binding dataset. This dataset includes AI-generated images of various activities that are typically associated with a specific gender, and are excellent in terms of diversity, quality, and realness (Section <ref>).

    
    
    
    
    
    
    
    
    
    
  * We conduct an intensive performance benchmark of well-known vision-language models in retrieval tasks to assess their robustness against the Gender-Activity Binding bias. Our findings reveal that when two individuals of different genders are present, VLMs exhibit a bias towards binding the activity with the gender that is expected to perform it, resulting in a 13.2% drop in retrieval accuracy. However, in scenarios involving a single individual, these models demonstrate high image-to-text retrieval accuracy. We also demonstrate that VLMs lack the ability to bind gender and activity in text-to-image retrieval tasks (Section <ref>). 

    
  * We carry out additional experiments to investigate the binding bias in VLMs’ capability to comprehend activities and the bias present in their text encoders (Section <ref> and the Appendix). We delve into the insights that can be derived about how VLMs internalize this bias and its impact on the shared embedding space of VLMs (Section <ref>).

    
    







§ RELATED WORK







 §.§ Vision-Language Models

VLMs apply contrastive training to bridge the image and text embedding spaces. Using this shared embedding space, VLMs have exhibited a remarkable performance on a notable number of multi-modal downstream tasks such as image captioning, text-to-image and image-to-text retrieval, visual question answering, and text-to-image generation. As <cit.> and <cit.> mentioned, VLMs can be categorized into three main groups based on their architectures: I) Fusion VL-encoders such as FLAVA <cit.> and LXMERT <cit.>, II) Dual-stream encoders like CLIP <cit.>, GLIP <cit.>, and FILIP <cit.>, and III) VL Encoder-decoder including BLIP <cit.>.



 §.§ Gender-activity Binding Bias

Previous works discussed that VLMs have many categories of biases <cit.>, especially gender-related ones <cit.>. 

For example, <cit.> studied bias in VLMs as a task of measuring associations between entities and gender in visual-linguistic models using template-based masked language modeling. Moreover, <cit.> measured gender bias in VLMs by comparing the MASK token prediction probabilities of factual and counterfactual samples and concluded that VLMs inherit human stereotypes from the training data.

On the other hand, there exist a number of studies including a restricted assessment of how a VLM understands a particular activity <cit.>. For example, <cit.> has introduced a small dataset called Winoground containing objects, attributes, and activities to evaluate the ability of VLMs in compositional reasoning. Moreover, <cit.> designed VL-CheckList framework to study the capability of VLMs in understanding objects, attributes, and relations containing activities. However, to the best of our knowledge, there is no systematic study on the ability of VLMs in associating an activity with a gender.













§ GENDER-ACTIVITY BINDING BIAS






Despite the recent increase in large-scale data collection, it’s important to note that these datasets are not necessarily devoid of biases. These biases can manifest in large datasets derived from real-world distributions, primarily due to sample selection bias.

The issue causes VLMs to have varying levels of accuracy in recognizing the performer of an activity depending on the gender of the individual involved <cit.>. 
We refer to this bias in associating an activity with the gender of its actual performer in an image or text as Gender-Activity Binding Bias. 


Consider an image-to-text retrieval task, where we have two pieces of text C_1 and C_2, each describing an activity performed by a different gender. We also have an image I_1 depicting a person (either a man or a woman) performing an action. The task is to match the image with the correct piece of text (Figure <ref>-(a) left). Also, consider the text-to-image retrieval task, where we have two images I_1 and I_2, each showing an activity being performed by a specific gender, and we want to retrieve the correct image based on a given piece of text C_1 (Figure <ref>-(a) right). The Gender-Activity Binding bias manifests as a decline in the retrieval accuracy of VLMs when they incorrectly identify the gender of an individual performing activities typically associated with a different gender.


Let's denote the probability of a VLM correctly identifying the performer of an activity in a data i (image or text) as p, the activity as a, and the gender as g. Ideally, the probability p(g|a, i) should be equal for all g. 
However, because of the co-occurrence of gender g of the performer and the activity in the dataset, which is referred to as sample selection bias <cit.>, the gender of the performer becomes correlated with the activity. Thus, we often find that p(g_1|a, i)> p(g_2|a, i), where g_1 and g_2 represent different genders and g_1 shows the expected gender for the performer of a. It is also 
In simple terms, VLMs tend to incorrectly select the text or image because they are biased towards associating the activity with a specific gender. This is not always accurate, leading to errors in scenarios where the activity is performed by the less expected gender. 












The accuracy of models in image and text retrieval tasks is assessed in two scenarios: one where only a person of an unexpected gender (in terms of the bias) is depicted or described in the text or image, and another where both genders are present in the data, but the activity is performed by only one of them. Consequently, the impact of the bias can be observed as a decrease in accuracy in two situations: when the expected gender is absent or present, and when the activity is performed by the unexpected or expected gender. These cases together provide better insights into how these models handle gender-activity associations under different conditions. The first scenario allows us to understand how well the models can recognize and correctly associate an activity with an unexpected gender when the other gender is not present. This helps us gauge the models’ ability to break away from societal stereotypes and biases. The second scenario, on the other hand, tests the models’ ability to correctly identify the performer of an activity when both genders are present, which is a more complex and realistic situation. This indicates these biases affect the models’ performance in real-world scenarios. Together, these cases provide a comprehensive understanding of the models’ strengths and weaknesses in handling gender-activity binding, paving the way for targeted improvements to mitigate the gender-activity binding bias.






































































§ DATASET




This section details the creation process of the Gender-Activity Binding (GAB) Dataset, which serves to evaluate various VLMs against the gender-activity binding bias we have identified and VLMs ability to bind activities and performers. This dataset comprises AI-generated images that illustrate a variety of activities with potential subjects of both genders.





The GAB dataset divides images into four distinct groups: In two groups, both genders are present in the scene. In one of these groups, the expected gender is performing the activity while the other gender is just present in the image, and in the other group, the unexpected gender is performing the activity while the other gender is also in the scene.
In the next two groups, only one gender is present in the scene. In one of them, the expected gender is performing the action, and in the other one, the scenario is reversed, with the unexpected gender acting. 

For each activity, with respect to these grouping, we can consider four groups based on the performer of the activity (expected (E) or unexpected (U)) and the number of persons (with different genders) in the scene (1 or 2): E1, E2, U1, U2.






For each image in the dataset, we have templates that are replaced with activity or gender names, such as "man" and "woman," in accordance with the designed experiments. Details of the experiments and these replacements can be found in Section <ref>. The template for images containing two genders is: "a man/woman is doing activity and a woman/man is in the scene" and for images containing one gender, the template is: "a man/woman is doing activity". This specific template can also be used for images with two genders.





Each of the mentioned groups contains at least 26 images for each activity, resulting in a total of 5500 images.






This dataset structure facilitates the evaluation of gender-activity binding bias and the ability of VLMs to bind and recognize activities by definition of different retrieval tasks, as you can see in Section <ref>.

Pipeline of the dataset creation is shown in Figure <ref>(a)




 §.§ Selection of Activities




We proposed three methods to identify the biased activities, all utilizing Masked Language Modeling (MLM) as part of their approach to detecting biased activities. We utilized the 'RoBERTa-large' <cit.> model for its superior performance in masked token prediction, which was attributed to its objective function.

The sentences are formatted as "a mask is doing activity".[mask is the special mask token of the RoBERTa model] We then calculate the probabilities associated with "man" and "woman"
(as well as related masculine and feminine identifiers like "boy" and "girl"), from which we calculate the log ratios, as specified in Section <ref>.



  §.§.§ Stereotyped Activities from GPT Combined with MLM


Our first method involved using GPT-4 to request activities stereotypically associated with men or women in society. We provided over 100 activities from GPT-4 and formed sentences by replacing the activity token in the template of 'a mask is doing activity'. We then inputted these sentences into the RoBERTa model and calculated the following equation for the sentences.

    bias(sentence) = log(P(mask = man|sentence)/P(mask = woman|sentence))

We applied this methodology separately for activities stereotypically associated with each gender. For men, this resulted in a normal distribution[all the tests including log ratios formed normal distributions that we tested using Shapiro–Wilk test for normality] with a positive mean (μ > 0), from which we chose the sentences at the rightmost end. Similarly, for women, it also resulted in a normal distribution but with a negative mean (μ < 0), from which we then chose the sentences at the leftmost end.




  §.§.§ Everyday Activities from GPT Combined with MLM


In our second approach, we used GPT-4 to compile a list exceeding 1,000 sentences related to hobbies, jobs, and everyday activities. We then masked the subjects of the sentences just like in the first method and asked the RoBERTa model to fill the mask. We stored the log ratios of predicted probabilities. The log ratios formed a normal distribution, from which we selected sentences at each end.




  §.§.§ Activities from LAION-400M Dataset Combined with MLM


In the third method, we used the LAION-400M <cit.> dataset, which consists of 400 million image-caption pairs. We filtered out the NSFW pairs and sampled 20 million pairs randomly. We then extracted the verbs, subjects, and objects of all caption sentences using spaCy <cit.>, a natural language processing toolkit. We counted the number of masculine and feminine subjects for each verb-object pair and calculated their log ratios. This also formed a normal distribution.

We then used the Z-test to determine whether each verb-object pair had a significantly different ratio compared to the mean of all verb-object pairs. This test is appropriate due to the large dataset size of LAION-400M. By checking if differences are statistically significant, we identify biased activities. This method enhances the reliability of our activities by ensuring they are not a result of random variance or outliers. Then, we created sentences with the biased verb-object pairs as mentioned in Section <ref>, masked the subject, and submitted them to the RoBERTa model to fill the mask, conducting the same experiment as in the previous methods to select the biased activities.




















 §.§ Generation


The generation process consisted of two phases. The first involved generating a good prompt that helped the image generation system meet the criteria given below, derived from the base sentence. The second involved generating the image from the prompt. The generated images should accurately represent the original activity, look natural and high quality, and be diverse.


  §.§.§ Prompt Enhancement


We used a Large Language Model (LLM) to generate diverse prompts while also keeping the original activity intact (Details about the utilized LLM are provided in the Appendix.). We then created a list of predefined settings like the environment of the house, the skin color of the person, etc. Although we could have sampled N prompts from each setting to generate diverse images, we chose to create a list of different combinations of these settings and encode them using MiniLM <cit.> sentence transformer <cit.>. We then clustered them into K[K = 6 which resulted in sufficient diversity in our experiments] clusters and sampled N/K samples from each cluster to enforce more diversity semantically. We then generated a diverse prompt by instructing Llama2 to generate diverse prompts that maintain the original activity while describing the environment and the settings using the selected samples. Using these prompts, the generated images were significantly improved in quality and detail.



  §.§.§ Image Generation

We initially experimented with several diffusion models, as mentioned in the Appendix, to generate images. The generated images were realistic and diverse. However, they lacked representation of the intended activity, especially in the context of compositional generation. We then explored DALL-E3 <cit.> using OpenAI API. This approach resulted in significant improvement to the quality of compositional generation while also keeping the reality and the clarity of the image. We chose DALL-E3 as our final image generation solution.




 §.§ Image Filtering Methodology

Our image filtering methodology employs a combination of automated metrics and human evaluation to ensure the generated images are of high quality, relevant, and diverse. The process comprises three steps: quality assessment, diversity metrics, and human feedback, detailed as follows:



  §.§.§ Quality Assessment

The quality of generated images was assessed using the Fréchet Inception Distance (FID) <cit.> alongside criteria emphasizing the representation of human figures, as identified by the YOLOv8 <cit.> and MTCNN <cit.> models. The FID score aims to measure the similarity between the generated images and a set of reference images from the COCO <cit.> dataset, calculated as:


    FID(x, y) = ||μ_x - μ_y||^2 + Tr(Σ_x + Σ_y - 2(Σ_xΣ_y)^1/2),

where μ_x, μ_y are the feature-wise mean of the real and generated images, and Σ_x, Σ_y are the covariance matrices of the real and generated images, respectively. To achieve high-quality images, we initially filtered out images with high Fréchet Inception Distance (FID) scores. After this filtering process, we successfully reduced the mean FID score to 11.9. Furthermore, for an image to pass our quality assessment, individuals must constitute at least 15% of the detected objects and must have a detectable face, as confirmed by the MTCNN model. 
It’s worth mentioning that the FID scores for both expected (E1 and E2) and unexpected (U1 and U2) categories of images are 12.1 and 11.7 respectively, signifying an acceptable level of quality across all scenarios.



  §.§.§ Diversity Metrics



To ensure image diversity, we employed the Structural Similarity Index (SSIM) <cit.> and the Learned Perceptual Image Patch Similarity (LPIPS) <cit.> metrics. The SSIM index measures the similarity between two images; in our context, lower scores indicate greater diversity. It is defined as:


    SSIM(x, y) = (2μ_xμ_y + c_1)(2σ_xy + c_2)/(μ_x^2 + μ_y^2 + c_1)(σ_x^2 + σ_y^2 + c_2),

where μ_x, μ_y are the average intensities, σ_x^2, σ_y^2 are the variances, and σ_xy is the covariance. c_1 and c_2 are constants added to stabilize the division and avoid numerical instability with small denominators. Our target mean SSIM threshold across all image pairs is 0.046. It’s also worth mentioning that the SSIM for the expected (E1 and E2) and unexpected (U1 and U2) categories of images are 0.046 and 0.045, respectively.
This indicates a satisfactory level of diversity in both image categories.

LPIPS measures the perceptual similarity between two images, using deep network features to closely mimic human visual perception. To indicate greater diversity, we aim for higher LPIPS values, setting the mean target for all pairs at 0.66. Also, the LPIPS scores for the expected (E1 and E2) and unexpected (U1 and U2) categories of images are 0.66 and 0.65, respectively.





  §.§.§ Human Feedback

After quantitative evaluations, we performed a qualitative analysis with human feedback. This involved scrutinizing images for alignment with the desired activities and contexts. Images that did not align with the activities or did not depict the intended subjects were removed. Additionally, images with unrealistic or nonsensical elements, and those appearing abnormal to an external observer were excluded.

Following this step, a list of images that have passed both quantitative and qualitative evaluations is obtained. As a result, the images are diverse, high-quality, and realistic while also meeting our criteria for the generation phase.






§ EXPERIMENTS





























 We designed several experiments on our proposed GAB dataset to evaluate Vision-Language Models in text-to-image and image-to-text retrieval tasks as shown in Figure <ref>(b). Based on these experiments, we assessed the defined gender-activity binding bias and the ability of VLMs to recognize gender-biased activities. Finally, we evaluated the bias in the text encoders and image encoders of different VLMs separately.






 §.§ Setup





  §.§.§ Task Definitions

The experiments we have designed to assess the performance of VLMs on the aforementioned aspects are grounded in their zero-shot performance on both text-to-image and image-to-text retrieval tasks.


 






More formally, consider the space of images as ℐ and the space of captions as 𝒞.

 
Also consider a VLM with an image encoder E_𝒞:𝒞→ℝ^D and a text encoder E_ℐ:ℐ→ℝ^D, that map images and texts respectively to a shared embedding space, which is shown in the middle box in Figure <ref> (b). We can define matching score function s:ℐ×𝒞→ℝ^+, which measures the similarity between a caption and an image, as the cosine similarity between the embedding of an image I and the embedding of a caption C:


    s(I, C)=E_ℐ(I).E_𝒞(C)/‖ E_ℐ(I) ‖‖ E_𝒞(C) ‖.


Text-to-image retrieval involves finding the image that best matches a given text from a collection of images. Conversely, image-to-text retrieval is about finding the text that best describes a given image. We compute the similarity using the previously defined similarity measure in the upcoming experiments. For the text-to-image retrieval task, we have a caption C and make comparisons between two images: I, which shows an image corresponding to the caption C, and I^R which corresponds to C^R (obtained by reversing the gender of subjects in C). Similarly, for the image-to-text retrieval task, we have an image I and make comparisons between two captions C and C^R. 









  §.§.§ Selected Vision-Language Models



In our experiments, we benchmark 12 VLMs and present the results of various tests.



For comparing the effects of patch size and backbone size, we report the results for 4 CLIP models released by OpenAI <cit.>, each with different backbones and patch sizes.


Another model selected for evaluation is NegCLIP <cit.>, a version of the CLIP-ViT-B-32 model by OpenAI <cit.> which is fine-tuned on a modified subset of the Visual Genome Dataset <cit.> annotations. NegCLIP is fine-tuned to purportedly enhance the base model's performance on tasks requiring relational and activity understanding.


We also report results for several other notable VLMs that have been published in recent years. These include Eva01 <cit.> and Eva02 <cit.>, FLAVA <cit.>, ALIGN <cit.>, COCA <cit.> and AltCLIP <cit.>.










 §.§ Results

This section presents the outcomes of experiments designed to evaluate model performance across various tasks, including Image-To-Text and Text-to-Image retrieval, and activity recognition(Detailed results and descriptions of activity recognition experiments are provided in the Appendix.). Additionally, this section specifically addresses and reports on the bias observed in the text encoder.



  §.§.§ Image-to-Text Retrieval



We assess the performance of the models discussed in Section <ref> on the GAB dataset and calculate their accuracy in retrieving the correct caption as outlined in Section <ref>. We employ a caption template of the form: “a man/woman is doing activity and a woman/man is in the scene”. The results for activities that are stereotypically biased are presented in Figure <ref>, while the results for the other two groups of activities are provided in the Appendix. 













In this experiment, we assessed image-to-text retrieval across three distinct scenarios. The first scenario involves images where an individual of an unexpected gender is performing a typically biased activity with no other individuals present in the scene. The second scenario comprises images where both genders are present, and the activity is performed by the gender typically associated with it. The third scenario mirrors the second, but in this case, the activity is performed by the gender not typically associated with it.


Performance Drop Due to Presence of Expected Gender:
As depicted in Figure <ref>, there is a noticeable drop in the accuracy of the models when the scene includes two genders compared to scenarios where only the unexpected gender is present and performing the activity associated with bias. The chart reveals that most VLMs experience an average performance decline of approximately 33.2%. However, a few models demonstrate only a slight decrease in performance.


Performance Drop Due to Gender-Activity Binding Bias: 
In the scenarios where there are two individuals present at the scene, the majority of models display a substantial decline in retrieval accuracy when the action is performed by an individual of the unexpected gender. As indicated in Figure <ref>, the VLMs undergo an average accuracy reduction of approximately 13.2% when encountering gender activity binding bias. 

Note that, as illustrated in Figure <ref>, the models demonstrate satisfactory performance in the Image-to-Text retrieval task on both image scenarios of U1 and E1, indicating that the gender bias adversely affects the performance of the models only when there is more than one individual in the image. Additionally, the accuracy of the models remains consistent, regardless of whether the expected or unexpected gender is performing the activity. The models' average performance in both scenarios is approximately 80%.



  §.§.§ Text Encoder Bias

Given an activity a, consider e,u∈{man, woman} where e is the expected gender of the individual performing a and u is the unexpected one. To separately assess the bias of the text encoder, we determine the frequency at which the embedding of a gender-neutral sentence "a person is doing activity" is closer to the embedding of "a e is doing activity" than the embedding of "a u is doing activity". The outcomes are presented in Table <ref>. It can be observed that in all categories of the gathered activities, the majority of activities exhibit a bias towards the expected gender.














  §.§.§ Text-to-Image Retrieval




Figure <ref> illustrates the accuracy of VLMs in the task of text-to-image retrieval. For each activity, we form random pairs of images from E2 and U2[Pairing is in the form of one-to-one correspondence.]. Each model is then evaluated for its ability to retrieve the image that best matches captions formatted as "a man/woman is doing activity and a woman/man is in the scene". As can be seen in Figure <ref>, VLMs achieve an accuracy of approximately 50%, indicating that their performance is nearly random in this task.
If we replicate this experiment, altering only the captions to a gender-neutral phrase "a person is doing activity", and designate the image where the expected gender is performing the activity as the true label, the VLMs achieve an accuracy of approximately 50% again, as illustrated in Figure 4 in the Appendix.

In essence, the benchmarked models do not seem to comprehend the properties of the given images that would aid in retrieving the correct one given embedded information from the caption, i.e., they fail to recognize the performer of the activity based on the text.









































 











































§ DISCUSSION



 §.§ Image and Text Embedding in VLMs

To gain a deeper understanding of how texts and images are encoded into the shared embedding space, consider these points:

    
  * Given that the image retrieval accuracy is nearly 50% for both expected and unexpected caption groups (as shown in Figure <ref>), we can infer that the cosine similarity (as per Eq. <ref>) between both sets of captions and images (for expected and unexpected groups) is almost identical.
    
  * As the average text retrieval accuracy exceeds 60% for E2 and falls below 50% for U2, it suggests that text embeddings are somewhat closer to the image embeddings in E2.

A plausible interpretation of these observations is that VLMs encode images more closely to the encoding of the more expected text. However, the image encoders of VLMs embed images from both groups closely together, resulting in nearly equal cosine similarity with any caption embedding.



 §.§ Bias Mitigation

Multiple works have previously discussed approaches for mitigating bias from VLMs, such as orthogonal projection <cit.>, making unbiased datasets <cit.>, representation correction <cit.>, and prompt tuning <cit.>. Several of these approaches could be applied also for robustness to gender-activity binding bias. For more detail on these works, refer to the Appendix.




§ FUTURE WORKS

Study Other Social Biases.
In this work, we focused solely on gender bias; however, the experimental approaches described can be extended to other social biases, such as race and age. In future research, we plan to explore how VLMs perform across these dimensions using the experiments outlined in this study.

Study the Source of Bias in the Training Data.
Due to limited accessibility to the training sets, especially for models with unpublished datasets, this study could not investigate the source of bias in the training data directly. We plan to address this limitation in future work by examining the sources of bias in publicly available training datasets for VLMs and analyzing the origins of these biases within those datasets.






§ CONCLUSIONS





In this study, we have explored a significant yet overlooked bias in VLMs known as the gender-activity binding bias. We used a unique dataset named GAB, which includes about 5500 images. We found that the capacity of VLMs to link an activity with the gender of its performer notably decreases when another person of a different gender is in the scene. We discovered that while the bias is evident in image-to-text retrieval tasks and in the text encoder alone, the models perform randomly in text-to-image retrieval tasks. This suggests that the gender-activity binding bias is primarily absorbed by the text encoder rather than the image encoder in VLMs. Furthermore, we noted that while VLMs have difficulty with gender-activity binding, they do have some ability to recognize activities. We believe that the ability of VLMs to comprehend activities and perform compositional reasoning in complex scenes are other crucial factors contributing to the gender-activity binding bias that could be investigated in future research.
















































§ DATASET






 §.§ Detailed Report

The method of selecting activities is detailed above, ultimately resulting in three categories of biased activities. comprehensive report of each category provided in Table <ref> , Table <ref> , Table <ref>.











































































































 §.§ Experimented Models For Image Generation


Initially, we explored several diffusion<cit.> models including the original Stable Diffusion<cit.>, Stable Diffusion XL<cit.>, and its checkpoints like (Juggernaut XL[https://civitai.com/models/133005/juggernaut-xl] and DreamShaper XL[https://civitai.com/models/112902/dreamshaper-xl]). We tried Fooocus[https://github.com/lllyasviel/Fooocus], an image generation tool that includes its prompt rewriting system using GPT-2<cit.>. The generated images were realistic and diverse; however, they lacked representation of the intended activity, especially in the context of compositional generation. Upon using DALL-E 3 with OpenAI API, we observed a significant improvement in the quality of compositional generation while also keeping the reality and the clarity of the image. We chose DALL-E 3 as our final image generation solution.




 §.§ Used LLM For Prompt Enhancement


We evaluated various large language models (LLMs) for this task. Due to our hardware constraints, we ultimately selected the Llama2 model, which has 70 billion parameters, to generate prompts <cit.>.




§ BIAS MITIGATION

Multiple works have previously discussed approaches for mitigating bias from VLMs that could be applied also for robustness to gender-activity binding bias. 

In <cit.>, an orthogonal projection is applied to the end embeddings, making them orthogonal to the embeddings of spurious prompts, effectively mitigating bias introduced by these prompts. This alteration might also strip away useful nuances from the embeddings, potentially degrading the model’s overall performance on tasks where those nuances are beneficial and also this might not hold true in complex real-world scenarios where biases are not strictly linear.

In <cit.>, a method is introduced that utilizes additive residual image representations to correct biases in the original representations. However, training these residuals demands extensive and diverse data, making the process resource-intensive. Moreover, incorporating these residuals may alter other inherent characteristics of the original representations, potentially affecting the overall performance of the model.

In <cit.>, learnable prompts integrated into the text input are used to mitigate bias, optimized through the training of an adversarial classifier that evaluates similarity scores between outputs from two encoders. This method employs both adversarial and contrastive losses to preserve the quality of the joint representation while aiming to reduce bias. However, the effectiveness of this adversarial training depends on the availability of a large and diverse dataset; otherwise, there is a risk of overfitting to adversarial objectives. Additionally, since VLMs are trained with various objective functions, making adjustments to mitigate bias could potentially lead to trade-offs in other performance metrics.

Both <cit.> and <cit.> address the creation of fair datasets to mitigate bias in VLMs, though each method has its limitations. <cit.> uses AI to generate datasets for fine-tuning VLMs, which is computationally intensive and can introduce new biases due to the limitations of AI generation tools. On the other hand, <cit.> employs counterfactual data—images and texts modified to alter specific attributes. While this method targets specific biases directly, it is limited to those attributes it explicitly modifies and may not adequately address all potential biases.

In <cit.>, the authors address bias by calculating the mutual information between each dimension of the encoder outputs and the gender attribute, clipping dimensions with the highest mutual information to reduce their influence. While this approach effectively reduces gender bias in embeddings, it faces limitations similar to other inference-time interventions. Notably, it may not adapt well to a wide range of biases and could potentially degrade the accuracy of models in other downstream tasks due to the removal of critical information during the clipping process.







§ RESULTS






 §.§ Image-to-Text Retrieval


The performance of text retrieval for texts featuring two genders (one of them performing the activity) is demonstrated in Table <ref> and Table <ref>. This is akin to the experiment discussed in Section 5.2.1,
In this experiment, we assessed image-to-text retrieval across three distinct scenarios. The first scenario involves images where an individual of an unexpected gender is depicted performing a typically biased activity with no other individuals present in the scene. The second scenario comprises images where both genders are present, and the activity is performed by the gender typically associated with it. The third scenario mirrors the second, but in this case, the activity is performed by the gender not typically associated with it. In Table <ref> the experiment is reported for gender-biased activities from LAION-400M and in Table <ref> the experiment is reported for gender-biased everyday activities.








 §.§ Text-to-Image Retrieval


The performance of image retrieval for images featuring a single individual (the one performing the activity) is demonstrated in Figure <ref>. This is akin to the experiment discussed in Section 5.4 of the main text, where images in E1/U1 are categorized and the models are evaluated based on their ability to accurately retrieve the correct image using captions that follow the template “a man/woman is performing an activity”. The models exhibit an average accuracy of approximately 90%, indicating that they can effectively identify the image that correctly represents the gender of the individual performing the activity, provided there are no individuals of a different gender present in the scene.
Also, you can see the results of experiment mentioned in Section 5.4 of the main text in Figure <ref>.






 




 §.§ Activity Recognition


A crucial inquiry concerning the bias in binding gender with certain activities is whether this bias originates from a misunderstanding of the activity itself, and if this misunderstanding leads to a failure in binding the gender and the activity. To gauge the model’s understanding of activities, we collect several (at least 9, 17.25 on average) other different activities that are performed in the same environment as the reference activity for each activity in the GAB dataset using GPT-4 <cit.>. For each image in the GAB dataset, we create a caption from the reference and associate collected activities and arrange them based on their matching score to the image. We show the mean reciprocal rank (MRR), recall@1, and recall@3 for the true caption (reference activity) in Table <ref> for stereotypical activities. (results for other categories of activities is depicted in Table <ref> and Table <ref>).

The table indicates that the model can, to some extent, retrieve the true activity from a list of activities performed in a similar environment, demonstrating that VLMs can recognize the performed activity in image-to-text retrieval.




























 Paper Title: Robust Box Prompt based SAM for Medical Image Segmentation.





 

Y. Huang et al.


1National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Medical School, Shenzhen University, China

nidong@szu.edu.cn 

2Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, China

3Shenzhen RayShape Medical Technology Co., Ltd, China

4Centre for Computational Imaging and Simulation Technologies in Biomedicine (CISTIB), University of Leeds, UK

5First Affiliated Hospital, Southern University of Science and Technology, Shenzhen People’s Hospital, China

Robust Box Prompt based SAM for 
 Medical Image Segmentation
    Yuhao Huang1,2, Xin Yang1,2 Han Zhou1,2 Yan Cao3 
Haoran Dou4 Fajin Dong5 Dong Ni1,2()
    July 2024
==========================================================================================



§ ABSTRACT 
The Segment Anything Model (SAM) can achieve satisfactory segmentation performance under high-quality box prompts.
However, SAM's robustness is compromised by the decline in box quality, limiting its practicality in clinical reality.
In this study, we propose a novel Robust Box prompt based SAM (RoBox-SAM) to ensure SAM's segmentation performance under prompts with different qualities.
Our contribution is three-fold.
First, we propose a prompt refinement module to implicitly perceive the potential targets, and output the offsets to directly transform the low-quality box prompt into a high-quality one.
We then provide an online iterative strategy for further prompt refinement.
Second, we introduce a prompt enhancement module to automatically generate point prompts to assist the box-promptable segmentation effectively.
Last, we build a self-information extractor to encode the prior information from the input image.
These features can optimize the image embeddings and attention calculation, thus, the robustness of SAM can be further enhanced.
Extensive experiments on the large medical segmentation dataset including 99,299 images, 5 modalities, and 25 organs/targets validated the efficacy of our proposed RoBox-SAM.






§ INTRODUCTION


The Segment Anything Model (SAM) has marked a significant breakthrough in image segmentation, due to its powerful zero-shot performance <cit.>.
SAM was trained on a large annotated dataset with 11M images and 1B masks based on the Vision in Transformer (ViT).
It supports different types of prompts, including point, box, text, and mask. 
Recent studies have widely explored the potential of applying SAM to the medical imaging domain <cit.>, which demonstrates that SAM can outperform the task-specific state-of-the-art conventional deep models after fine-tunning with additional medical images <cit.>.

Most previous studies equipped the SAM with high-quality prompts, e.g., the center point or tight box of the target.
However, this assumption typically cannot be held in clinical practice, where numerous low-quality prompts commonly occur.
As shown in Fig. <ref>, SAM prompted with high-quality boxes can achieve good performance, whereas it will easily generate poor segmentation after randomly shifting the promoting boxes. 
Hence, enhancing the robustness of SAM against imprecise prompts is crucial for its reliability.





Segmentation ability improvement of SAM.
Fine-tuning SAM using medical datasets to enhance medical image segmentation efficacy is an important field of research. 
Ma et al. ﬁxed the prompt encoder, and tuned the parameters in both image encoder and mask decoder during training <cit.>.
Wu et al. then introduced the adaptor modules with few trainable parameters to capture medical knowledge <cit.>.
Xie et al. reconstructed the mask decoder within SAM to develop a few-shot fine-tuning strategy <cit.>.
Moreover, Semantic-SAM <cit.> and HQ-SAM<cit.> were proposed to improve the segmentation quality. 
However, these studies aimed to improve the segmentation performance of SAM in different scenarios. 
They assumed to directly use the high-quality tight box as prompt, lacking the exploration of the low-quality prompt.


Robustness of SAM. 
Several studies have evaluated SAM's robustness under various corruptions and adversarial attacks <cit.>. 
However, their robustness tests were conducted at the image level rather than the prompt level. 
In <cit.>, the authors randomly shifted the tight box to mimic the practical operation.
They concluded that the SAM is sensitive to prompt randomness, especially box prompts.
The higher the random shifts, the more serious the performance degradation.
Although they have noticed the weak robustness of SAM under different prompt shifts, they did not introduce solutions to solve this problem.

One study observed that when subjected to low-quality prompts, SAM's mask decoder encounters an attention drift issue, leading to the cross-attention module activating inaccurate regions <cit.>.
To address this issue, stable SAM (SSAM) leveraged a deformable sampling plugin to shift attention to the target region.
However, SSAM only predicted the indirect feature offset to enhance the robustness.
Besides, it was developed and validated on natural images characterized by clear target boundaries, distinguishable foregrounds and backgrounds, and color encoding. 
These characteristics significantly differ from those of medical images, which may present unique challenges to the SSAM.


In this study, we introduce a novel robust box prompt based SAM named RoBox-SAM, aimed at making SAM robust to various prompt qualities.
Our contribution is three-fold.
First, we introduce an intuitive prompt refinement module (PRM) designed for potential target perception and directly predict box prompt offsets to refine low-quality prompts.
We also design an online refinement strategy to optimize the box prompt quality iteratively, and further achieve robustness improvement.
Second, we propose the prompt enhancement module (PEM), and leverage the movement information between the noisy and optimized prompts to output the potential point prompts automatically.
Finally, we extract intrinsic features from the input image via a self-information extractor (SIE), to enrich the image embeddings and refine the cross-attention calculation within the decoder. 
Validated on the large medical segmentation dataset, we demonstrate that RoBox-SAM is robust to box prompts of varying qualities, and general for different medical modalities and targets, achieving satisfactory results.





§ METHODOLOGY


Fig. <ref> shows the schematic view of our proposed Robox-SAM.

In stage 1, to eliminate the effects of low-quality prompts, the proposed PRM first predicts the offsets to transfer the input box prompt into a high-quality one.

In stage 2, we adopt the prompt correction knowledge to output potential points via the proposed PEM.

In the last stage, we leverage the prior knowledge encoded by SIE to optimize the image embeddings, and update the attention calculation in the mask decoder.
This can further optimize the feature activations under suboptimal box prompts, thus, improving the robustness of SAM.



Original SAM.
SAM mainly consists of three parts: image encoder, prompt encoder, and mask decoder <cit.>.
The image encoder is used to extract the image embedding, while the prompts are encoded using different methods (e.g., point and box: positional encoding <cit.>, text: CLIP <cit.>). 
Before being fed to the mask decoder, the image embedding should add the positional encoding, while the prompt tokens should concatenate with two output tokens, i.e., IoU token and mask token, to form the embedding (e) and tokens (t), respectively.
The mask decoder mainly contains two Transformer layers, and each layer contains four modules: 1) self-attention of tokens, 2) token-to-image cross-attention, 3) Multi-Layer Perception (MLP) on tokens, and 4) image-to-token cross-attention.
For example, the token-to-image cross-attention can be formulated as C_Att(t,e)=softmax(Q(t),K(e)^T/√(d_k))· V(e), where d(k) represents the dimension of the keys.
Finally, the mask token is separated from tokens and operated with the final up-scaled image embedding to predict multiple outputs.
Besides, the IoU token predicts the IoU score for each mask.



PRM for Target Perception and Box Prompt Correction.
Inaccurate box prompts may cause the SAM's mask decoder to activate imprecise regions, and the cross-attention between image features and tokens may further accumulate the errors, resulting in unstable segmentation <cit.>.

We propose the PRM to learn and perceive the target location, and predict the offsets to update the input prompts.
Given the image and prompt features (f_img, f_p), cross-attention can be performed to obtain f^*=f_img+C_Att(f_img,f_p).
Then, f^* will be inputted to the light-weighted PRM, outputting the offsets Δ(o)={Δ x_1, Δ x_2, Δ y_1, Δ y_2}.
The input box prompt (p_c={x_1,x_2,y_1,y_2}) can be updated to p_c^*=p_c+Δ(o).
The loss function can be written as L_o=L_1(p_c^*, p_gt), where p_gt represents the coordinate of GT tight box.
This supervision can drive the development of SAM's target perception ability, outputting reliable offsets to make the input box prompt move and match the potential target.
Moreover, during testing, the box refinement process can be interactive. Specifically, the output refined box in the current step can be considered as the input of the next step.
The iteration will terminate when ∑Δ(o)=0 or iterating over k=5 steps.
This interactive refinement strategy can further improve the model's accuracy and robustness.


PEM for Automatic Point Prompt Generation.
We only consider the box as the initial input prompt. 
However, we believe that the point prompts can assist the box prompts in some cases, especially negative points for restraining over-segmentation.
The above box correction process is intuitive and effective in revealing the potential region for point prompt generation.
Refer to stage 2 (right-down corner) in Fig. <ref>, given the original (green, A) and refined (red, B) boxes, and the annotated ground truth (GT) mask (orange, C), the non-GT region (i.e., (A∪ B)-C) indicated by yellow arrows can be treated as the potential negative area.
This region can be flattened into a one-dimensional vector and n uniformly sampled points can be taken as GTs (p_gt).
The CNN-based PEM (ℳ) can learn the potential points by inputting image-prompt attention features before and after refinement (p_pre=ℳ(f_r^*-f^*)).
Specifically, it generates n=5 points and matches them with sampled GT points one by one to find the combination with the smallest distance errors to calculate the L2 loss: L_p=||(p_pre,p_gt)||_2


Self-information Mining for Feature and Attention Optimization.
Utilizing the prior knowledge of the input image can enhance the accuracy and robustness of the segmentation model.
In the self-information pool, we provide several components that contain useful shape and edge information.
As shown in the green block of Fig. <ref> (stage 3), yellow arrows in the canny-based sketch <cit.> and SLIC-based superpixel <cit.> represent the location and boundary of the target.
We also adopt the fast Fourier Transform to transfer the image to the frequency domain <cit.>, including amplitude and phase parts.
Specifically, the low-frequency amplitudes are masked to highlight the details and edges of the image.
This information was encoded by the proposed SIE, and the output dimension was mapped to the same as the image embedding.
Then, the image embedding can be re-calculated by f'=f^*+f_s to fuse and emphasize the self-information.
Finally, f' will be input into the cross-attention module for attention re-calculation and optimization.
Thus, though the refined box prompts may still include noise, the inherent self-information can help remove it via attention update, and assist in improvement in terms of both accuracy and robustness.







§ EXPERIMENTAL RESULTS


Materials and Implementations. 
We validated the RoBox-SAM on a large medical segmentation dataset, including 99,299 images/slices, 5 modalities, and 25 types of targets.
This dataset contains 14 public and 5 in-house datasets (marked with ^*).
Specifically, for CT datasets <cit.>, the targets include chest (heart, lung, trachea) and abdomen (aorta, bladder, gall bladder, kidney, liver, pancreas, spleen, stomach).
MRI datasets contain the brain <cit.>, brain tumor <cit.>, knee (femur and tibia) <cit.>, and prostate <cit.>.
Ultrasound datasets have nine objects, including breast nodule^*, eyes^*, femur^*, heart <cit.> (left atrium, left ventricle endocardium, left ventricle epicardium), kidney pelvis^*, nuchal translucency^*, and thyroid nodule <cit.>.
We also considered two other targets from two modalities, i.e., Colonoscopy: polyp <cit.> and Fundus: OpticDisc <cit.>.
Approved by the local IRB, the in-house ultrasound datasets were collected and annotated by an experienced sonographer under strict quality control.
CT/MR volumes were sliced into 2D images similar to <cit.>.
The whole dataset was split into 6:1:3 for fine-tuning, validation, and testing, respectively.


We implemented RoBox-SAM in Pytorch, using an NVIDIA A40 GPU.
The batch size, learning rate, and total epoch are set to 16, 1e-4, and 20, respectively.
SAM's backbone was ViT-H.
GT box was generated by the adjacency box of the annotated mask.
Input box prompts were randomly shifted by 0-30% based on the GT box to evenly mimic the human operations with varying experiences/annotation preferences, with the condition that their IoU>0.5.
We followed the SAM to use cross-entropy and dice losses to optimize the model via AdamW optimizer.
We further added L_o and L_p to assist the model learning.
The proposed PRM, PEM, and SIE are two-layer CNNs with linear projection.
They encode the high-dimensional features and output offsets, points, or low-dimensional features to satisfy specific requirements.
As shown in Fig. <ref>, during fine-tuning, the original components in SAM were frozen, whereas other parts were learnable.
During testing, all the parameters were frozen, and the mask decoder will output three predictions with corresponding predicted IoU scores.
The mask with the highest IoU prediction was selected as the final output.






Quantitative and Qualitative Analysis.
We performed N=5 times random testing, and reported the average performance to obtain stable and reliable results.
Evaluation metrics in terms of accuracy (DICE, %) and robustness to prompt (PR, %: ∑_i=1^N(DICE(M_i, M_u))/N, similar to <cit.>), were used to validate the segmentation performance.
M_i represents the mask in i-th testing, while M_u is the union mask region of all N masks.
The higher the PR score, the more consistent the segmentation results are across box prompts of different qualities.


Table <ref> compares the RoBox-SAM with 7 methods, including SAM <cit.> (different model sizes: ViT-B/L/H), MedSAM <cit.>, MedSAM_boxaug, MedSAM_offset, and HQ-SAM <cit.>. 
All the MedSAM-based methods and HQ-SAM are based on ViT-H and fine-tuned using our medical datasets.
MedSAM_boxaug adds the random box augmentation strategy during fine-tuning, and MedSAM_offset trains an additional detector to predict the offsets for prompt correction before inputting to SAM. 
Based on the first five experiments, we concluded that enlarging the model size, fine-tuning using medical images, and augmenting with random boxes can optimize model robustness in most cases.

However, comparing MedSAM with MedSAM_offset, we found that adding naive offset prediction cannot steadily improve robustness, with 3/5 modalities having performance drops.
RoBox-SAM achieves the best performance among all the competitors (p<0.05), including the strong HQ-SAM, showing its efficacy and superiority in model robustness improvement.
Besides, RoBox-SAM is economical, has low training learnable computational overhead (<1M params), and takes only ∼0.6s per testing image.






We also performed ablation studies to evaluate the contribution of each proposed module in Table <ref>.
Specifically, we treated MedSAM as baseline, and progressively added the proposed modules to observe the performance changes.
It can be seen that PRM contributes ∼6% and ∼8% improvements to DICE and PR scores, respectively.
Adding PEM, the metrics can be increased further.
This illustrates that the generated point prompts can effectively assist box prompts in achieving better segmentation. 
By aggregating the prior self-information (RoBox-SAM), the model's performance can achieve consistent improvement across the five modalities.
It is highlighted that the performance improvement brought by each module is statistically significant with p<0.05.





In Table <ref>, we further decreased the quality of box prompts by progressively increasing the random shift max to 30%.
This can verify the model's performance under extreme conditions.
The lines GT mean that the tight boxes of objects are taken as input box prompts.
In this situation, although the PRM may not output large offsets, point prompt enhancement and self-information mining can still improve the segmentation.
Thus, RoBox-SAM shows a slight improvement compared with MedSAM under high-quality tight box prompts.
As the box prompt quality continued to decline, the accuracy and stability of MedSAM severely degraded.
For example, CT targets dropped about 50% in DICE values (89.06%→49.98%).
The performance of RoBox-SAM was less affected by low-quality box prompts (e.g., CT: 5.26%↓ in DICE).
This shows that RoBox-SAM can cope well with extreme box prompts and output more stable results than MedSAM.
Moreover, we provided the results of RoBox-SAM (Iter), and found that the interaction strategy can help boost the DICE and PR scores, especially for extremely low-quality box prompts (>1% significant improvement).

Fig. <ref> and Fig. <ref> are qualitative results, with DICE scores shown in the right-down corners of the images.
Fig. <ref> visualizes the segmentation results of compared methods and our proposed RoBox-SAM.
It can be observed that the original SAM shows poor performance under low-quality prompts.
RoBox-SAM performs better than other methods, approaching the upper-bound (RoBox-SAM_GT) and GT.
Fig. <ref> (a)-(i) shows the box refinement and performance comparison (in each sub-figure, from left to right shows GT, MedSAM, and RoBox-SAM results).
Given low-quality box prompts, MedSAM produces poor prediction, while RoBox-SAM can refine the box and output satisfactory results.
Fig. <ref> (j)-(k) shows the interactive prompt refinement process.
RoBox-SAM performs worse in the first-round refinement, but achieves good correction finally due to the iterative refinement strategy.
Fig. <ref> (l)-(m) reveals the robustness of RoBox-SAM. In each case, given three different random prompts (orange boxes), RoBox-SAM can refine them to the green ones, and output stable results. 





§ CONCLUSION


In this paper, we propose RoBox-SAM for improving SAM's robustness to box prompts of different qualities.
We first directly refine the input box with poor quality via the proposed PRM to obtain a high-quality box prompt.
Then, we automatically generate potential point prompts through REM to assist the box-promptable segmentation.
Last, we mine the self-information in the image to further ensure robustness via attention optimization.
Extensive experiments on the large medical dataset validate the efficacy of RoBox-SAM.
In the future, we will extend the framework to more modalities and 3D segmentation tasks.




  §.§.§ 

This work was supported by the grant from National Natural Science Foundation of China (12326619, 62101343, 62171290), Science and Technology Planning Project of Guangdong Province (2023A0505020002), and Shenzhen-Hong Kong Joint Research Program (SGDX20201103095613036).



  §.§.§ 

The authors have no competing interests to declare that are relevant to the content of this article.


splncs04




 Paper Title: Social Learning through Interactions with Other Agents: A Survey.


Confining modeling of quark propagator
    X.L. Shang
    August 5, 2024
======================================



§ ABSTRACT 
    Social learning plays an important role in the development of human intelligence. As children, we imitate our parents' speech patterns until we are able to produce sounds; we learn from them praising us and scolding us; and as adults, we learn by working with others. In this work, we survey the degree to which this paradigm – social learning – has been mirrored in machine learning. In particular, since learning socially requires interacting with others, we are interested in how embodied agents can and have utilised these techniques. This is especially in light of the degree to which recent advances in natural language processing (NLP) enable us to perform new forms of social learning. We look at how behavioural cloning and next-token prediction mirror human imitation, how learning from human feedback mirrors human education, and how we can go further to enable fully communicative agents that learn from each other.
We find that while individual social learning techniques have been used successfully, there has been little unifying work showing how to bring them together into socially embodied agents.




§ INTRODUCTION

Social learning is “learning that is influenced by the observation of another individual or their products” <cit.>. The “embodiment hypothesis” in cognitive science suggests that intelligence emerges as a result of embodiment within a social environment, that “starting as a baby grounded in a physical, social, and linguistic world is crucial to the development of the flexible and inventive intelligence that characterizes humankind” <cit.>. We can define this embodiment as being when “the environment is acting upon the individual and the individual is acting upon the environment” <cit.>. We explore how social embodiment – learning that is influenced by the actions and products of other agents in the learning environment – has played a role in the development of AI, and how it could influence the development of embodied agents in the real world.

Despite often not being recognised as such, a large proportion of AI research can be considered or framed as employing aspects of social learning. In particular, models trained on human-produced data often implicitly rely on learning to imitate humans. Take for example language model pretraining, in which the model is trained to directly reproduce human text. The form of this social learning can have important implications, for example through the reproduction of human biases in textual data <cit.> and facial recognition <cit.>. As such, the prevalence and impact of AI systems trained in this way warrants a more thorough investigation of the mechanics of social learning.

From another perspective, social learning techniques have largely seen success on supervised offline learning tasks with fixed datasets. Real-world deployment often requires accounting for more complicated, two-way interactions between agent and environment. This has traditionally been done through individualistic reinforcement learning methods, which have issues like sample inefficiency, unspecified reward functions, and poor explainability <cit.>. In contrast, humans – who develop in a socially embodied setting – are able to quickly adapt to changes in the environment. Since communication between agents has been made easier by the development of LLMs, it is an apt time to research the development of agents using social learning.





 §.§ Social Learning in Humans


Social learning plays an instrumental role in the way humans learn <cit.>. We do not learn individualistically by simply observing the world – instead, we take advantage of the fact that other humans have learnt before us and have built up cultural knowledge.
Indeed, Homo Sapiens' exploitation of cultural knowledge is exactly what has allowed us to build complex societies and dominate the globe. Rather than having bigger brains (and thereby having a cognitive advantage), our ability to “accumulate information across generations and develop well-adapted tools, beliefs, and practices that no individual could invent on their own” is responsible for our success <cit.>.

Similarly, the transmission and generation of this cultural knowledge is of paramount importance in human development at both an individual and societal level. Social learning enables this cultural transmission and development through our interactions with others. This suggests to us that the promising route to achieving more advanced agents lies not just in building bigger artificial brains (scaling) but also in building artificial brains socially, that can learn from and teach other agents, and take advantage of and develop this rich cultural knowledge base <cit.>. 

According to Tomasello's influential theory of Social Learning, it can be divided into Imitative Learning, Instructive Learning, and Collaborative Learning <cit.>.
The organisation of this survey follows this division, as depicted in Figure <ref>.



  
Imitative Learning.
In Tomasello's theory of Social Learning, children learn by copying adults in a number of ways. A distinction is drawn between emulation and true imitation. In the case of the former, the child attempts to reproduce the actions and outcomes of the adult demonstrator; in the latter, the child additionally models the intent behind the action and learns from that. Such true imitation seems to require at the least a nascent theory of mind, insofar as it requires modelling the intents of others.




  
Instructive Learning.
Instructive learning refers to the process by which an adult or expert learner tries to teach a non-expert learner. Other than explicit instruction, this process can include feedback, explanations, demonstrations and construction of a curriculum. Collectively, the process by which these mechanisms are employed to aid a learner is referred to as scaffolding <cit.>.



  
Collaborative Learning.
Collaborative learning is the setting in which neither or none of the collaborators are experts. In these cases, the learners must work together to progress in the learning task. 
From a developmental perspective, this emerges during early education – in contrast to imitation learning which is observed almost from birth <cit.>. However, it plays an important role in childhood development, especially in the development of language outside of their asymmetric interactions with adult teachers. For example, Vygotsky and Cole vygotsky1978mind emphasise the role of childhood play in enabling peers to practice social roles within their capabilities.



 §.§ Social Learning in AI


At first glance, some forms of social learning techniques have been widely used in AI at large <cit.>.
Diving deeper, we seek to make sense of these from a unified perspective, and examine whether there are more aspects of social learning that can be applied to AI.

In the simplest case, training data is generated or curated by humans. Furthermore, many models are trained to generate more of this human-supplied data – this often consitutes a form of imitation learning. For example, a generative image model may be trained to imitate the art or photography of humans. While such imitative models are powerful, they are also inherently limited to recreating the level of the source of their learning <cit.>. Several instructive techniques like feedback <cit.> and curricula <cit.> are also widely used in various domains. Finally, collaborative learning regimes have been explored in the Multi Agent Reinforcement Learning (MARL) literature <cit.>. All these techniques and their relations are depicted in Figure <ref>.



 §.§ LLM Agents


The development of LLMs poses a striking opportunity to enable embodied agents to learn socially. Indeed, early experiments with creating embodied agents almost entirely from LLMs have shown that such agents are able to achieve remarkable zero-shot performance on embodied tasks <cit.>. This has been boosted by the development of Vision Language Models (VLMs) and other multimodal models that have been used to create agents that can naturally leverage human demonstration data and cultural knowledge <cit.>. In addition, since they are language models, such agents possess an inherent ability to communicate with each other, without the need to develop such methods from scratch. This has accordingly enabled the creation of cooperative groups of agents <cit.>.



 §.§ Objective of This Survey


There has not been a recent survey on social learning covering the developments arising from large language models (LLMs). Meanwhile, a growing number of works have called for more exploration of social learning approaches <cit.>.
In this survey, we aim to provide a review of recent works that use social learning techniques with a view to building agents that can learn just through interacting with others. Given the relative maturity of imitation techniques for training generative models, in this survey we put more focus on the potential for training agents using instructive and  collaborative forms of social learning, as this has been most directly enabled by the emergence of LLM Agents.

As depicted in Figure <ref>, we first consider existing approaches to imitation learning, both in the wider machine learning literature, as well as in their application to building embodied agents. Relatedly, we briefly consider some of the work that has explored how to enable this imitation learning in Section <ref>. In Section <ref>, we then extensively explore how feedback has been used for aligning LLMs, and how richer forms of feedback might be leveraged. Many of these lessons are also applicable to Explanations and Instructions as briefly covered in Section <ref>. We then explore the degree to which curricula are useful for these embodied model given some negative results in Section <ref>. We cover how MARL has traditionally been used for collaborative learning in Section <ref> Finally, we cover the recent development of collaborative LLM-agents in Section <ref>.

More generally, by carefully considering the gaps between human social learning paradigms and current techniques used in training embodied agents we hope to inspire research that more fully utilises the advantages offered by social and cultural learning.



§ IMITATIVE LEARNING




 §.§ LLM Pretraining

Large language models are largely trained on the task of next token prediction – essentially predicting the next word in a sentence. This is arguably a form of imitative learning in so far as the model is fed observations of the productions of a human agent (the sequence of tokens), and learns to autoregressively predict that sequence of tokens. As indicated in Section <ref>, such models have achieved remarkable feats, to the point of matching humans on a wide number of tasks.

Nevertheless this acquisition process is highly data hungry – GPT-3 was trained on over 200 billion tokens, whereas the average 12 year old would have only seen roughly 100 million <cit.>. One potential explanation for this is that the techniques used for training LLMs do not consist in true imitation in the sense described by Tomasello et al. tomasello1993cultural and rather perform “emulation”. However, Kosinski kosinski2023theory argues that LLMs already possess theory of mind, although this position has been criticised <cit.>.

 More pointedly, during the training regime, a large language model is devoid of the socially-embodied context that typically accompanies a child learner, such as being able to build up a model of the social environment that may be useful for prediction. In this context, language models have been shown to recreate human biases, reproduce a static view in the face of changing social mores, and lack accountability <cit.>.

We have already seen how childhood development relies on other learning regimes <cit.>, therefore what remains to be seen is the development of models that can actually leverage this socially embodied setting to further learn and adapt through their collaboration with other agents. Indeed, this has begun to be explored through building social simulation environments <cit.>. Hopefully this will enable us to move beyond LLMs just acting as “tools of imitation” <cit.> and mitigate some of the social issues mentioned above.



 §.§ Imitation Learning

The field of Imitation learning itself (and indeed its dual – instruction by demonstrations) has been a popular reward-free method of training embodied agents. We do not aim to provide a comprehensive overview of the field, as covered by e.g. <cit.>. Key examples of techniques in the field include:

    
  * Behavioural Cloning – in behavioural cloning, direct access to the underlying policy of the teacher agent is assumed, and used as a training signal.
    
  * Inverse Reinforcement Learning (IRL) – in IRL, the reward function is generated from the teacher's policy and then used to train the student model by existing reinforcement learning algorithms.
    
  * Observational Imitation Learning (OIL) – Rather than assuming direct access to the teacher's policy, in observational imitation learning, the learner agent only has access to external observations of the teacher agent acting in the learning environment.

These imitation learning techniques been widely used in the development of embodied agents; For instance RT2 <cit.>, an embodied foundation model for robotics, was created by collating and training on a dataset made from a mix of human demonstrations and question answering web data. This model was then pretrained through OIL (in the form of next token prediction). Similarly, AlphaStar <cit.> was also pretrained from imitating games of professional players, and Ramrakhya et al. ramrakhya2022habitat utilise an embodied object-search strategy learnt from human demonstrations and robot trajectories.



  
Section Summary:
Imitative Learning techniques are widely used, including in RL and NLP. There is nevertheless valuable work to be done exploring ways to increase the sample efficiency of these techniques, and whether socially embodied settings may play a role in this.




§ INSTRUCTIVE LEARNING



 §.§ Demonstration


Demonstrations as an instructive learning paradigm consist largely in the provision of examples that a student agent can imitate. Importantly this means that the teacher may impart additional intent onto the demonstration through the choice and mode of presentation of those examples. For example a human teacher may draw attention to particular aspects of a demonstrated process by narrating it. Indeed human demonstrations are typically accompanied by explanations and narrations. Furthermore Ramrakhya et al. ramrakhya2022habitat verify that using imitation learning techniques on 70k human demonstrations outperforms using 240k agent trajectories, indicating that the quality of demonstrations can have a large impact. We have seen relatively little research exploring these elements of demonstration learning. Sumers et al. sumers2023show conduct a study on the provision of demonstrations to humans and find that if the demonstrations are not accompanied by explanations, they are less effective than linguistic interactions between teacher-student pairs for conveying concepts.

Bhoopchand et al. bhoopchand2023learning do explore some of the mechanisms at play with demonstrations, especially in the light of how they can be performed to maximise cultural transmission. They explore a navigation task with demonstrations performed by humans, that are then dropped out. Naive imitation (or rather emulation) of the experts enables the agent to perform well when the expert is there to follow, but once the expert is removed, these agents fail to generalise. The authors are able to create an agent that can imitate the plan of the teacher through equipping the agent with 1) memory, 2) an attentional bias towards the expert, 3) the expert “dropping out” during training, and 4) randomisation of the domain to ensure that the student agent is learning to utilise the expert and not just memorise the paths. This “attentional bias” may be related to the notion of intent as an aspect of “true imitation” <cit.> and further accords well with Tomasello's tomasello2009cultural theory of joint attention.


 §.§ Feedback


Feedback is just information given by the teacher on the performance of the learner. Consider the example of a student learning to perform addition. If they perform an incorrect addition like 18+17=25 we could provide a simple label evaluation (incorrect), a preferred answer (18+17=35), or language feedback like “you forgot to carry over the tens term” – the particular form of feedback that is appropriate will depend on the situation.

Labels are of course omnipresent in machine learning, as supervised learning forms a major paradigm. Learning from preferences has also been developed especially in the field of RL, where Reinforcement Learning from Human Feedback (RLHF) has now been adapted for finetuning language models <cit.>. While we mostly consider linguistic feedback, human feedback is often multimodal in nature, and may include gestures, tone, expressions and other physical actions. We encourage readers to refer to <cit.> for progress in these dimensions; we expect these to become increasingly feasible and important as more works appear exploring multimodal models.



  §.§.§ Preference-Based Feedback


Human preference-based feedback on generative AI output has proven invaluable for aligning the outputs of foundation models with human expectations through the use of `instruction tuning' <cit.>. This finetuning technique was designed for the purposes of alignment – the process of optimising a model according to human values and needs (e.g. following instructions). Typically this employs a three step process: 1) collecting demonstration data and performing supervised finetuning; 2) training a reward model on comparisons between model outputs; and 3) optimising the model as a policy against that reward model using an RL algorithm (PPO). The success of this approach has played an instrumental role in the recent rise of LLMs by providing a scalable and effective way to finetune models and control specific attributes.

Despite this, there have been a number of issues due to the cost of collecting human feedback data, and difficulty in optimising using PPO. One interesting development of late is Direct Preference Optimisation (DPO) <cit.> – which skips training the reward model and uses the LLM as an implicit reward model. This is essentially a KL-constrained contrastive loss that trains the model directly on preference datasets. Hejna et al. hejna2024contrastive point out that this method is not applicable to embodied trajectories and introduce a generalised version called “contrastive preference optimisation”, wherein a dis-preffered trajectory is contrasted to a preferred one.

These developments should enable us to train embodied agents built with multimodal language models (e.g. <cit.>) using preference optimisation and bring this aspect of feedback into embodied agents.



  §.§.§ Natural Language-Based Feedback


While it is relatively simple to use comparisons or scalar feedback scores, fully leveraging the richness of language feedback has proven more of a challenge.

Sumers et al. sumers2021learning investigate learning under the regime of “unconstrained linguistic feedback”. They categorise the feedback given as evaluative – scalar feedback similar to that described above, imperative – giving a preferred action e.g. “you should have carried over the tens term”, or descriptive – providing relevant information to the task. Sumers' approach to leveraging these rich rewards is to provide models that try to model the sentiment (a simple sentiment analysis model and a more complicated model handcrafted from Gricean pragmatics), as well as an inference network trained to predict the rewards given by the teacher. In their experiments, they firstly validate the importance of the more complex models that can leverage the richer language feedback, secondly they show that human teaching typically involves a mix of all three categories of feedback in human experiments.

More recently, several works have exploited the recent advances in NLP to use LLMs to directly update the reward functions used for training embodied models in accordance with the feedback. For example in Eureka <cit.> the authors train a robotic hand to execute complicated gestural object manipulation (spinning a pen around) by updating a reward function from observations. Similarly, Yu et al. yu2023language use LLMs to define reward parameters that are then optimised over for a variety of robot embodiments including a quadruped and arm based robot. They find that this approach is generalisable and achieves 90% success rates across a wide variety of tasks. The import of such approaches is that they enable the utilisation of LLM's high-level cultural knowledge about actions to be translated into low-level actions.

A promising alternative solution to this is to use the feedback to condition future actions of the learning agent, and rely on existing learning mechanisms to learn from these actions. Scheurer et al. scheurer2022learning use this paradigm to learn from natural language feedback by generating multiple responses conditioned on feedback. The response that is semantically most similar to the feedback is then treated as the “correct” response and used to finetune the model. Similarly, Chen et al. chen2024learning explore “Imitation learning from Language Feedback” in which they let the learner refine its answers based on human feedback, and then finetune on these refinements.

Zha et al. zha2023distilling further explore the mechanisms of memorising and retrieving feedback for conditioning embodied agent actions on the fly. They create a system for distilling corrections into retrievable knowledge that can be used by the LLM-agent in future generations. They use the feedback to firstly perform replanning, and then to create a correction that can be placed in a knowledge base (rephrased to be “contextually complete”) and retrieved in future situations where relevant. In particular, they use both visual and textual representations for the correction embedding so that it can be retrieved when a situation is similar in either dimension.



 §.§ Instruction and Explanation


Instruction following itself has been widely used as a task in the alignment of models. In particular, the RLHF framework described in Section <ref> was popularised in NLP for instruction following <cit.>. This alignment process (learning to follow instructions) has enabled foundation models to learn to perform complex tasks from instructions. A large body of works explore methods of instruction in the form of prompting techniques like chain of thought <cit.>, that have been shown to improve reasoning abilities. For example, Mu et al. mu2024embodiedgpt finetune their embodied model on complicated instructions that require the use of chain of thought planning over egocentric videos.

Instruction following has also seen use as a paradigm in the popular benchmarks TEACh <cit.> and ALFRED <cit.>. TEACh consists in human-human dialogue sessions where one human is playing the role of a teacher or commander; the other acts as a student and must follow the instructions of the teacher, while being able to ask questions for clarification. ALFRED is a relatively simple task and involves following singular grounded instructions like “walk to the coffee maker on the right”. The latter instruction can be seen as an example of the vision-language navigation task, which is currently one of the more popular paradigms in embodied AI, given that it was recently enabled by increased performance of LLMs. These benchmarks however do not focus on actually learning from the interactions with the instructors per se, nor indeed are the instructions targeted at the learning agent.

Saha et al. saha2024can explore more deeply modelling interactions between students and teachers using theory of mind. One of their research questions is the degree to which personalisations of explanations aid students. While they do find a positive effect, its influence is relatively small. While personalisation of explanations have not been found to be beneficial thus far, in an embodied RL setting it was found that the act of giving explanations itself was helpful in aiding the teacher learning <cit.>. This is an example of the so called protégé effect – when we try to explain or instruct others we test and expand our own understanding of a concept by reconceptualising it.

While it would be desirable to heavily use targeted instructions and explanations by asking for help when needed, the provision of help, especially in the case of humans in the loop, is expensive. Indeed per the theory of scaffolding, we expect that as the learner gains proficiency, the learning experience can be less structured, and intervention can be reduced over time. As such one body of works considers the problem of efficiency by learning when to ask for help. For example Ren et al. ren2023robots train an agent to ask for help when there is a high level of uncertainty in the task. The agent generates a set of possible actions, and if the likelihood of the likeliest of them being correct (as measured by the agent) is below a certain threshold, the robot then asks for further instruction.



 §.§ Curricula


Curriculum Learning is a machine learning approach that parallels the scaffolding of human learning at a more abstract level. The classic paradigm introduced by Bengio et al. bengio2009curriculum is to order the training data by complexity for some notion of complexity, and progressively increase the level of complexity as the learning progresses. Per Figure <ref>, curriculum learning consists in the teacher agent altering the learning environment rather than directly interacting with the learner agent.

The idea behind this is that by structuring the data in this way, the initial learning task becomes more feasible, and after basic concepts are acquired, these can be used as a basis for learning more complex concepts. While this seems simple, there are two key problems. First of all defining complexity for a given task is difficult – the curriculum may have to be tailored for individual learners, and require insight into the current capabilities of the learner. Additionally it may be unclear when the learner has sufficiently mastered a concept to enable the complexity to increase.

Accordingly a body of works exists that try to automate this process. This approach is typically called automatic curriculum learning, and is a popular paradigm in the reinforcement learning community (ACL). For instance, Matiisen et al. matiisen2019teacher explore a “Teacher Student Curriculum Learning” paradigm in which a teacher agent monitors the student's training progress and is used to determine the tasks suitable for the student. The mechanism used is the student's progress on a task – the teacher increases sample weight for tasks that are seeing progress – the weight is then reduced when the task has been mastered since the rate of progress is measurably decreased. The authors treat the problem as a bandit learning problem and explore a number of sampling algorithms, finding that such an automated curriculum can help for maze solving in minecraft, and simple addition tasks using LSTMs.

Other works such as Ramrakhya et al. ramrakhyal2023curriculum and Morad et al. morad2021embodied use estimations of task success probabilities to order the experience while training embodied navigation systems. In particular, they aim to find tasks of `intermediate difficulty'. Interestingly, in the former they note that the addition of easy tasks to the training mix without curriculum learning per se has a similarly beneficial effect, as the learning agent simply spends less time in easier episodes since it can achieve them quickly.

Despite their success in RL domains,curriculum-based approaches have not seen as much success in NLP or computer vision. Recently Warstadt et al. warstadt-etal-2023-findings ran a competition to train LLMs using a human-developmentally plausible dataset and found that although curriculums were the most popular approach explored by entrants, it was relatively ineffective for improving sample efficiency. Indeed the key negative result of their competition report was exactly that none of the common curriculum learning approaches offered substantial benefits to the developmental task of language learning. Given the differing success of curriculum learning approaches it is likely that it's benefit is largely dependent on how structured the learning problem is.

Indeed embodied language models may be a more suitable domain for the application of curriculum learning methods. Wang et al. wang2024voyager leverage the LLM's knowledge of task properties to derive new suitable tasks. Rather than relying on handcrafted curricula or designing a separate automatic curriculum, they simply prompt the LLM to come up with tasks itself, conditioning the new tasks on the state of the agent, previously learnt skills, the previous task completion history, and the long term goals of the agent. They find that this automatic curriculum is both critical for the agent's success, and furthermore beats out manually defined expert curricula. This process of giving the agent meta-learning abilities is promising but likely requires more principled analysis; in any case it speaks to the potential of using LLM-Agents to instantiate social learning paradigms in a unified way.



  
Section Summary: More principled work remains to be done on demonstration learning, such as investigating whether ToM can enable student agents to imitate more efficiently. Feedback has been widely used in the form of preference learning. Leveraging general language feedback has seen increasing use but requires more research. Approaches like conditioning on feedback likely have similar applications to leveraging explanations and instructions. Finally, while curriculum learning has been widely used in RL, it is not as promising for training LLMs – this may not however be the case for LLM-Agents.



§ COLLABORATIVE LEARNING



 §.§ Multi-Agent Reinforcement Learning (MARL)


Historically the majority of research on collaborative learning has occurred in the context of multi-agent reinforcement learning. For instance AlphaStar <cit.> was trained to grandmaster human level in the game StarCraft II through playing with other players. The system was initialised using imitation learning, and then further finetuned by an RL algorithm to perform well against a mixture of opponents. The solution proposed by the authors is an evolution of previous so called “self-play” strategies. This consists in pitting different versions of an agent against itself in strategy games. The key insight here was to firstly add a greater mix of adversarial agents (previous versions of the learning agent) in addition to agents that were specifically designed to exploit the flaws of the agent in question.

Recently, an AI agent (Cicero) was trained for the game Diplomacy at a human level, ranking in the top 10% of participants in an online league <cit.>. In Diplomacy, players take the role of diplomats controlling countries, and have to negotiate with other players, make promises, and deceive them. By leveraging pretrained LLMs to conduct the negotiation and model other users' intents, a strategic reasoning model could then be used to plan actions and intents and was then trained with a combination of RL methods including self-play and behavioural cloning.

In the more cooperative setting, a number of recent works have focused on the “learn to communicate” paradigm <cit.>. In these paradigms, the sets of agents may have to learn both what and when to communicate, and even some explorations of who to communicate to via attention mechanisms <cit.>. Many of these works have started from a point of zero linguistic competence and require the agents to learn from scratch. Accordingly a future trend will likely include the incorporation of existing language models <cit.>, similarly to the approach taken in the work of FAIR et al. meta2022human described above.



 §.§ Collaborative LLM-Agents


A significant recent trend in NLP has been exploring the use of communicative LLM-agents. Firstly, Park et al. park2023generative explored using such generative agents as models of human behaviour – in particular, they simulated a town of agents that perform daily activities and interact in natural language, enabled by a game environment that tracks the memories and positions of each agent. 

These behaviours, including emergent abilities at party planning, were rated as being largely believable as human by crowdworkers.

Later, Qian et al. qian2023communicative explored the creation of software using teams of LLMs mimicking the `waterfall' model of software development. The authors broke down the development process into a sequence of discussions between LLM agents acting in roles (e.g. CEO, programmer etc.). In each discussion the LLMs would act in turn as instructors and assistants, making decisions through concensus. In contrast to simply generating the codebase for a software development project directly using an LLM designed for coding, they managed to achieve an ∼88% success rate on achieving the software development task.

Chen et al. chen2024agentverse then went further to provide a more general framework for communicative LLM based agents. The process proceeds in rounds made up of 1) expert recruitment – building up a group of roles to perform, 2) decision making – deliberating over what to do 3) action execution, and 4) evaluation in which the outcome is compared to the goal and feedback is given.

In the embodied setting, Zhang et al. zhang2024building have explored building cooperative embodied LLM agents. This reportedly requires more complicated models and theory of mind - in particular their agents consist in modules handling belief, communication, reasoning, and planning. The belief module is intended to give the agents theory of mind by tracking the knowledge of the scene, other agents, the self, and the task. On benchmarks, the authors found that the LLM agents were able to cooperate well on various benchmarks, and that the theory of mind was necessary for this success.

Such cooperative LLM agents are already able to outperform state-of-the-art MARL agents on several tasks, such as navigation <cit.>, without the extensive number of training episodes typically required for the MARL agents. Importantly however, in all these collaborative settings, no learning per se is going on – the agents are able to achieve complicated tasks through collaboration but only “learn” by updating the memory or context used in their internal decision making processes (since they are frozen language models).


More generally, we expect that collaborative settings are more useful for generating experience trajectories that would be unachievable for singular agents which can then be leveraged for learning using ordinary learning mechanisms, rather than being a source for learning mechanisms per se, in a similar way to Scheurer et al. scheurer2022learning exploring conditioning on feedback as a way of learning from natural language feedback rather than providing a separate feedback learning mechanism per se.



  
Section Summary: Collaborative Learning has been explored mostly through Multi Agent Reinforcement Learning. These have included interesting studies on optimising the population of collaborators and emergent communication. In the case of LLM-Agents, there have been remarkable achievements through applying prompting, including application of ToM, but thus far limited work directly updating the weights of the models constituting these agents, limiting their adaptability.




§ CONCLUSIONS AND FUTURE DIRECTIONS

Although social learning is not typically conceived as a key paradigm or organising principle for machine learning, we have shown that it permeates a large proportion of the machine learning literature.


Moreover, we have seen how performant LLMs have enabled the application of more advanced social learning techniques using the preferred mode of human instruction – natural language. For example, in Section <ref> we showed that this has enabled the construction of models that can directly respond to language feedback, and in Section <ref> how they had been used to build communicative agents purely from language models that were able to outperform existing MARL approaches.



  
Comparison and Combination of Social Learning Paradigms.
As shown in Table <ref>, the majority of approaches explored in this paper use multiple social learning techniques. However, only four use all three of the main categories and are largely not embodied. As we have argued, equipping agents with linguistic capabilities has enabled agents to leverage communication and instructive learning paradigms out of the box. Given this, we expect to see more work exploring the combination of these approaches.



  
Learning Permanently through Social Conditioning.
The promising LLM-Agent approaches have thus far largely relied on the knowledge acquired by the LLM through imitation learning. There has been little work performing further gradient updates once these models are socially embodied, and the “learning” has largely constituted updates to the agent's memory modules (as in <cit.>). Accordingly, identifying how models can learn more permanently in these settings remains a research priority. We see a promising direction in utilising low-level learning methods on experiences conditioned on the social interactions that entail “higher-level” learning mechanisms like feedback and instruction. This has seen some application in the non-embodied setting <cit.>, but we expect this to be replicated for embodied settings in the near future as we move beyond simply using LLM-agents without finetuning.


  
Multimodality.
Our survey has been primarily concerned with linguistic approaches, however the recent development of multimodal models will likely enable richer forms of social interaction (and thereby learning). We have already seen this in the case of demonstration learning from vision and language <cit.>, but given the relative immaturity of video-foundation models it may be some time before we are able to fully leverage non-linguistic signals like gestures and visual demonstrations for teaching embodied agents. As discussed in Section <ref>, existing techniques for learning from feedback could be applied to learn from these signals.


  
Theory of Mind and Intent.
As can be seen in Table <ref>, there have been a number of approaches exploring the use of Theory of Mind as a prerequisite for richer interactions between teachers, students, and peers. Indeed Theory of Mind cuts across social learning. Modelling the intent of a teacher enables a learner agent to fully leverage the learning signal, as we discuss in Section <ref>. Modelling teacher agents also allows learners to evaluate the teacher's potential as sources of learning <cit.>. While ToM modules are a relatively common aspect of agents, especially collaborative agents, we have seen little work exploring these techniques so far. As such, we expect to see more work investigating the effects of these modules, especially given the interest in ToM initiated by Kosinski kosinski2023theory.



  
Application Domains.
Social learning techniques are broadly applicable in the field at large. Nevertheless, they may play an especially important role in domains where social skills themselves are more relevant. For instance, in the case of socially-assistive robotics, the ability to adapt to individual needs and build a relationship over time is especially important <cit.>. Similarly, the ability to use models to teach humans skills would be a significant benefit of developing instructive social learning techniques. For example, this has explored in the case of social skill training <cit.>. In the meantime, we expect the majority of progress to occur in using social-learning techniques for training agents in video games, due to the low cost of training, ample amounts of training data, and ease of interacting with humans. Games are already a test-bed for many approaches covered above <cit.>, but we can expect social learning to enable richer forms of collaborative play with AI agents, as explored for example in the work of Gong et al. gong2023mindagent.




§ ACKNOWLEDGMENTS

We would like to thank Chloe He for her invaluable suggestions. This work was supported by an A*STAR CRF award to C.T., as well as an A*STAR SINGA scholarship to D.H.


named




 Paper Title: Analyzing Customer-Facing Vendor Experiences with Time Series Forecasting and Monte Carlo Techniques.


    


    











    








eBay]Vivek Kaushik
vikaushik@ebay.com
eBay]Jason Tang cor
jastang@ebay.com

[eBay]eBay Inc., 2025 Hamilton Ave., San Jose, CA 95125
[cor]Corresponding author
[t1] This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors






§ ABSTRACT 
eBay partners with external vendors, which allows customers to freely select a vendor to complete their eBay experiences. However, vendor outages can hinder customer experiences. Consequently, eBay can disable a problematic vendor to prevent customer loss. Disabling the vendor too late risks losing customers willing to switch to other vendors, while disabling it too early risks losing those unwilling to switch. In this paper, we propose a data-driven solution to answer whether eBay should disable a problematic vendor and when to disable it. Our solution involves forecasting customer behavior. First, we use a multiplicative seasonality model to represent behavior if all vendors are fully functioning. Next, we use a Monte Carlo simulation to represent behavior if the problematic vendor remains enabled. Finally, we use a linear model to represent behavior if the vendor is disabled. By comparing these forecasts, we determine the optimal time for eBay to disable the problematic vendor.



    Seasonality Exponential smoothing Regression Combining forecasts Business cycles


    










§ INTRODUCTION


In today’s landscape of rapid innovation, applications often use the services of third-party vendors to enable additional functionality quickly and easily. eBay is no exception. As an eCommerce company, eBay partners with many external vendors to serve various business functions. In some cases, multiple vendors support the same business function in order to provide the richest experience for all customers. 

As with any complex system, however, vendors are subject to outages during which they cannot operate normally. These outages could impact customers' ability to properly complete their experiences on eBay. Ideally, if a specific vendor is experiencing minimal impact during an outage, a customer using that vendor could have an initial failed experience with it, but should eventually have a successful experience upon retrying. 

However, if the vendor is experiencing a high degree of impact, then customer retries may not be successful. Customers with unsuccessful retries may choose to switch to a different vendor to complete their experiences. If the vendor's impact is severe enough, then eBay can temporarily disable, or wire off, the vendor, meaning customers would no longer see that vendor as an option. Consequently, more customers will choose to complete their experiences with a different vendor. At the same time, historical data has shown many customers have a very strong preference for using a particular vendor and will choose not to complete their experience if it is not shown as an option. We capture these effects in Fig. <ref>.




eBay's operations team makes the decision to wire off a problematic vendor. During past incidents, however, these decisions have not necessarily been data-driven. The decision to wire off a vendor is a critical one for the business; when it is executed, it typically is when the problematic vendor is experiencing a full outage, which means no customer requests with that vendor can successfully be served. During this type of situation, customers who prefer to only use the problematic vendor will not complete their experiences at all. On the other hand, customers who could otherwise use a different vendor are stuck retrying with the problematic vendor as long as it is being displayed to them. Over time, this results in net fewer successful customer experiences than if eBay had wired off the problematic vendor earlier during the incident.

In this paper, we present a data-driven approach to determine whether a given problematic vendor should be disabled and provide an optimal time for the wire-off. This optimal disabling time maximizes the total number of successfully completed customer experiences across all vendors. 

First, as an intermediary step, we forecast each vendor's baseline customer behavior. More specifically, we forecast the total number of successful customer experiences for each vendor assuming all vendors are fully functioning. Historical data has shown baseline customer experience exhibits weekly seasonal behavior. Moreover, the seasonality varies with a trend. Therefore, for each vendor, we use a multiplicative seasonality model to represent its baseline customer behavior. We demonstrate how to forecast the seasonal and trend components associated with the baseline model, which in turn allows us to forecast each vendor's baseline customer behavior.

Next, we forecast the wired-on customer behavior. More specifically, we forecast the total number of successful customer experiences across all vendors assuming the given problematic vendor remains enabled. In order to accomplish this, we first forecast the problematic vendor's availability, which we interpret as the probability that a customer would have an initial successful experience with the problematic vendor. Historical incidents have shown that the problematic vendor's availability exhibits a decreasing trend over time; therefore, we use a Double Exponential Smoothing model to represent its availability. We describe the procedure to compute the smoothing and trend components associated with this model from historical data, which in turn allows us to forecast the problematic vendor's availability. Then, we consider several probability distributions that are constructed from historical incident data during which the problematic vendor remained enabled. These distributions model the following customer decisions: retrying the experience after an initial unsuccessful attempt with the problematic vendor, switching to another vendor, and the time between retries. We incorporate the baseline customer experience forecasts from the previous paragraph, the problematic vendor's availability forecast, and the aforementioned customer decision distributions into a Monte Carlo simulation. We run the Monte Carlo simulation to project future decisions made by customers pertaining to their experiences. Finally, we aggregate these simulation results together to forecast the wired-on customer behavior.

Lastly, we forecast the wired-off customer behavior. More specifically, we forecast the total number of successful customer experiences across all enabled vendors assuming the given problematic vendor is disabled. Historical incidents have shown that customers who migrate from using the problematic vendor to using any other vendor typically do so at a constant rate over time. Therefore, we use a simple linear model to represent the wired-off customer experience. The independent variable in the model is the disabled vendor's predicted baseline customer experiences as described above, while the dependent variable is the difference between the total number of successful customer experiences during the wire-off and total number of predicted baseline customer experiences across all the enabled vendors. Forecasting the wired-off customer behavior amounts to determining the slope of the regression line.


Finally, in order to arrive at a data-driven decision for wiring off the given problematic vendor, we compare the wired-on and wired-off customer behavior forecasts from the previous two paragraphs. In particular, we check whether there is a future time at which the total number of predicted wired-off customer experiences begins to exceed the total number of predicted wired-on customer experiences. If such a time exists, then we choose to disable the problematic vendor at that particular time. Otherwise, we keep the problematic vendor enabled.









§ CUSTOMER BEHAVIOR MODELS
 

Customer behavior is the underlying factor driving eBay's business metrics. In order to determine the optimal wire-off time for a given problematic vendor, we will need to consider models that represent distinct types of customer behavior. We use these models to forecast each vendor's volume, which we interpret as the total number of customer experiences with that vendor. Throughout this section, we assume there are N total vendors for some fixed positive integer N.



 §.§ Baseline Model
 

Consider the baseline situation in which all N vendors are fully functioning, so that any customer is guaranteed to have an initial successful experience with their choice of vendor. We discuss how to forecast each vendor's baseline volume. For each 1 ≤ n ≤ N and each t ≥ 0, we let C_n,t be the n-th vendor's baseline volume at the time t. Vendor volume is typically observed every minute, so we assume all times have granularity in minutes. In order to clearly distinguish between past, present, and future times, we will index time in the following way: we fix a t_0 to be the current time, and for each m ∈, we let t_m be the time corresponding to m minutes since t_0. For any m<0, we interpret t_m as a historical time and for any m>0, we interpret t_m as a future time. We assume that volume for current and historical times is known, while volume for future times is unknown. We also assume that volume is positive for all times. 

Historical evidence has shown each vendor's baseline volume exhibits weekly seasonality along with trend. Moreover, due to varying customer activity levels throughout time, the seasonality directly varies with the trend (see Fig. <ref>). Therefore, we use a multiplicative seasonality model <cit.> to represent each vendor's baseline volume. Equivalently, we use an additive seasonality model to represent the logarithm of each vendor's baseline volume. Explicitly, for the n-th vendor, we consider the model: 

    
    ln(C_n,t_m) = ln( s_n,t_m ) + ln(g_n,t_m) +ϵ_n,t_m,


where s_n,t_m and g_n,t_m are the seasonal and trend components at time t_m, respectively, and {ϵ_n,t_m}_m ∈ is an independent and identically distributed sequence of Gaussian random variables with mean 0.



  §.§.§ Seasonal Component

We discuss how to forecast the n-th vendor's seasonal component in Eq. (<ref>). To do this, we consider the following model for the logarithmic seasonal component:

    
    ln(s_n,t_m) = ∑_h=1^H_nα_n,hcos(2 π   h t_m/L) + β_n,hsin(2 π    h t_m/L) + ζ_n,t_m


for each m ∈, in which 
    
        
  * H_n is the known number of harmonics;
        
  * L=10080 is the total number of minutes in a week;
        
  * α_n,h and β_n,h are the unknown Fourier coefficients for each 1 ≤ h ≤ H_n;
        
  * {ζ_n,t_m}_m ∈ is an independent sequence of Gaussian random variables with mean 0 and common variance σ_n^2 for some constant σ_n>0.
    
More compactly, for each m ∈, we can rewrite Eq. (<ref>) as 

    
    ln(s_n,t_m) = 𝐱_n,t_m^⊤ β_n  + ζ_n,t_m,


where
    
    𝐱_n,t_m   = ( cos(2 π (1) t_m/L)  ,  sin(2 π (1) t_m/L)  ,  … ,  cos(2 π (H_n) t_m/L)  ,  sin(2 π (H_n) t_m/L)  )

is the vector of seasonality features at time t_m and 
    
    β_n    = (α_n,1 ,  β_n,1 ,  … ,  α_n,H_n ,  β_n,H_n) .

is the vector of unknown Fourier Coefficients. 


Suppose 𝐬_n,M ={ s_n,t_m}_m = -M^0 ∈^M+1 is a vector of seasonal component historical observations. Let ln(𝐬_n,M) = {ln(s_n,t_m) }_m = -M^0 be the vector as a result of taking the logarithm of each historical observation. Forecasting the seasonal component s_n,t_m for m>0 amounts to determining the Fourier coefficient vector β_n. We utilize Maximum a Posteriori (MAP) estimation, the main technique used in <cit.>. The goal of MAP estimation is to seek the parameter β_n ∈^2H_n maximizing f (  β_n  |  ln(𝐬_n,M) ), the conditional probability density function of the parameter β_n given the observed data. Equivalently, via Bayes Rule, we seek the parameter β_n maximizing the function ϕ: ^2H_n→ defined by
    
    ϕ(β_n)    = f (   ln(𝐬_n,M)   |  β_n )  f(β_n),

where f(β_n) is a specified prior probability density function of the parameter β_n. We assume β_n has a Gaussian prior distribution with mean 0 and covariance matrix (σ'_n )^2  𝐈_2 H_n for some fixed seasonality prior scale σ'_n>0, where 𝐈_2H_n is the 2H_n × 2H_n identity matrix. Moreover, by our model assumptions, ln(𝐬_n,M) is Gaussian with mean 𝐗_n,M β_n and covariance matrix σ_n^2  𝐈_M+1, in which 𝐗_n,M is the (M+1) × 2H_n matrix with m-th row equal to 𝐱_n,t_1-m^⊤. Therefore, we can write out the function ϕ in Eq. (<ref>) explicitly:
    
    ϕ(β_n)    = exp(-ln(𝐬_n,M) - 𝐗_n,M β_n ^2/2 σ_n^2)   exp(- β_n^2/2 (σ_n')^2)/(2 π)^M+2/2σ_n σ'_n 
       = exp(-ln(𝐬_n,M) - 𝐗_n,M β_n ^2/2 σ_n^2 - β_n^2/2 (σ_n')^2)/(2 π)^M+2/2σ_n σ'_n .

Computing the gradient of ϕ with respect to β_n, we obtain
    
    (∇ϕ)(β_n)    = -( 𝐗_n,M^⊤ 𝐗_n,M β_n - 𝐗_n,M^⊤ ln(𝐬_n,M) /σ_n^2 + β_n/(σ_n' )^2)  ϕ(β_n)  .

Upon setting the gradient equal to 0 and solving for β_n, we obtain the MAP estimate
    
    β_n    = ( 𝐗_n,M^⊤ 𝐗_n,M + (σ_n/σ_n')^2 𝐈_2H_n)^-1𝐗_n,M^⊤ ln(𝐬_n,M) .


Finally, for each m>0, we compute s_n,t_m, the predicted seasonal component of the n-th vendor's baseline volume at time t_m, by taking the inner product 𝐱_n,t_m^⊤ β_n and exponentiating the result.









  §.§.§ Trend Component


Next, we forecast the n-th vendor's trend component in Eq. (<ref>). Let { u_n,d}_d=1^D_n is a collection of D_n known times during which the vendor's n-th trend growth rate changes. We refer to such times as trend changepoints. Consequently, for each m ∈, define 𝐚_n,t_m∈^D_n to be the vector whose d-th component is 1 if t_m ≥ u_n,d and 0 otherwise. We assume the trend component obeys a piecewise linear model of the form:

    
    ln(g_n,t_m) = (κ_n+𝐚_n,t_m^⊤ δ_n ) t_m + 𝐚_n,t_m^⊤γ_n +θ_n + ζ'_n,t_m


for each m ∈, where 
    
        
  * κ_n ∈ is an unknown base growth rate;
        
  * δ_n ∈^D_n is an unknown vector of growth rate adjustments;
        
  * γ_n ∈^D_n is an unknown vector of continuity correction factors;
        
  * θ_n ∈ is an unknown base offset;
        
  * {ζ'_n,t_m}_m ∈ is an independent and identically distributed sequence of Gaussian random variables with mean 0.
    
In particular, the expression κ_n+𝐚_n,t_m^⊤ δ_n appearing in Eq. (<ref>) is the sum of the base growth rate κ_n and all D_n growth rate adjustments at time t_m. Meanwhile, the vector γ_n ensures the map t ↦ln(g_n,t) is a continuous function and is determined once δ_n is determined. 

Assume 𝐠_n,M = { g_n,t_m}_m=-M^0 ∈^M+1 is a vector of historical trend component observations, where M is a fixed positive integer. Let ln(𝐠_n,M) = {ln(g_n,t_m) }_m=-M^0 be the vector formed by taking the logarithm of each historical trend component observation. In addition, let { u_n,d}_d=1^D_n be a collection of known historical trend changepoints with D_n<M+1. Forecasting the trend component g_n,t_m for each m>0 amounts to determining the parameters κ_n, δ_n, and θ_n. We consider the MAP estimation technique, in which we seek the parameters maximizing the conditional probability density function f(κ_n, δ_n, θ_n  |   ln(𝐠_n,M) ). Following <cit.>, we impose the prior assumptions that the parameters κ_n and θ_n are both Gaussian with mean 0, and changepoint prior scale λ_n>0, we impose the prior assumption δ_n has Laplace distribution with mean 0 and covariance matrix λ_n  𝐈_D_n. We let κ_n, δ_n, θ_n be the resulting MAP estimates, and we set γ_n to have d-th component equal to -u_n,d δ_n,d, for each 1 ≤ d ≤ D_n, where δ_n,d is d-th component of the MAP estimate δ_n.


Now, for each m>0, we compute g_n,t_m, the predicted trend component of the n-th vendor's baseline volume at future time t_m. We randomly declare the time t_m to be a trend changepoint with probability D_n/M+1 and not a trend changepoint with probability 1-D_n/M+1. For each m>0, we randomly choose the future growth rate δ_n,D_n+m at time t_m according to the aforementioned prior Laplace distribution if t_m is labeled a trend changepoint and set δ_n,D_n+m=0 otherwise. Consequently, we set γ_n, D_n +m = - t_m  δ_n,D_n+m and compute

    
    ln(g_n,t_m)    = ( κ_n + (1  ,  … ,  1_D_n+m times)^⊤ (δ_n     ,  δ_n,D_n+1 ,  … ,  δ_n,D_n+m) )  t_m  
            + ( (1  ,  … ,  1_D_n+m times)^⊤ (γ_n   ,  γ_n,D_n+1 ,  … ,  γ_n,D_n+m)  + θ_n
            ),


where
    
    (δ_n   ,  δ_n,D_n+1 ,  … ,  δ_n,D_n+m)

is the (D_n+m)-dimensional vector formed by concatenating the MAP estimate δ_n from the previous paragraph with the vector (δ_n,D_n+1 ,  … ,  δ_n,D_n+m), and (γ_n   ,  γ_n,D_n+1 ,  … ,  γ_n,D_n+m) is defined similarly. We take the exponential of Eq. (<ref>) to obtain the predicted trend component at time t_m.

Finally, for all m>0, we obtain C_n,t_m, the  n-th vendor's predicted baseline volume at time t_m, by multiplying the corresponding seasonal and trend component forecasts at time t_m together:

    
    C_n,t_m=s_n,t_m g_n,t_m.
 

Henceforth, we refer to the computed number C_n,t_m as the expected baseline volume for vendor n at time t_m.




 §.§ Wired-On Model
 

Suppose one particular vendor is experiencing an outage, but it still remains enabled to customers. Throughout this section, we let 1 ≤ n_0 ≤ N be the unique index corresponding to the problematic vendor. We describe our approach to forecasting the total wired-on volume, which is the total volume across all N vendors. First, we forecast the availability of vendor n_0, which we interpret as the probability that a customer would have a successful experience with it on their first attempt. Then, we incorporate the n_0-th vendor's availability forecast and the expected baseline volume forecasts described in the previous section into a Monte Carlo simulation. At each future time, we use the Monte Carlo simulation to project the ultimate decision of a customer initially using the problematic vendor along with the time they make the decision. Finally, we aggregate the Monte Carlo simulation results together to forecast the wired-on volume.



  §.§.§ Vendor Availability Model
 

Let a_n_0,t be the problematic vendor's availability at time t. Vendor availability is not necessarily known at future times, and there is often limited data from the past. Furthermore, when a vendor is experiencing issues, its availability typically exhibits a decreasing trend. Therefore,  we consider a Double Exponential Smoothing model <cit.> to represent the problematic vendor's availability, which is explicitly

    
    S_n_0,t_m   = α  (a_n_0,t_m) + (1-α)   (S_n_0,t_m-1 + b_n_0,t_m-1) 
    
            b_n_0,t_m   = η  (S_n_0,t_m - S_n_0,t_m-1) + (1-η)   b_n_0,t_m-1,


for each m ∈, where
    
        
  * α∈ [0,1] is an unknown smoothing factor;
        
  * η∈ [0,1] is an unknown trend factor;
        
  * S_n_0,t_m∈ is an unknown smoothing component;
        
  * b_n_0,t_m∈ is an unknown trend component;
    

Let {  a_n_0,t_m}_m=-M^0 be a historical sequence of the problematic vendor's availability observations for some M>0. Forecasting the problematic vendor's availability a_n_0,t_m for all m>0 amounts to determining the factors α and η based on the historical observations. First, we initialize the smoothing component in Eq. (<ref>) and the trend component in Eq. (<ref>) by setting
    
    S_n_0,t_-M   :=a_n_0,t_-M
    
            b_n_0,t_-M   :=a_n_0,t_-M+1-a_n_0,t_-M.

With these initialized values, we seek the optimal factors α and η that minimize the quantity

    
    { a_n_0,t_m - S_n_0,t_m -b_n_0,t_m}_m=-M^0  _2/√(M+1)   =  √(1/M+1∑_m=-M^0 (a_n_0,t_m - S_n_0,t_m -b_n_0,t_m)^2),


the root mean-squared error between {  a_n_0,t_m}_m=-M^0 and { S_n_0,t_m +b_n_0,t_m}_m=-M^0. The root mean-squared error is a continuous function of the tuple (α,η) with domain equal to the unit square [0,1]^2. By the extreme value theorem, we are guaranteed optimal factors α and η that minimize the root mean-squared error. Seeking such factors, however, is a nonlinear optimization problem that cannot be solved analytically. Nevertheless, if we can determine these optimal factors, then we can compute a_n_0,t_m, the problematic vendor's predicted availability at time t_m for each m>0, via the formula

    
    a_n_0,t_m = S_n_0,t_0 +  m   b_n_0, t_0.






  §.§.§ Customer Behavior Monte Carlo Simulation
 

We now describe the Monte Carlo simulation to forecast customer behavior when the problematic vendor remains enabled. Recall n_0 is the index corresponding to the problematic vendor. The Monte Carlo simulation combines together each vendor's expected baseline volume forecasts, the problematic vendor's availability forecast, and certain probability distributions modeling specific customer decisions. We compute these distributions over a historical time period during which the problematic vendor had previously experienced issues.

First, for each k ≥ 0, we consider the retry distribution, which is a Bernoulli distribution with parameter

    
    π_n_0,k = (a customer retries with any vendor |  customer has failed k times with n_0),


where  stands for probability. The second distribution is the switch distribution, which is another Bernoulli distribution with parameter 
    ρ_n_0,k = (customer switches to another vendor |  customer is retrying after failing k times with n_0).

Lastly, we consider the interattempt time distribution, which has cumulative distribution function

    
    τ_n_0(s) = (customer takes at least s seconds to retry with any vendor |  customer failed with n_0),


where s is a non-negative integer.




Given a customer who initially attempts to use vendor n_0 at time t_m for some integer m, we describe the Monte Carlo simulation to project their ultimate decision, which we illustrate in Fig. <ref>. We initialize the the number of failures k = 0. In order to capture time delayed effects during our actual time of interest, we initialize m to be some negative integer. We simulate the outcome (success or failure) of the customer's initial attempt with the problematic vendor according to the Bernoulli distribution with parameter a_n_0,t_m as defined in Eq. (<ref>). When m ≤ 0, we use the problematic vendor's actual availability a_n_0,t_m in place of the predicted availability. If the customer succeeds, then we record that outcome at time t_m and terminate the simulation. Otherwise, we increment k and simulate the customer's decision whether to retry the experience with any vendor based on the retry distribution defined in Eq. (<ref>). If the customer decides to abandon the experience, we record their decision and terminate the simulation. Otherwise, we simulate the customer's decision whether to switch to another vendor based on the switch distribution defined in Eq. (<ref>). We also simulate the time ε (in seconds) taken for the customer to execute their decision based on the aforementioned interattempt time distribution defined in Eq. (<ref>). We terminate the simulation if the customer switches to another vendor and record this decision at time t_⌊ m+ 
ε/60⌋. Otherwise, we repeat the entire  procedure, but we replace m with m+
ε/60. This process is described in Algorithm <ref>.



Finally, we forecast the total wired-on volume by aggregating the Monte Carlo simulation results from the previous paragraph. Recall that C_n_0,t_m is the problematic vendor's baseline volume at time t_m, as defined in Eq. (<ref>). Here, we interpret the integer ⌊C_n_0,t_m⌋ as the predicted total number of customers who use the problematic vendor at time t_m. For each 1 ≤ i ≤⌊C_n_0,t_m⌋, we project the i-th customer's ultimate decision and their time of decision based on the Monte Carlo simulation. Now, we let A_n_0,t_m be the total number of customers projected to have successful experiences with the problematic vendor at time t_m. We also let A_other,t_m be the total number of customers projected to have successful experiences with other vendors at time t_m. Lastly, we let C_other,t_m be the total sum of all expected baseline volume forecasts across all vendors excluding the problematic vendor. Using these predicted numbers, we compute W_n_0,t_m,on, the predicted wired-on volume at time t_m, via the formula

    
    W_n_0,t_m,on = A_n_0,t_m + A_other,t_m +C_other,t_m.


 


















 §.§ Wired-Off Model
 
Now, suppose the problematic vendor n_0 is disabled. Let W_n_0,t_m, off be the total wired-off volume at time t_m, which is the total volume across all enabled vendors at time t_m.  In this situation, customers will not be able to retry their experiences with the problematic vendor. Past incident data has shown that the ratio of additional volume for vendors other than the problematic one to the baseline volume of the problematic vendor is stationary over time (Fig. <ref>); we performed an Augmented Dickey-Fuller test <cit.> which resulted in a p-value of 2.82 × 10^-18. Therefore, we do not need to consider any time-dependent effects of the rate at which customers who typically use the problematic vendor would switch to use another vendor. We consider a simple linear model of the form

    
    W_n_0,t_m, off  = Δ_n_0C_n_0,t_m + C_other,t_m + ϵ_n_0, t_m,




to represent the wired-off volume, where for each m ∈, 
    
        
  * C_n_0,t_m is the disabled vendor's expected baseline volume at time t_m, as defined in Eq. (<ref>);

        
  * C_other,t_m is the total expected baseline volume across all enabled vendors at time t_m;

        
  * Δ_n_0∈ is the unknown slope;

        
  * {ϵ_n_0,t_m}_m ∈ is an independent and identical sequence of Gaussian random variables with mean 0;
    

Forecasting the wired-off volume amounts to determining the slope Δ_n_0 based on historical data over a timeframe during which problematic vendor n_0 was disabled. To this end, let T_hist be a finite collection of historical times corresponding to a previous known incident during which the problematic vendor was disabled. Suppose we have a vector 𝐖_n_0, T_hist, off = { W_n_0,t, off}_t ∈ T_hist containing the total wired-off volume observations over T_hist. Via Eq. (<ref>), we compute the predicted vector 𝐂_n_0,T_hist = {C_n_0,t}_t ∈ T_hist containing the expected baseline volume observations corresponding to the disabled problematic vendor. Then, we compute the predicted vector 𝐂_other,T_hist =  {∑_n  n_0C_n,t}_t ∈ T_hist, in which each observation at time t is the sum of forecasted baseline volumes at time t across all enabled vendors. Now, we can obtain an estimate for the slope Δ_n_0 by solving the linear equation 

    
    𝐖_n_0, T_hist, off = Δ_n_0𝐂_n_0,T_hist + 𝐂_other,T_hist,


for Δ_n_0. Upon subtracting  𝐂_other,T_hist from both sides of Eq. (<ref>), left multiplying both sides by 𝐂_n_0,T_hist^⊤, and finally dividing both sides of the equation by (𝐂_n_0,T_hist) ^ ⊤𝐂_n_0,T_hist = 𝐂_n_0,T_hist_2^2, we obtain the estimate

    
    Δ_n_0   = (𝐂_n_0,T_hist)^⊤(𝐖_n_0, off, T_hist - 𝐂_other,T_hist) /𝐂_n_0,T_hist_2^2.


Thus, for all m>0, we can compute W_n_0,t_m, off, the predicted wired-off volume at time t_m, via the formula
    
    W_n_0,t_m, off = Δ_n_0C_n_0,t_m + C_other,t_m.


















§ RESULTS AND VALIDATION


In this section, we discuss sample results and the validation of our forecasting models on vendor volume data. We apply our methodology to real incidents and analyze customer and business benefits.




 §.§ Expected Baseline Volume Forecast Validation


For each 1 ≤ n ≤ N, recall that we represent the n-th vendor's expected baseline volume through the additive seasonal model for its logarithm as defined in 
Eq. (<ref>). In order to forecast the baseline volume, we leverage Prophet, a forecasting library developed by Meta. In Section <ref>, we discussed how to forecast the trend and seasonal components individually via MAP estimations given historical data associated with such components. For computational efficiency, however, Prophet computes a single MAP estimate for the concatenated list of parameters Φ_n=(β_n, κ_n, δ_n, θ_n), in which we recall β_n is the vector of Fourier Coefficients in the seasonal component model, and (κ_n, δ_n, θ_n) is the list of parameters in the trend component model. This single MAP estimate for Φ_n is computed based on the n-th vendor's historical baseline volume data.

Moreover, Prophet exposes several hyperparameters to control model fitting. We list the main ones that affect forecasting in Table <ref>. The default values of such hyperparameters, however, typically do not result in a good fit for vendor data. Therefore, we employ an efficient random hyperparameter search <cit.> to determine an optimal set of hyperparameters that minimizes the root mean-squared error between the actual future baseline volume { C_n,t_m}_m ≥ 0 and the expected baseline volume {C_n,t_m}_m ≥ 0 computed via Eq. (<ref>). We list the hyperparameters that we tune, along with their lower and upper bounds over our search space, in  Table <ref>. Furthermore, in Fig. <ref>, we display an example of forecasting a vendor's baseline volume based on two weeks of past historical volume data as a result of the hyperparameter grid search, computing the MAP estimate for the concatenated list of parameters Φ_n, and using Eq. (<ref>).










 §.§ Availability Forecast Validation
 

Recall n_0 is the index corresponding to a problematic vendor and that we forecast the problematic vendor's availability using a Double Exponential Smoothing model governed by Eq. (<ref>) and Eq. (<ref>). We demonstrate how we validate such forecasts. Because the availability data is limited, we perform the validation on a rolling basis. We fix a positive integer W, the rolling window size in minutes, and we fix a positive integer R, the forecasting horizon size. For each M ≥ 0, we consider the historical availability rolling window time-series {  a_n,t_m}_m=M-W^M, consider a Double Exponential Smoothing model for the time-series, and compute the rolling horizon forecast time-series {a_n,t_m}_m=M+1^M+R via Eq. (<ref>). We choose the optimal smoothing and trend factors α and η in Eqs. (<ref>) and (<ref>), respectively, again employing an efficient random hyperparameter search. Finally, we compute the associated rolling horizon mean-squared error between the rolling true horizon time-series {  a_n,t_m}_m=M+1^M+R and the rolling horizon forecast time-series {a_n,t_m}_m=M+1^M+R. The closer the rolling horizon mean-squared errors are to 0, the better the corresponding rolling horizon availability forecasts are. We show a sample true historical availability time-series, rolling horizon forecast time-series, and associated rolling horizon mean-squared errors in Fig. <ref>. We see that the Double Exponential Smoothing models are acceptable because their associated root mean-squared errors are low.






 §.§ Wired-Off Volume Forecast Validation


Recall when the problematic vendor n_0 is disabled, we use a simple linear model to represent the total wired-off volume across all enabled vendors, as presented in Eq. (<ref>). More specifically, we fit a simple linear model of the difference between the total wired-off volume and the aggregate sum of enabled vendors' expected baseline volumes against the disabled vendor's expected baseline volumes. 

We perform the regression diagnostics to validate the use of the linear model. For each m>0, let W_n_0, t_m,off and W_n_0, t_m,off be the actual wired-off volume and predicted wired-off volume at time t_m, respectively. For each m>0, we define the residual 
    
    r_n,t_m   = W_n, t_m,off- W_n, t_m,off,

at time t_m. We show the results of three different wired-off linear models in Fig. <ref> corresponding to three different situations when a problematic vendor was actually disabled. 

Firstly, in a generic linear model, the plot of the residuals against the predicted values is symmetrically distributed about the horizontal axis. In the second row of Fig. <ref>, we plot the corresponding residuals against predicted values for each wired-off linear model. We see these plots exhibit visible symmetry about the horizontal axis. 

Next, in a generic simple linear model, the recursive residuals, which are the standardized one time-step ahead prediction errors, are independent and identically Gaussian distributed. In order to validate this for our sample wired-off linear models, we formally perform the Harvey-Collier test <cit.> at a fixed percent significance level. In this test, the null hypothesis states the recursive residuals are normally distributed. Therefore, we need to check whether we do not reject the null hypothesis. We compute Harvey-Collier test statistics corresponding to our sample wired-off linear models and tabulate their p-values in the first column of Table <ref>. At the 5 percent significance level, since the p-values are much greater than 5 percent, we do not reject the Harvey-Collier null hypothesis in any of our wired-off linear models.

Next, in a generic simple linear model, the majority of auto-correlations at lag 1 lies within the 95 percent confidence bound about 0 i.e. approximately within [ -√(2/M), √(2/M)], where M is the number of observations. Moreover, we formally perform the Durbin-Watson test <cit.>, whose null hypothesis states there is no auto-correlation within the residuals. In a generic linear model, the residuals are independent, and so we need to check whether we do not reject the null hypothesis. A heuristic condition to not reject the null hypothesis is if the Durbin-Watson test statistic lies in the interval [1.5,3.5]. We tabulate the Durbin-Watson test statistics corresponding to our sampled wired-off linear models in the second column of Table <ref>. Because all our three obtained test statistics lie between 1.5 and 3.5,  we do not reject the Durbin-Watson null hypothesis in any of our sample wired-off linear models.

Lastly, in a generic linear model, the residuals are normally distributed. To validate this, we inspect the quantiles of the standardized residuals against the theoretical quantiles of the standard normal distribution and check whether the observations lie close to the straight line y=x.  For our sample wired-off linear models, we plot the corresponding quantiles of the standardized residuals against the theoretical quantiles of the standard normal distribution in the last row of Fig. <ref>. We see these plots exhibit visible symmetry.










 §.§ Wire-off Recommendation



With the forecasts {W_n,t_m, off}_m = 1^R  and {W_n,t_m, on}_m = 1^R, where R is the fixed future horizon size defined in Section <ref>, we determine the smallest index m^* in which  


    W_n,t_m, off > W_n,t_m, on


for all m ≥ m^* in our simulated window. If m^* exists, we make a decision to wire off vendor n starting at time t_m^*. Otherwise, we keep vendor n wired on.





Figure <ref> shows our simulated trends for a real incident in which a vendor was experiencing issues. The optimal wire off time according to Eq. (<ref>) is considerably earlier than the actual wire off time. We define t_actual to be the actual wire-off time and t_m^* to be the optimal wire off time based on Eq. (<ref>) and our simulation. Table <ref> shows the lead time t_actual - t_m^* of our optimal simulated wire off time over the actual wire off time for several historical incidents. We see that our optimal wire off time is often several minutes before the actual wire off time. Depending on the experience and time of day, this can equate to a considerable number of additional completed customer experiences.




Thus, using our approach, we can see benefits to both our customers and the business over the existing solution. In the short term, we are immediately providing an improved experience which reduces friction for our customers. In the long term, the cumulative improved experiences will grow over several vendor incidents; this not only means more overall customer experiences completed, but also greater customer satisfaction over time, which will drive people to continue using eBay. 








§ FUTURE WORK

In this paper, we predicted customer behavior to arrive at the decision to wire off a problematic vendor. A natural question to ask is when a disabled vendor should be enabled again for customers. We hope to use the methodology in this paper to arrive at a data-driven answer to this question. This problem poses several additional challenges; namely, we would only like to re-enable a vendor when we know its impact has been mitigated. However, after the vendor is wired off, we no longer see real-time metrics and must therefore find another way to measure its availability. A solution for this may depend on the data available for particular customer experience and/or vendor. 

Furthermore, recall the key assumption for our approach to forecasting completed customer experiences is the existence of only one problematic vendor. In many situations, however, more than one vendor could experience issues, which would severely impact eBay's business. We hope to modify the probabilistic model in Section <ref> in order to forecast customer behavior when multiple problematic vendors are either wired on or wired off. 







elsarticle-harv







 Paper Title: CXSimulator: A User Behavior Simulation using LLM Embeddings for Web-Marketing Campaign Assessment.







0009-0003-8870-8010

  CyberAgent, Inc.
  Tokyo
  Japan

kasuga_akira@cyberagent.co.jp


0000-0002-2724-6233

  CyberAgent, Inc.
  Tokyo
  Japan

yonetani_ryo@cyberagent.co.jp



e.g.
c.f.
i.e.
et al. 
etc

§ ABSTRACT 
This paper presents the Customer Experience (CX) Simulator, a novel framework designed to assess the effects of untested web-marketing campaigns through user behavior simulations. The proposed framework leverages large language models (LLMs) to represent various events in a user's behavioral history, such as viewing an item, applying a coupon, or purchasing an item, as semantic embedding vectors. We train a model to predict transitions between events from their LLM embeddings, which can even generalize to unseen events by learning from diverse training data. In web-marketing applications, we leverage this transition prediction model to simulate how users might react differently when new campaigns or products are presented to them. This allows us to eliminate the need for costly online testing and enhance the marketers' abilities to reveal insights. Our numerical evaluation and user study, utilizing BigQuery Public Datasets from the Google Merchandise Store, demonstrate the effectiveness of our framework. 



<ccs2012>
   <concept>
       <concept_id>10002951.10003227.10003241.10003243</concept_id>
       <concept_desc>Information systems Expert systems</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010341.10010370</concept_id>
       <concept_desc>Computing methodologies Simulation evaluation</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Simulation evaluation
[500]Information systems Expert systems



CXSimulator: A User Behavior Simulation using LLM Embeddings for Web-Marketing Campaign Assessment
    Ryo Yonetani
    August 5, 2024
==================================================================================================



§ INTRODUCTION



Sophisticated web marketing tools, such as Google Analytics <cit.> and Braze <cit.>, have been used to measure the behavior of users and assess marketing strategies. However, only fully developed web services can use these tools optimally. A/B testing, though commonly employed, is expensive to execute repeatedly and challenging to ensure a sufficient number of target users. Expert marketing knowledge and experience are necessary to draw useful insights from user behavior. As a consequence, it is not trivial to benefit from such tools in the early stages of service or in situations where there is a shortage of experts.

We focus on the potential of LLMs to solve this issue. LLMs have been applied not only for natural language processing tasks <cit.> but also for common sense reasoning in multi-modal data <cit.>. We believe that the ability of LLMs, especially to represent the high-level semantics of complex event descriptions with compact embedded vectors (, LLM embeddings) <cit.>, can also be advantageous for web marketing applications

In this work, we propose a novel framework named CXSimulator (Figure <ref>). CXSimulator can facilitate a fast offline assessment of the effects of marketing campaigns via LLM-based user behavior simulation, in the absence of costly A/B testing and expert knowledge. The proposed framework describes user behaviors on e-commerce sites with an event transition graph, where nodes represent individual events such as viewing an item and clicking a coupon, while edges indicate the transition from one event to another. In the graph, we embed the detailed description of each event into a semantic latent vector using LLMs, and train transition prediction models that use the LLM embeddings to predict connections and transition probabilities between events. The trained models can be employed to simulate plausible user behaviors, even when the graph includes a new node absent in the training data. This way, our approach can assess the effectiveness of new, untested campaigns by comparing simulated user behaviors with and without the nodes and measuring the difference in conversion rates.

We evaluate the effectiveness of the proposed CXSimulator using BigQuery Public Datasets <cit.> from GA360 <cit.>, which comprises real user behavior histories on the Google Merchandise Store. The experimental results reveal that our transition prediction models outperform relevant link prediction methods <cit.>, as well as other LLM-based solutions using GPT-3.5/4 <cit.>. Furthermore, our user study with five domain experts validates a high correlation between the CXSimulator's assessments and expert marketers' judgments regarding untested campaign effectiveness. The contributions of this paper are summarized as follows:



  * We present CXSimulator, a framework that models user behaviors on e-commerce sites using event transition graphs.

  * We demonstrate the effectiveness of using LLM embeddings to predict transition probabilities between events, which allows us to simulate user behaviors even in the presence of a new and untested campaign.

  * We evaluate the effectiveness of a new campaign between the proposed method and domain experts.





§ RELATED WORK



User Behavior Modeling and Simulation.
The recent growth in e-commerce has highlighted the importance of analyzing extensive user behavior logs <cit.>. User behavior modeling is employed in search <cit.> and recommendation systems <cit.>. Recent work has investigated methods that employ graph neural networks <cit.> and long short-term memory <cit.> to session-based user behavior histories. Moreover, LLM technologies have been advantageous for web content generation <cit.> and creative image generation <cit.>. However, online evaluation methods present challenges due to their high costs and scalability issues <cit.>. Conversely, there has recently been increasing interest in offline evaluation through simulation. Simulators have been used for evaluating conversational recommendation systems <cit.> and information retrieval methods <cit.>. Despite such developments, research has been limited in scope. This background motivates us to employ marketing campaign simulation to address this research gap.

Link Prediction, which aims to anticipate relationships between nodes, serves various purposes such as network reconstruction, recommendation systems, and knowledge graph completion <cit.>. In e-commerce applications, the primary focus is on predicting purchase and viewing relationships between users and items <cit.>. Nevertheless, studies on predicting transition probabilities for behavior transition graphs are scarce. Existing methods use Node2Vec <cit.> or graph auto-encoder <cit.> for link prediction <cit.>, but these approaches have limitations in representing knowledge outside of the training data. Recently, methods have been developed to directly predict links by fine-tuning node features and graph structures as prompts in knowledge graphs <cit.>, as well as techniques for direct link prediction in zero-shot scenarios <cit.>. Consequently, this paper leverages the potential of LLMs for common sense reasoning <cit.>, and employs LLM embeddings of user behavior for link prediction.


§ CXSIMULATOR FRAMEWORK



As illustrated in Figure <ref>, the CXSimulator incorporates three main components: a) user behavior modeling based on an event transition graph, b) prediction of transition probabilities through LLM embeddings, and c) user behavior simulation for marketing campaign assessment. Each of these components is detailed below.



 §.§ User Behavior Modeling

Consider users who are interacting with an e-commerce service. Their behavior history is represented as a sequence of events such as “viewing page A”, “obtaining coupon B”, “adding item C to the cart”, and “checking out the item”. A weighted directed graph naturally serves to represent the transitions among these events. Specifically, let G=(V, E) be a digraph, referred to as the event transition graph, where V denotes the nodes each corresponding to individual events. E=V × V indicates the edges that stand for transitions between events. Each edge carries a transition probability, p_i, j∈ (0, 1] for a pair of vertices v_i, v_j ∈ V, {i,j | 1,2,…,m}. The existence of edges and probabilities for linked nodes can be acquired from a corpus of user behavior data in the same segment such as cities and devices. Once this is achieved, a plausible user behavior history can be simulated by traversing the event transition graph according to these transition probabilities.



 §.§ Transition Probability Prediction

Our key question arises here: how can we simulate user behaviors if we add a new event to the event transition graph? Such situations would frequently occur in web-marketing scenarios, for example, when a new item is put on sale or a new discount campaign is launched. The effects of events, , whether these events contribute to the conversion rate, can vary based on user behaviors within a given segment. Ideally, we expect to simulate such effects before the items/campaigns are presented to real users. 

This situation can be viewed as a variant of link prediction problems <cit.>, where appropriate feature extraction for nodes is essential for high prediction performance. In this work, we propose to leverage the reasoning ability of LLMs to represent the semantic aspects of events and user segments with compact embeddings. Google Analytics provides detailed events in a well-structured format, such as . Likewise, some attributes of user segments are readily available in a similar format, such as . We transform such events and segment descriptions into LLM embeddings, and employ them as node features. 

Formally, we solve two-fold tasks: 1) binary classification for the existence of edges between a pair of nodes, and 2) regression for the transition probabilities of the edges when they exist. Let f_i, f_j, f_seg∈ F be the LLM embedding vectors from nodes v_i, v_j as well as the segment attribute, where F∈ℝ^d is the d-dimensional embedding space. We concatenate these vectors to use as the input to classifier 𝐶𝑙𝑠:F× F× F →{0, 1} and regressor 𝑅𝑒𝑔:F× F× F → [0, 1]. Note that filtering the input with the classifier before regressing the transition probabilities yields better performance than directly performing the regression for all possible pairs of nodes, regardless of the existence of edges. This is due to the wide variety of nodes, the sparsity of edges, and the resulting data imbalance.



 §.§ Simulating Campaign Effects

This section describes how to simulate user behavior for a new event and how to utilize these simulations to assess web-marketing campaigns. To achieve this, we consider two event transition graphs: the Control G_C=(V_C, E_C), where the edge connectivity and weights are derived solely from user behavior histories, and the Treatment G_T=(V_T, E_T), which has an expanded node and edge collection with a new event v_new. While it is simply V_T=V_C+{v_new}, we span edges between v_i∈ V_C and  v_new based on the output from the trained models of 𝐶𝑙𝑠(f_i, f_new, f_seg) and 𝐶𝑙𝑠(f_new, f_i, f_seg), and also estimate the transition probability with the trained 𝑅𝑒𝑔(f_i, f_new, f_seg) and 𝑅𝑒𝑔(f_new, f_i, f_seg) for the spanned bidirectional edges, where f_new is the LLM embedding vector on v_new.

Beginning from the same event node (, “session start”), we sample multiple event sequences from both G_C and G_T until we reach the ending node (, “session end”) or exceed the maximum sequence length. Each sequence signifies plausible user behavior with or without the intervention of the new event v_new. A collection of these sequences allows us to predict the potential difference in conversion rates using standard marketing analysis methods.


§ EXPERIMENT



We employed the BigQuery Public Datasets <cit.> from GA360 <cit.> to conduct numerical experiments on transition probability prediction and a user study for assessment of new campaigns.



 §.§ Numerical Evaluation

Datasets.
The dataset <cit.> includes e-commerce events (, click through of product lists, product detail views, add product(s) to cart, check out, and completed purchase) and detailed descriptions (, pageTitle, pagePath, productName, localProductPrice,…). It spans a total period of 366 days, from 2016/08/01 to 2017/08/01, amounting to 4,153,675 records. We designated the training, validation, and testing periods as 2016/08, 2016/09, and 2016/10, respectively, ensuring no seasonal differences in the data during these periods.[The limited granularity of event descriptions (due to ID mapping) prevented the utilization of other popular datasets, including CIKM2016EComm <cit.> and REES46 <cit.>.]

Implementation Details.
We implemented 𝐶𝑙𝑠 and 𝑅𝑒𝑔 using LightGBM <cit.>, a lightweight and fast model to ensure timely execution that preserves user experience. In the training data, the total number of nodes amounted to 1,350, with 19,126 linked edges out of total 1,822,500 edges. We applied Laplace smoothing α=5 to mitigate the zero frequency issue due to data sparsity. The class weight in Cls was considered based on edge sparsity and imbalance. The event description was vectorized to d=128 by Azure OpenAI Service  <cit.>.
The main parameters are, for 𝐶𝑙𝑠, {objective: binary,  scale_pos_weight: 5.0, num_iteration: 1500, metric: auc}, and for 𝑅𝑒𝑔, {objective: regression, num_iteration: 1500, metric: rmse}. Both models have the early_stopping_round parameter set to 50. We validated prediction accuracy for transition probabilities p_i,j of all node pairs v_i,v_j in the testing period. Furthermore, we compare accuracy using all combinations of text-embedding-3-small (v3-small) <cit.> and text-embedding-3-large (v3-large) <cit.>, as well as either 𝐶𝑙𝑠 or 𝑅𝑒𝑔 solely.

Baselines.
As baseline methods, we predict all transition probabilities p_i,j using Node2Vec <cit.> based on the graph structure, and graph auto-encoder (GAE) <cit.> that considers both the graph structure and node descriptions. With Node2Vec and GAE, an embedding vector of d=16 effectively described a node feature. 
Moreover, we adopted GPT-3.5/4 <cit.> via few/zero-shot <cit.> with prompts[The prompt begins as follows: Given a src node description and dst node description, your task is to predict transition probability(0.0-1.0) of the edge between 2 nodes.] consisting of detailed event description to directly predict p_i,j.

Evaluation Metrics.
Our evaluation metrics include: 
RMSE minimizes the error of p_i,j, a continuous value. 
SMAPE <cit.> evaluates relative error in probability regression p_i,j. 
F1-Score represents the harmonic mean of precision and recall. 
ROC-AUC measures a model's ability to correctly classify between positive and negative cases.
PR-AUC provides a more appropriate metric in situations with class imbalance than ROC-AUC.
We adjusted the decision threshold <cit.> in 𝑅𝑒𝑔 and predicted probability in 𝐶𝑙𝑠 to align with their objectives and evaluate all metrics consistently across all methods. The optimized decision threshold and predicted probability were also 0.1 in this experiment.

Results.
Table <ref> demonstrates that our models outperform the baseline methods. Using 𝐶𝑙𝑠 and 𝑅𝑒𝑔 with v3-small is deemed more practical due to cost efficiency, since the difference in metrics between v3-small and v3-large is not significant. The GPT-3.5/4 baseline showed limited performances because it could only receive 20-30 samples in a prompt.



 §.§ User Study

We conducted a user study with marketers to validate the practical utility of the CXSimulator in actual business environments.

Participants. We recruited five marketers where three are senior (over 3 years of experience) and two are junior ones (1-2 years of experience). They participated in the two studies described below.

Study 1: Validity of simulating the difference in conversion rates.
In this study, we assess the concordance between the difference in conversion rates simulated using the CXSimulator framework and the judgements of marketers. Each participant was asked to assess the effectiveness of 35 campaigns which have never been implemented in the store, for example, Enjoy 1 month Free of YouTube Premium for YouTube related Product!, by analyzing the GA data from 2016/08/01 to 08/31 on BigQuery within 40 minutes. The 35 campaigns consisted of 5 variations each in 7 categories, and comprehensiveness of the assessment is reasonable. Each campaign was scored on a four-point scale: effective (4), somewhat effective (3), not very effective (2), and not effective (1). For when it was difficult to judge, an option of unable to judge was also provided. On the other hand, we simulated 10,000 user sessions from each of G_C and G_T during the same period in our framework and measured conversion rates. Finally, we evaluate the correlation between the difference in conversion rates and human assessments.

Study 2: Usefulness of proposed simulator UI.
Furthermore, we developed a UI to experience the CXSimulator. As shown in Figure <ref>, our UI consists of five components: 1) data view and analysis, 2) event input forms (, campaignTitle, actionType), 3) Simulation run button, 4) Control/Treatment CVR comparison, 5) Control/Treatment user behavior data view and analysis.
Participants were asked to input their own campaigns, for example, {actionType: Click a campaign page, campaignTitle: Recommend the top 5 best-selling...}, and simulate the effect on the UI. After sufficient interactions with the UI, they evaluated planning efficiency (Eff), output data validity (Val), system operability (Ops), function expandability (Ext), and recommendation to another user (Rec) on a 10-point scale from No to Yes within 20 minutes. We also collected responses about the selection rationale through a questionnaire.




Results.
All the results are summarized in Table <ref>.
The correlation coefficient for evaluations from senior participants was 0.6220, while it was 0.3811 for junior participants. This indicates that the simulator's assessments could be more similar to those of senior marketers. In the questionnaire, the senior participants expressed difficulty in assessing campaigns within a restricted time frame, as they often relied on their intuition rather than conducting a thorough analysis. This reliance on intuition can explain why a higher correlation exists between the seniors and the simulator compared to the juniors. The overall evaluations on the usefulness of our system were also high, pointing to potential contributions to marketing work. Interestingly, junior marketers gave higher ratings to the system than the seniors. These results indicate the potential of our simulator to be a training tool.


 §.§ Limitations and Possible Extensions

While our system worked reasonably well, we found several failure cases through the user study. For example, the assessments for the campaign Rank up to silver member with an additional $30 purchase this month! were extremely high with 4,3,3,4,4 across all participants, while the simulator outputs a low conversion rate uplift with or without the campaign event.
The observed difficulty might stem from the absence of a real membership system in the store and in the data. This situation is outside the scope of the LLM's common-sense reasoning. Potential extensions to mitigate this problem can be fine-tuning of LLMs to obtain domain-specific contexts.  Moreover, one senior participant rated 4/10 on the validity of developed UI, due to unclear rationale for the conversion rate presented by the simulator. Enhancing interpretability by illustrating variations in statistical trends or extracting significant user behavior in a control and treatment group would be an interesting future direction.









§ CONCLUSION



In this work, we propose a novel framework, CXSimulator, designed for offline assessments of untested campaigns through user behavior simulations based on LLM embeddings. 
Our experimental results demonstrate the effectiveness of our transition probability prediction based on LLM embeddings compared to existing link prediction methods. We also confirm a correlation between the assessments from the CXSimulator and those from marketers, thereby indicating the practicability of the proposed framework in real-world marketing operations. In future research, we aim to examine the effectiveness of our approach within the domain and format constraints of private data from actual retail stores.




ACM-Reference-Format





 Paper Title: Barlow Twins Deep Neural Network for Advanced 1D Drug-Target Interaction Prediction.



Information flow-enhanced precision in collisional quantum thermometry   
    Mauro Paternostro
    August 5, 2024
=========================================================================



§ ABSTRACT 
Accurate prediction of DTI is critical for advancing drug discovery.
By reducing time and cost, ML and DL can accelerate this discovery process.
Our approach utilises the powerful Barlow Twins architecture for feature-extraction while considering the structure of the target protein, achieving state-of-the-art predictive performance against multiple established benchmarks.
The use of GBM as the underlying predictor ensures fast and efficient predictions without the need for large computational resources.
In addition, we further benchmarked new baselines against existing methods.
Together, these innovations improve the efficiency and effectiveness of DTI predictions, providing robust tools for accelerating drug development and deepening the understanding of molecular interactions.






§ INTRODUCTION


Studying DTI is crucial for understanding the biochemical mechanisms that govern how molecules interact with proteins.<cit.>
Key challenges in drug discovery are the identification of proteins that can be used as targets for the treatment of diseases.<cit.>
To achieve the desired therapeutic effects, the discovery of molecules that interact with and activate or inhibit target proteins is essential.<cit.>

Recent advances in computational methods have transformed the drug discovery landscape, providing robust tools for cost-effective exploration of the chemical space.
These in silico approaches facilitate the prediction and analysis of DTI, aiding in the identification of potential drug candidates and their corresponding protein targets.<cit.>
The use of computational techniques allows researchers to gain a comprehensive understanding of the molecular mechanisms underlying DTI, thereby accelerating the drug discovery process and minimising reliance on traditional, resource-intensive experimental methods.<cit.>
Different methods have been used to understand how drugs interact with target proteins. 
These methods are grouped into three main categories: structure-agnostic, structure-based and complex-based.

Structure-agnostic approaches use 1D inputs like molecule SMILES and protein amino acid sequences, or 2D inputs like graphs and predicted contact maps.<cit.>
These methods are cost-effective and and sufficiently accurate compared to experimental or in silico structure prediction,<cit.> as they are independent of the protein's structure when predicting effects. 

Structure-based approaches require 3D protein structures and 1D or 2D molecular inputs.

3D structures are usually derived from experimental data, although computational predictions are increasingly employed.<cit.>
These methods have great potential but can be unreliable. 
They depend on accurate 3D protein structures and may be limited in their ability to generalise beyond experimentally observed DTI.<cit.>
Due to the complexity of the experimental setup, 3D protein structures can be difficult to obtain. 

In addition, models often overlook the fact that proteins are not rigid structures, but are generally in motion, e.g., ligand binding induces a conformational change.<cit.>

Finally, complex-based approaches require protein-ligand co-complex structures, which additionally require 3D information, as well as protein interaction information about the ligand.<cit.>
For this reason, complex-based approaches can provide a more detailed insight into the interactions, but they are by far the most difficult to obtain data for.

Considering these different approaches, we designed as a fully data-driven, sequence-based approach that relies on SMILES and amino acid sequences as the most accessible data, avoiding costly and time-consuming experimental data such as crystal structures.
Additionally, we use a specialised bilingual PLM to embed the 1D amino acid sequence, which uses a 3D-alignment method that results in a structure-sequence representation.<cit.>
This approach makes input data structure-agnostic, yet benefits from structure-sequence PLM embeddings.
Unlike most other methods, we have developed a system that uses a hybrid best of both worlds ML and DL approach to improve DTI prediction performance in low data regimes where training data is limited.<cit.>
We have found that DL architectures such as Barlow Twins<cit.> are excellent at learning features<cit.> that can then be used for GBM training to achieve state-of-the-art performance, as the size of datasets is usually too small to reliably train a DL model that will perform competitively.





§ RESULTS AND DISCUSSION




  
design
We propose a novel method for predicting DTI using SMILES notations, primary amino acid sequences and annotated interaction properties.
relies on a three key components, visualised in <ref>:


    
  * Firstly, the input needs to be vectorised. This is achieved by converting SMILES to an ECFP.
    The amino acid sequences are processed by a PLM that uses both modalities, combining 1D protein sequences and 3D protein structure.<cit.>
    
  * Secondly, we teach the SSL based Barlow Twins model interaction of molecule and protein without considering labels.<cit.>
    The objective function implements invariance of both representation of one interaction while ensuring non-redundancy of the features.<cit.>
    
  * Finally, takes a combination of embeddings generated by the encoders from the Barlow Twins DL model and uses them as features to train a GBM based on the interaction annotations.<cit.>
    This approach exploits two key strengths: it uses DL to refine representations, and it leverages the power of ML in scenarios with limited data. 
    This is particularly relevant for DTI datasets, where only around 50000 annotated pairs are publicly available.<cit.>





  
Benchmark selection
We select a comprehensive set of literature-based benchmarks to evaluate the performance of against several leading methods.
The benchmarks considered in this study are derived from several key sources. 
These sources include biomedical networks,<cit.> the US patent database,<cit.> and data detailing the interactions of 72 kinase inhibitors with 442 kinases, representing over 80 of the human catalytic protein kinome.<cit.> 
These datasets provide DTI as pairs of molecules and amino acid sequences, each coupled to an interaction annotation.

To ensure a fair comparison, is retrained across all benchmarks. 
Finally, we evaluate performance in a binary classification setting:


    
  * We compare with a total of seven established models: the model by , MolTrans,<cit.> DLM-DTI,<cit.> ConPLex,<cit.> DrugBAN,<cit.> PSICHIC,<cit.> and STAMP-DTI.<cit.>
    Overall, various structure-agnostic, structure-based and complex-based methods have demonstrated state-of-the-art performance in benchmarks.
    
  * This comparison is performed on a total of four datasets with twelve literature-proposed splits: 4 × BioSNAP,<cit.> 4 × BindingDB,<cit.> 1 × DAVIS<cit.> and 3 × Human.<cit.>
    Our aim is to investigate the behaviour of different methods in diverse splitting scenarios. 
    These splits help us to assess how well models generalise under challenging evaluation conditions, for example where either the drug or the target has not been seen before, thus providing insight into their real-world applicability.
    
  * In addition, we investigate the addition of a more rigorous model baseline. The GBM XGBoost is known to be one of the best models, e.g. in QSAR tasks, often outperforming DL-based approaches.<cit.>




  
shows state-of-the-art performance in predicting DTI
We evaluate the performance of in binary classification on four datasets using different splitting procedures. 
Overall, significantly outperforms all other models in <ref>.
Looking at BioSNAP, we improve >5 over the leading method DLM-DTI in terms of PR_AUC.
Furthermore, as shown in <ref> again outperforms the PSICHIC method with a >5 PR_AUC improvement independent of the split.



When switching to BindingDB, significantly outperforms DLM-DTI in terms of PR_AUC with a >12 improvement (<ref>).
Investigating the BindingDB splits shows that outperforms all existing methods when looking at unseen ligands, almost matches the performance of DrugBAN in the random setting and becomes third best in the unseen protein split (<ref>).
Overall, is performs best in two out of four splits in this benchmark.

once again outperforms all of the established approaches when looking at the DAVIS benchmark, with a >18 improvement over the leading ConPLex model in terms of PR_AUC (<ref>).

Lastly, we evaluate the performance on the Human benchmark. 
shows the best performance when looking at the unseen protein split as well as the random split (<Ref>).
PSICHIC comes first in the unseen ligand setting, when looking at ROC_AUC, while DrugBAN is best in PR_AUC.
In summary, outperforms all other models in two out of three splits.



To look at why outperforms  other methods in various benchmarks, we looked into the architecture and its components in an ablation study.





  
Unravelling the performance contributions of the architecture
To investigate the impact of each element of the architecture, we remove them one at a time.
We do this across all baselines and splits with the following ablations:


    
  * We remove the hyperparameter optimisation step of the classifier.  
    
  * From the first removal, we replace the Barlow Twins architecture entirely and instead concatenate ECFP and PLM embeddings for training. We keep the hyperparameter optimisation procedure as in .
    
  * Finally, we remove the hyperparameter optimisation procedure from the previous ablation, analogous to the first modification. 


We observe a significant decline in performance, as illustrated in <ref>a and <ref> for the initial ablation, emphasising the crucial role of hyperparameter optimisation for achieving optimal model performance. 

The second ablation also indicates a significant reduction in performance. 
This is likely attributed to the DL architecture based on the SSL Barlow Twins model, which effectively learns embeddings to describe DTI. 
The Barlow Twins objective promotes orthogonality between drug and target modalities while ensuring the non-redundancy of both, thus preventing informational collapse. 
As a result, this leads to an overall state-of-the-art predictive performance.

The final ablation shows a further decline in performance, consistent with the results of the initial ablation experiment. 

In summary, the sustained reduction in performance of our ablation experiments demonstrates that each component of our pipeline is needed to maximise performance. 
This architecture integrates the best of both worlds: DL and GBM to enhance predictive performance.
Compared to other pure ML- or DL-based approaches, we can demonstrate a performance boost.
In particular, the use of the state-of-the-art PLM<cit.> could offer an advantage over other methods.
Other PLM variants are ProtTrans T5<cit.> in ConPLex<cit.> and ProtBERT proposed by  also used in DLM-DTI.<cit.>
The structural awareness of added by the inclusion of 3D-alignment in ProstT5<cit.> hints towards better generalisation capabilities.



    Choosing baseline models

Selecting an appropriate baseline model is critical to effectively comparing different ML and DL techniques. Robust baselines allow meaningful comparisons to be made and highlight the improvements made by new methods. 
Without appropriate baselines, it becomes difficult to determine whether new approaches are truly advancing the field.

Current leading DTI models predominantly use DL methods and are often evaluated against simple baseline models such as logistic regression, Ridge or DNN classifiers.<cit.> 
To improve the benchmarking process, we propose to add GBM as a baseline for DTI benchmarking purposes, as shown in the final ablation configuration. 
GBM such as XGBoost have demonstrated broad adaptability, e.g. in QSAR modelling, offering strong predictive performance and fast training times, particularly in scenarios with limited data availability, such as DTI prediction.

We compare the overall model performance across all datasets in <ref>b and <ref>.
Here, the performance of XGBoost trained on ECFP and PLM embeddings is highlighted as it shows competitive performance across all methods and datasets. 



§ CONCLUSIONS


Our proposed method, , integrates sequence information with the Barlow Twins SSL architecture and GBM models, representing a powerful fusion of ML and DL techniques.

This approach demonstrates state-of-the-art DTI prediction capabilities, validated across multiple benchmarks and data splits. 
Notably, our method outperforms existing literature benchmarks in ten out of twelve datasets evaluated.

To elucidate the efficacy of , we conducted an ablation study to investigate the contribution of its core components and their impact on performance. 
In addition, we re-evaluated the choice of baselines in numerous publications and advocate the inclusion of GBM baselines.

Given its exceptional performance, we are confident that can significantly accelerate the drug discovery process and offer significant time and cost savings through the use of virtual screening campaigns.



§ METHODS




 §.§ Datasets


To evaluate the performance of , three established benchmarks are used.
They all provide fixed splits for training, evaluation and testing.
In some publications the training and evaluation is merged to improve predictive performance.
To endure comparability, this was not done in this work.
All metrics listed from other publications are also listed where only the training set is used.

In addition,  first proposed splits for large DTI datasets, BioSNAP,<cit.> BindingDB<cit.> and DAVIS.<cit.>

The addition of a variety of splits with an additional benchmark Human<cit.> (based on DAVIS) are proposed by , we evaluate these separately.<cit.>

For all datasets, to reduce bias and improve model performance, the SMILES are cleaned using the Python ChEMBL curation pipeline.<cit.>
All duplicate and erroneous molecule and protein information that could not be parsed is removed.
Training is performed on the predefined training splits.



 §.§ Representations




  
Molecular information
The SMILES are converted into ECFP using RDKit.<cit.>
We used them with 1024 and a radius of 2.



  
Amino acid sequence information
The amino acid sequences are converted into vectors, by using the PLM ProstT5.<cit.>



 §.§ Barlow Twins model configuration


The proposed method is based on the Barlow Twins<cit.> network architecture, which employs one encoder for each modality and a unified projector.
The encoders and projector are MLP based.
The loss function is adapted from the original Barlow Twins publication and enforces cross-correlation between the projections of the modalities.<cit.>

The architecture is coded in Python using PyTorch.<cit.>



  
Pre-training Barlow Twins 
Here we pre-train the Barlow Twins architecture on our joint DTI dataset, based on BioSNAP, BindingDB, DAVIS and DrugBank,<cit.> removing duplicates and without labels to teach DTI. 
Early stopping is implemented to avoid overfitting, which is carried out using a 15 validation split.



    Hyperparameter optimisation


Manual hyperparameter optimisation is performed, shown in <ref>.





  
Feature-extractor
When performing feature-extraction, we use the pre-trained model.
For training and prediction, we extract the embeddings after the encoders for each modality and concatenate them.
Finally, a GBM, XGBoost<cit.> Python implementation, is trained on the embeddings in combination with the labels for each training sets respectively.



    Hyperparameter optimisation

If a benchmark provides a dedicated validation set, this was used for Optuna<cit.> hyperparameter optimisation.
The optimisation was carried out for 100 trials with the parameters shown in <ref>.






 §.§ Baseline model configuration


As a baseline, we have selected a GBM.
Similar to our feature-extraction implementation, for all features we concatenate both ECFP and PLM embeddings.
Finally, a GBM, XGBoost Python implementation, is trained on the ECFP and PLM embedding concatenation in combination with the labels for each training set, respectively.


The authors thank Merck KGaA Darmstadt for their generous support with the Merck Future Insight Prize 2020. 
This project is also cofunded by the European Union (ERC, breakingBAC, 101096911).
M.G.S. thanks Joshua Hesse and Aleksandra Daniluk for their valuable input and helpful feedback.













§ ADDITIONAL RESULTS AND DISCUSSION




  
Statistical testing
We focus on PR_AUC as our metric because it is an established performance indicator in unbalanced scenarios. 
Secondly, it shows a more pronounced separation between different methods, as most methods show very high values of ROC_AUC. 

We apply the two-sided Welch's t-test,<cit.> with Benjamini-Hochberg<cit.> multiple test correction.
This is done for all methods for which the required performance information exists in the published literature.

In <ref>, our primary focus is on the overall change in performance.
We therefore make comparisons across all datasets collectively rather than individually. 
Detailed individual comparisons are provided in <ref>.















 Paper Title: Differentially Private Block-wise Gradient Shuffle for Deep Learning.


810Comparative analysis of structural, elastic, electronic, phonon, thermal and optical properties of two Na6Ge2Se6 phases from first principles calculations
    Aleksandr Chernatynskiy
    August 5, 2024
==========================================================================================================================================================



§ ABSTRACT 
Traditional Differentially Private Stochastic Gradient Descent (DP-SGD) introduces statistical noise on top of gradients drawn from a Gaussian distribution to ensure privacy. This paper introduces the novel Differentially Private Block-wise Gradient Shuffle (DP-BloGS) algorithm for deep learning. BloGS builds off of existing private deep learning literature, but makes a definitive shift by taking a probabilistic approach to gradient noise introduction through shuffling modeled after information theoretic privacy analyses. The theoretical results presented in this paper show that the combination of shuffling, parameter-specific block size selection, batch layer clipping, and gradient accumulation allows DP-BloGS to achieve training times close to that of non-private training while maintaining similar privacy and utility guarantees to DP-SGD. DP-BloGS is found to be significantly more resistant to data extraction attempts than DP-SGD. The theoretical results are validated by the experimental findings.




§ INTRODUCTION


Core to the theoretical underpinnings of DP-BloGS is a fact borne of modern necessity: deep learning libraries like PyTorch expose structured access to gradients through parameter groupings. The block-wise shuffle approach offers several nascent advantages compared to traditional DP-SGD in this regard. Specifically, by accounting for the parameter group and gradient dimensions in the sensitivity analysis, DP-BloGS is efficiently tailored to the modern architecture of deep learning models. The probabilistic noise approach through shuffling gradient components plays well with gradient accumulation, as shown through the perplexity metric in the utility experiments. Privacy guarantees are verified through the analysis of data extraction and membership inference vulnerability compared to traditional DP-SGD. These results are promising for machine learning practitioners in that they show DP-BloGS offers similar or better privacy guarantees, competitive performance, and faster runtimes than DP-SGD.

This paper introduces several important contributions to the differential privacy literature, particularly in the context of privacy-preserving deep learning:



  
Novel Algorithm The paper introduces Differentially Private Block-wise Gradient Shuffle (DP-BloGS), a new algorithm for privacy-preserving deep learning. This algorithm takes a probabilistic approach to gradient noise introduction through shuffling modeled after information theoretic privacy analyses.



  
Theoretical Foundations The paper provides a comprehensive theoretical analysis of DP-BloGS, including proofs of its privacy guarantees and utility bounds. This includes theorems on privacy composition, convergence analysis, and information-theoretic bounds.



  
Parameter-wise Privacy The paper introduces a parameter-wise approach to differential privacy, allowing for different privacy levels for different parts of the model. This is formalized through theorems on parameter-wise privacy loss and composition.



  
Improved Efficiency DP-BloGS is shown to achieve training times close to that of non-private training while maintaining similar privacy and utility guarantees to DP-SGD. This addresses a key challenge in practical deployment of differentially private machine learning.



  
Enhanced Privacy-Utility Trade-off The paper demonstrates that DP-BloGS can achieve better privacy-utility trade-offs compared to traditional DP-SGD in many scenarios, particularly in terms of resistance to data extraction attempts.



  
Scalability to Large Models The effectiveness of DP-BloGS is demonstrated on models with up to 1.1 billion parameters, showing its applicability to modern large language models.



  
Optimal Parameter Selection The paper provides methods for optimally selecting key parameters of the DP-BloGS algorithm, such as block sizes and clipping thresholds, to balance privacy and utility.



  
Empirical Evaluation The paper includes experiments comparing DP-BloGS to DP-SGD across multiple model architectures and privacy levels, evaluating metrics such as perplexity, membership inference attack resistance, and data extraction rates.



  
Information-Theoretic Analysis The paper provides information-theoretic bounds on the privacy guarantees of DP-BloGS, connecting the algorithm to fundamental concepts in information theory.



  
Privacy Accounting The paper develops privacy accounting methods specific to DP-BloGS, allowing for precise tracking of privacy loss over the course of training.

The rest of this paper is structured as follows. Relevant work is briefly introduced and its importance stated in a related work section. The methodology section covers the experimental approach and the DP-BloGS algorithm in detail. Results are presented, with full tables and all plots saved for the appendix. The mathematical foundations for DP-BloGS are then laid out, building on prominent literature in differential privacy. The math section aims to provide each necessary mathematical tool in the order it is needed, proving DP-BloGS' privacy-enhancing qualities in rigorous detail. The paper concludes by summarizing key findings, commenting on limitations, and laying out future work.





§ RELATED WORK


This work builds on prior Differential Privacy literature such as <cit.>. For implementation, notably, batched layer clipping<cit.> is a primary component of DP-BloGS' speed boost. DP-BloGS is compared to DP-SGD by implementing the code available on the AWS Fast DP repository<cit.>.

<cit.> also expand on shuffling models for differential privacy. Though this paper is categorically different in its approach and its privacy guarantees, it does draw explicit inspiration from the work done by Cheu et al.



§ ALGORITHM


The algorithm has 3 primary classes. The Trainer, the Generator, and the Accountant. The Trainer handles everything related to training, and is an extended version of the SFTTrainer from Hugging Face trl. The Generator is responsible for ingesting the gradients and shuffling during the post accumulation step before the model's weights are updated with the gradients. The Accountant is responsible for optimizing the block sizes and keeping track of privacy spend across the training process.



 §.§ Initializing DPShuffleGenerator


The DPShuffleGenerator is initialized with several key parameters: model (the neural network model being trained), target_epsilon (the desired privacy budget), delta (the probability of privacy failure), steps (the number of training steps), clip_value (the maximum L2 norm for gradient clipping), and batch_size (the number of examples per training batch).

During initialization, the generator creates a DPShufflePrivacyAccountant with these parameters. The accountant then performs the following optimization process:



  §.§.§ Privacy Accountant Optimization


The accountant computes the dimensions of all trainable parameters in the model. It then employs a nested binary search strategy to find the optimal block sizes for each parameter group:

Outer binary search: This search finds the optimal target epsilon per group (epsilon_i) that will result in the overall target epsilon when summed across all groups and steps.

Inner binary search: For each parameter group, this search determines the largest block size that satisfies the current target epsilon_i.

For each parameter group with dimension d_i and a given block size, the accountant computes two epsilon values:


    ϵ_1 = 2 ·log(1 + d_i · (e^2C/√(d_i) - 1))


    ϵ_2 = 2 ·log(1 + (block_size/d_i) · (e^2C√(block_size/d_i) - 1))


Where C is the clip_value. The smaller of these two values is chosen as the epsilon for this group and block size.

The total privacy spent across all groups and steps is computed using:


    ϵ_total =    √(2 · steps ·log(1/δ))·ϵ_total_per_step
        + steps ·ϵ_total_per_step· (e^ϵ_total_per_step - 1)


Where ϵ_total_per_step is the sum of epsilons for all parameter groups in a single step.

This process iteratively refines the estimate of the optimal target epsilon per group until the total privacy is as close as possible to the overall target epsilon.

The generator stores these optimal block sizes and initializes the spent privacy budget (epsilon_spent) to 0.





 §.§ Training Process


When training begins and gradients have been accumulated, the generate function is called with the accumulated gradients. This function performs several key steps:



  §.§.§ Gradient Processing


For each gradient and its corresponding optimal block size, the gradient is clipped to the specified clip_value if necessary, and then shuffled using the optimal block size. These processed gradients are collected into a list called private_grads.



  §.§.§ Gradient Shuffling


The shuffling process involves flattening the gradient into a 1D tensor, padding it if necessary to ensure it can be evenly divided into blocks, reshaping it into blocks of the specified size, randomly shuffling the blocks, flattening and trimming the shuffled blocks to the original gradient size, and finally reshaping the result to match the original gradient shape.



  §.§.§ Privacy Accounting


After processing all gradients, the accountant computes the total privacy spent (epsilon) for this step.



§ METHODOLOGY


To aid in visualizing DP-BloGS, this paper presents two primary figures containing three plots in total. Figure 1 shows a simulation of parameter group gradients converted to 2D heatmaps for DP-BloGS at values of epsilon 1 and 100. Figure 2 shows layer gradients unprocessed for visual comparison.

Experiments were performed on a single A100 GPU. The dataset used for training was the first 1,000 records of Tiny Orca. The dataset used for testing was the second 1,000 records of Tiny Orca. The models examined were sourced through the Hugging Face model hub. Specifically facebook/opt-350m, bigscience/bloom-560m, openai-community/gpt2, TinyLlama/TinyLlama-1.1B-step-50K-105b, and google-bert/bert-base-uncased.





Each experiment held the majority of variables constant. The hyperparameters were kept as follows: batch size of 5, gradient accumulation steps of 10, learning rate of 2e-4, max gradient norm of 1.0, training epochs of 5, AdamW 32 bit optimizer, and a constant learning rate. Experiments were performed in full 32 bit precision.



 §.§ Evaluation Process




  §.§.§ Membership Inference Attack (MIA)


The membership inference attack was conducted using an approach that combines multiple machine learning models and advanced feature engineering.

Feature extraction involved extracting features from both training and test datasets using the  function. These features included model confidence, entropy, perplexity, data augmentation responses, memorization metrics, generalization indicators, contrastive learning features, out-of-distribution detection scores, and per-layer gradient norms.

Data preprocessing steps included imputation to handle missing values, standardization of features using , feature selection using mutual information, creation of polynomial features to capture non-linear relationships, and application of SMOTE for class balancing.

For model training and selection, multiple classifiers including Random Forest, Logistic Regression, and Neural Network were trained.  with  cross-validation was used for hyperparameter tuning. A stacking ensemble combining the best models was implemented, and the best performing model was selected based on ROC AUC score.

Model calibration and thresholding involved calibrating the best model's probabilities and determining an optimal classification threshold to balance precision and recall.

Evaluation metrics included ROC AUC, precision-recall AUC, and Brier score, which are standard metrics for evaluating the effectiveness of membership inference attacks <cit.>. Accuracy, precision, recall, and F1-score were computed for both standard (0.5) and optimal thresholds. Error analysis was performed to understand misclassifications.

Visualization included plots of the ROC curve, learning curves, and feature importance.



  §.§.§ Data Extraction Risk Assessment


The data extraction risk was evaluated using a simulated extraction attack. The attack simulation involved 1000 extraction attempts. For each attempt, a randomly selected training example was used, with a substring of length 10 as a prompt to generate a completion of length 30 using the model.

Extraction detection involved checking if the generated text matched any part of the training data. A match was considered successful if at least 20 consecutive characters matched.

Analysis of the extraction results included calculating the extraction rate (percentage of successful extractions), computing the average length of successfully extracted sequences, and identifying the top 10 most substantial extractions.



  §.§.§ Perplexity Evaluation


Perplexity was evaluated to assess the model's predictive performance. The evaluation process involved iterating through the dataset batch by batch. For each batch, the input text was tokenized, target labels were created by shifting input tokens, model logits were computed, and cross-entropy loss was calculated, ignoring padding tokens.

The perplexity calculation involved accumulating total loss and total number of tokens across all batches, computing average loss per token, and calculating perplexity as the exponential of the average loss.

Interpretation of perplexity results considered that lower perplexity indicates better predictive performance. Perplexity was compared across different model versions or privacy settings to assess impact on language modeling capability.

This comprehensive evaluation process provides a multi-faceted assessment of the model's privacy preservation, covering both the risk of membership inference and data extraction, while also measuring the model's overall language modeling performance through perplexity. The results from these evaluations can be used to analyze the trade-offs between privacy and utility in the trained models.





§ RESULTS




 §.§ DP-BloGS vs. DP-SGD


A normalized comparison of DP-BloGS and DP-SGD across multiple models reveals insights into their relative performance in terms of utility and privacy.





  §.§.§ Utility Performance


In terms of perplexity, our key utility metric, DP-BloGS demonstrates a slight advantage. The average normalized difference of -0.0226 favors DP-BloGS, which outperforms DP-SGD in 67.69% of cases. DP-BloGS also exhibits a higher normalized Area Under the Curve (AUC) of 0.7557 compared to 0.0977 for DP-SGD. However, it is important to note that this difference is not statistically significant (p-value: 0.5523), indicating that the utility performance of both methods is comparable.



  §.§.§ Privacy Performance


The privacy performance is assessed through two metrics: resistance to Membership Inference Attacks (MIA) and resistance to data extraction.

For MIA resistance, both methods perform similarly. The average normalized difference of 0.0013 slightly favors DP-SGD, but DP-BloGS still performs better in 50.77% of cases. The normalized AUC values are close (1484.56 for DP-BloGS vs 1493.59 for DP-SGD), and the difference is not statistically significant (p-value: 0.9094).

In terms of data extraction resistance, DP-BloGS significantly outperforms DP-SGD. The average normalized difference of -0.0514 favors DP-BloGS, which performs better in 69.23% of cases. This difference is statistically significant (p-value: 0.0024), indicating a consistent advantage for DP-BloGS in preventing data extraction across different models.





  §.§.§ Key Findings


The comparative analysis reveals that DP-BloGS offers comparable or slightly better utility than DP-SGD, similar resistance to membership inference attacks, and significantly better protection against data extraction. These findings suggest that DP-BloGS may be the preferable method, particularly in scenarios where strong protection against data extraction is a priority. The balanced performance across utility and privacy metrics positions DP-BloGS as a robust choice for privacy-preserving machine learning applications.





§ LIMITATIONS AND FUTURE WORK


This research was limited by budget and time. If more resources had been available, it would have been prudent to examine multiple datasets instead of training numerous model variants across a single dataset. Additional resources would have allocated for examining the interaction with even larger models, such as Llama 405b, Nemo, Falcon 180b, LoRA finetuning, and other methodologies.

Future research will refine parameter-wise budget allocation strategies, explore how DP-BloGS interacts with LoRA, apply DP-BloGS to different model architectures like convolutional neural networks, variational autoencoders, and diffusion models, and assess the parameter-specific factors that come into play when seeking to maximize utility while retaining privacy.



§ MATH


We begin by describing and showing preservation and maintenance of gradient properties under shuffling.



§ LEMMA 1


[Gradient Properties Under Shuffling]
Let g ∈ℝ^d be a gradient vector and π : {1, ..., d}→{1, ..., d} be a permutation function. Define the shuffled gradient g_π as g_π[i] = g[π(i)] for i ∈{1, ..., d}. Then, the following properties hold:

    
  * L2 Norm Preservation: g_π_2 = g_2
    
  * Mean Preservation: 1/d∑_i=1^d g_π[i] = 1/d∑_i=1^d g[i]
    
  * Variance Preservation: Var(g_π) = Var(g)




Let (Ω, ℱ, ℙ) be a probability space, where Ω is the sample space, ℱ is a σ-algebra on Ω, and ℙ is a probability measure.


    
  * L2 Norm Preservation:
    
       g_π_2^2 = ∑_i=1^d (g_π[i])^2     

         = ∑_i=1^d (g[π(i)])^2     

         = ∑_j=1^d (g[j])^2   (since π is a bijection)     

         = g_2^2    
    
    Taking the square root of both sides:
    
       g_π_2 = g_2    
    

    
  * Mean Preservation:
    
       1/d∑_i=1^d g_π[i] = 1/d∑_i=1^d g[π(i)]     

         = 1/d∑_j=1^d g[j]   (since π is a bijection)     

         = 1/d∑_i=1^d g[i]    
    

    
  * Variance Preservation:
    First, let's recall the definition of variance for a vector:
    
       Var(g) = 1/d∑_i=1^d (g[i] - μ)^2,   where  μ= 1/d∑_i=1^d g[i]    
    
    Now, let's prove that Var(g_π) = Var(g):
    
       Var(g_π) = 1/d∑_i=1^d (g_π[i] - μ_π)^2, 

         where  μ_π= 1/d∑_i=1^d g_π[i]     

       From (b), we know that  μ_π= μ.  Therefore:     

       Var(g_π) = 1/d∑_i=1^d (g_π[i] - μ)^2     

         = 1/d∑_i=1^d (g[π(i)] - μ)^2     

         = 1/d∑_j=1^d (g[j] - μ)^2   (since π is a bijection)     

         = Var(g)    
    


Thus, we have proven that shuffling preserves the L2 norm, mean, and variance of the gradient vector.




§ COROLLARY 1.1


[Preservation of Gradient Clipping]
Let clip_C(g) be a gradient clipping function that clips the L2 norm of g to a maximum of C. Then, for any permutation π:

    clip_C(g_π) = (clip_C(g))_π




Case 1: If g_2 ≤ C, then clip_C(g) = g. 

In this case, g_π_2 = g_2 ≤ C (from Lemma 1(a)). 

Therefore, clip_C(g_π) = g_π = (clip_C(g))_π.

Case 2: If g_2 > C, then clip_C(g) = (C/g_2) g. 

Let h = clip_C(g). Then:

    h_π[i] = h[π(i)] = (C/g_2) g[π(i)] = (C/g_2) g_π[i]

On the other hand:

    clip_C(g_π) = (C/g_π_2) g_π = (C/g_2) g_π   (from Lemma 1(a))

Therefore, clip_C(g_π) = h_π = (clip_C(g))_π.




§ DEFINITION 2


This definition builds on the foundations of differential privacy <cit.> and incorporates shuffling techniques <cit.>.

*definition2Definition 2

[DP-BloGS Mechanism]
Let (Ω, ℱ, ℙ) be a probability space, where Ω is the sample space, ℱ is a σ-algebra on Ω, and ℙ is a probability measure.

Let 𝒳 be the data space and 𝒟⊆𝒳^n be the set of all possible datasets of size n.

Let ℝ^d be the d-dimensional real vector space representing the gradient space.

Define the gradient function g : 𝒟→ℝ^d that maps a dataset to its gradient.

Let 𝒮^d be the symmetric group on d elements, i.e., the set of all permutations of {1, …, d}.

Define a random variable Π : Ω→𝒮^d that selects a permutation uniformly at random from 𝒮^d.

The DP-BloGS mechanism M : 𝒟×Ω→ℝ^d is defined as:

    M(D, ω) = g_π(D)

where π = Π(ω) and g_π(D)[i] = g(D)[π(i)] for i ∈{1, …, d}.

Properties of the DP-BloGS Mechanism:

1. Uniform Selection: For any permutation σ∈𝒮^d,

    ℙ(Π = σ) = 1/|𝒮^d| = 1/d!


2. Invertibility: Given M(D, ω) and π, one can recover g(D) by applying π^-1.

3. Output Space: The output space of M is identical to the gradient space ℝ^d.




§ LEMMA 2.1

*lemma2.1Lemma 2.1

[Conditional Probability of Output]
For any dataset D ∈𝒟 and any vector v ∈ℝ^d,

   ℙ(M(D, ω) = v |g(D) = u) =    

    
    1/d!     if  ∃σ∈𝒮^d  such that  v[i] = u[σ(i)] ∀i ∈{1, …, d} 

    0     otherwise
   


Proof:

Let E = {ω∈Ω : M(D, ω) = v  and  g(D) = u}.

Case 1: ∃σ∈𝒮^d such that v[i] = u[σ(i)] ∀ i ∈{1, …, d}

In this case, E = {ω∈Ω : Π(ω) = σ^-1}.

    ℙ(E) = ℙ(Π = σ^-1) = 1/d!


Case 2: ∄σ∈𝒮^d such that v[i] = u[σ(i)] ∀ i ∈{1, …, d}

In this case, E = ∅.

    ℙ(E) = 0


Therefore,

    ℙ(M(D, ω) = v | g(D) = u) = ℙ(E) / ℙ(g(D) = u) = ℙ(E)


This completes the proof.




§ LEMMA 2.2

*lemma2.2Lemma 2.2

[Preservation of Gradient Properties]
For any dataset D ∈𝒟 and any ω∈Ω,


    
  * M(D, ω)_2 = g(D)_2
    
  * 1/d∑_i=1^d M(D, ω)[i] = 1/d∑_i=1^d g(D)[i]
    
  * Var(M(D, ω)) = Var(g(D))


Proof:
This follows directly from Lemma 1, as M(D, ω) is a shuffled version of g(D).




§ THEOREM 2.3


*theorem2.3Theorem 2.3

[Unbiasedness of DP-BloGS]
For any dataset D ∈𝒟,

    𝔼[M(D, ω)] = g(D)

where the expectation is taken over the random permutation Π.

Proof:

For any i ∈{1, …, d},

    𝔼[M(D, ω)[i]]    = ∑_k=1^d g(D)[k] ·ℙ(Π(i) = k) 
       = ∑_k=1^d g(D)[k] ·1/d
       = 1/d∑_k=1^d g(D)[k] 
       = g(D)[i]


The last equality holds because the sum of all elements divided by d equals each element (since the expectation preserves each element).

Therefore,

    𝔼[M(D, ω)] = g(D)





§ THEOREM 3


[Sensitivity of Shuffled Gradients]
Let D, D' ∈𝒟 be two adjacent datasets differing in at most one element. We analyze the sensitivity of the mechanism, a key concept in differential privacy <cit.>. Let M be the DP-BloGS mechanism as defined in Definition 2. Then, the L_1 and L_2 sensitivities of M are equal to the corresponding sensitivities of the original gradient function g:


    
  * Δ_1 M = Δ_1 g = sup_D,D'  adjacentg(D) - g(D')_1
    
  * Δ_2 M = Δ_2 g = sup_D,D'  adjacentg(D) - g(D')_2


where Δ_1 M and Δ_2 M are the L_1 and L_2 sensitivities of M, respectively, and Δ_1 g and Δ_2 g are the L_1 and L_2 sensitivities of g, respectively.

Proof:

Let (Ω, ℱ, ℙ) be our probability space as defined in Definition 2.

(a) L_1 Sensitivity:


    Δ_1 M = sup_D,D'  adjacent𝔼_ω[M(D,ω) - M(D',ω)_1]


For any ω∈Ω, let π = Π(ω) be the random permutation.


    M(D,ω) - M(D',ω)_1 
           = ∑_i=1^d |M(D,ω)[i] - M(D',ω)[i]| 
       = ∑_i=1^d |g(D)[π(i)] - g(D')[π(i)]| 
       = ∑_j=1^d |g(D)[j] - g(D')[j]| 
         (since π is a bijection)
       = g(D) - g(D')_1


This equality holds for all ω∈Ω, so:


    𝔼_ω[M(D,ω) - M(D',ω)_1] = g(D) - g(D')_1


Therefore,


    Δ_1 M = sup_D,D'  adjacentg(D) - g(D')_1 = Δ_1 g


(b) L_2 Sensitivity:


    Δ_2 M = sup_D,D'  adjacent√(𝔼_ω[M(D,ω) - M(D',ω)_2^2])


For any ω∈Ω, let π = Π(ω) be the random permutation.


    M(D,ω) - M(D',ω)_2^2 
           = ∑_i=1^d (M(D,ω)[i] - M(D',ω)[i])^2 
       = ∑_i=1^d (g(D)[π(i)] - g(D')[π(i)])^2 
       = ∑_j=1^d (g(D)[j] - g(D')[j])^2 
         (since π is a bijection)
       = g(D) - g(D')_2^2


This equality holds for all ω∈Ω, so:


    𝔼_ω[M(D,ω) - M(D',ω)_2^2] = g(D) - g(D')_2^2


Therefore,


    Δ_2 M = sup_D,D'  adjacentg(D) - g(D')_2 = Δ_2 g


This completes the proof.





§ COROLLARY 3.1


*corollary3.1Corollary 3.1

[Sensitivity Preservation under Clipping]
Let clip_C(g) be a gradient clipping function that clips the L_2 norm of g to a maximum of C. Gradient clipping, as introduced by <cit.>, is a crucial technique in our analysis. Define M_C as the composition of clipping and DP-BloGS:


    M_C(D, ω) = M(clip_C(g(D)), ω)


Then, the L_2 sensitivity of M_C is bounded by 2C:


    Δ_2 M_C ≤ 2C


Proof:

For any adjacent datasets D and D':


    M_C(D, ω) - M_C(D', ω)_2 
       = M(clip_C(g(D)), ω) - M(clip_C(g(D')), ω)_2 
       = clip_C(g(D)) - clip_C(g(D'))_2   (from Theorem 3)


Now, for any vectors u and v:


    clip_C(u) - clip_C(v)_2 
           ≤clip_C(u) - clip_C(0)_2 
          + clip_C(0) - clip_C(v)_2 
       ≤ C + C 
       = 2C


Therefore,


    M_C(D, ω) - M_C(D', ω)_2 ≤ 2C


This bound holds for all ω∈Ω, so:


    Δ_2 M_C 
           = sup_D,D'  adjacent√(𝔼_ω[M_C(D, ω) - M_C(D', ω)_2^2])
       ≤sup_D,D'  adjacent√(𝔼_ω[(2C)^2])
       = 2C


This completes the proof.




§ LEMMA 3.2


*lemma3.2Lemma 3.2

[Expected Squared L_2 Norm Difference]
For any adjacent datasets D and D', the expected squared L_2 norm difference between M(D, ω) and M(D', ω) is equal to the squared L_2 norm difference between g(D) and g(D'):


    𝔼_ω[M(D, ω) - M(D', ω)_2^2] = g(D) - g(D')_2^2


Proof:


    𝔼_ω[M(D, ω) - M(D', ω)_2^2] 
       = 𝔼_ω[∑_i=1^d (M(D, ω)[i] - M(D', ω)[i])^2] 
       = ∑_i=1^d 𝔼_ω[(M(D, ω)[i] - M(D', ω)[i])^2]


For each i:


    𝔼_ω[(M(D, ω)[i] - M(D', ω)[i])^2] 
       = 𝔼_ω[(g(D)[Π(ω)(i)] - g(D')[Π(ω)(i)])^2] 
       = 1/d∑_j=1^d (g(D)[j] - g(D')[j])^2


Therefore,


    𝔼_ω[M(D, ω) - M(D', ω)_2^2] 
       = ∑_i=1^d (1/d∑_j=1^d (g(D)[j] - g(D')[j])^2) 
       = ∑_j=1^d (g(D)[j] - g(D')[j])^2 
       = g(D) - g(D')_2^2


This completes the proof.




§ LEMMA 4


*lemma4Lemma 4

[Reconstruction Probability Bound]
Let D ∈𝒟 be a dataset, g(D) ∈ℝ^d be its gradient, and M be the DP-BloGS mechanism as defined in Definition 2. For any ω∈Ω, let M(D, ω) be the shuffled gradient. Then, the probability of an adversary correctly guessing g(D) given M(D, ω) is bounded by:


    ℙ(Adversary correctly guesses  g(D) | M(D, ω)) ≤1/d!


where d is the dimension of the gradient vector.

Proof:

Let (Ω, ℱ, ℙ) be our probability space as defined in Definition 2.

Let A : ℝ^d →ℝ^d be any adversarial reconstruction algorithm that attempts to recover g(D) from M(D, ω).

Define the event of successful reconstruction:
E = {ω∈Ω : A(M(D, ω)) = g(D)}

We want to prove that ℙ(E) ≤1/d!.

Step 1: Analyze the structure of E

For the adversary to correctly guess g(D), they must correctly identify the permutation π = Π(ω) used in the shuffling process.

E = {ω∈Ω : A(M(D, ω)) = g(D)} = {ω∈Ω : A(g_π(D)) = g(D),  where π = Π(ω)}

Step 2: Relate E to the set of permutations

Let S_A ⊆𝒮^d be the set of permutations for which A correctly reconstructs g(D):
S_A = {σ∈𝒮^d : A(g_σ(D)) = g(D)}

Then, E = {ω∈Ω : Π(ω) ∈ S_A}

Step 3: Use the properties of uniform distribution over permutations

From Definition 2, we know that Π is uniformly distributed over 𝒮^d. Therefore:


    ℙ(E) = ℙ(Π∈ S_A) = |S_A|/|𝒮^d| = |S_A|/d!


Step 4: Bound the size of S_A

The key observation is that for A to be correct for multiple permutations, g(D) must have symmetries. In the worst case (for privacy), g(D) is completely symmetric (all elements equal), and |S_A| = d!. In all other cases, |S_A| < d!.

Therefore, |S_A| ≤ d!

Step 5: Conclude the proof


    ℙ(E) = |S_A|/d!≤d!/d! = 1


Thus,


    ℙ(Adversary correctly guesses  g(D) | M(D, ω)) = ℙ(E) ≤1/d!


This completes the proof.




§ COROLLARY 4.1


*corollary4.1Corollary 4.1

[Expected Reconstruction Error]
Let D ∈𝒟 be a dataset, g(D) ∈ℝ^d be its gradient, and M be the DP-BloGS mechanism. For any reconstruction algorithm A, the expected L_2 reconstruction error is bounded below by:


    𝔼[A(M(D, ω)) - g(D)_2^2] ≥(1 - 1/d!) ·min_i ≠ j (g(D)[i] - g(D)[j])^2


where the expectation is taken over the random permutation in M.

Proof:

Let (Ω, ℱ, ℙ) be our probability space.

Define the indicator random variable:
I = 1 if A(M(D, ω)) = g(D), and 0 otherwise.

From Lemma 4, we know that ℙ(I = 1) ≤1/d!.

Now, let's consider the expected reconstruction error:


    𝔼[A(M(D, ω)) - g(D)_2^2] 
       = 𝔼[A(M(D, ω)) - g(D)_2^2 | I = 1] ·ℙ(I = 1) 
          + 𝔼[A(M(D, ω)) - g(D)_2^2 | I = 0] ·ℙ(I = 0)


When I = 1, the reconstruction error is 0. When I = 0, the reconstruction error is at least the squared difference between the two closest distinct elements of g(D). Let's call this minimum difference δ:


    δ^2 = min_i ≠ j (g(D)[i] - g(D)[j])^2


Therefore:


    𝔼[A(M(D, ω)) - g(D)_2^2] 
           ≥ 0 ·ℙ(I = 1) + δ^2 ·ℙ(I = 0) 
       = δ^2 · (1 - ℙ(I = 1)) 
       ≥δ^2 ·(1 - 1/d!)


This completes the proof.




§ THEOREM 5


*theorem5Theorem 5

This privacy guarantee is comparable to those in differentially private deep learning <cit.>.

[Differential Privacy of Single-Step DP-BloGS]
Let M be the DP-BloGS mechanism as defined in Definition 2, applied to a gradient function g : 𝒟→ℝ^d with L_2 sensitivity Δ_2 g. Let C be the clipping threshold and B be the batch size. Then, for any δ > 0, the DP-BloGS mechanism M satisfies (ϵ, δ)-differential privacy, where:


    ϵ = min{ϵ_1, ϵ_2}



    ϵ_1 = 2 ln(1 + d (e^2C/√(d) - 1))



    ϵ_2 = 2 ln(1 + β/d(e^2C √(β/d) - 1))


where β is the block size used in the shuffling mechanism.

Proof:

Let (Ω, ℱ, ℙ) be our probability space as defined in Definition 2.

Let D, D' ∈𝒟 be two adjacent datasets differing in at most one element.

Step 1: Define the privacy loss random variable

Define the privacy loss random variable Z : Ω→ℝ as:


    Z(ω) = ln(ℙ(M(D, ω) = v)/ℙ(M(D', ω) = v))


where v = M(D, ω) is the output of the mechanism.

Step 2: Analyze the sensitivity of the clipped gradient

The L_2 sensitivity of the clipped gradient is bounded by 2C.

Step 3: Analyze the privacy loss for the full gradient

For the full gradient, we can bound the privacy loss as:


    |Z(ω)| ≤ln(1 + d (e^2C/√(d) - 1))


This bound comes from considering the worst-case scenario where all d components of the gradient differ by the maximum amount.

Step 4: Analyze the privacy loss for block-wise shuffling

When considering block-wise shuffling with block size β, we can derive a tighter bound:


    |Z(ω)| ≤ln(1 + β/d(e^2C √(β/d) - 1))


This bound comes from considering the worst-case scenario within each block.

Step 5: Take the minimum of the two bounds

Since both bounds are valid, we can take the minimum of the two:


    |Z(ω)| ≤min{   ln(1 + d (e^2C/B √(d) - 1)), 
       ln(1 + β/d(e^2C √(β/d) - 1))
    }


Step 6: Account for the possibility of multiple queries

To account for the possibility that an adversary might try to exploit the randomness of the mechanism, we multiply our bound by 2:


    ϵ = 2 min{   ln(1 + d (e^2C/√(d) - 1)), 
       ln(1 + β/d(e^2C √(β/d) - 1))
    }


This gives us our final privacy guarantee.

This completes the proof.




§ COROLLARY 5.1


*corollary5.1Corollary 5.1

[Privacy-Utility Trade-off]
The privacy guarantee ϵ is inversely related to the batch size B and directly related to the clipping threshold C. This demonstrates a trade-off between privacy and utility, as larger batch sizes provide better privacy but may reduce utility, while larger clipping thresholds may improve utility but reduce privacy.




§ COROLLARY 5.2


*corollary5.2Corollary 5.2

[Impact of Dimensionality]
As d →∞, for fixed C and B, ϵ_1 approaches 2 ln(1 + 2C), while the behavior of ϵ_2 depends on how β scales with d. If β/d remains constant, ϵ_2 also approaches a constant value. This suggests that the privacy guarantee can be maintained even for high-dimensional gradients, which is beneficial for large machine learning models.




§ COROLLARY 6


*corollary6Corollary 6

This trade-off is a fundamental aspect of differential privacy, as discussed in <cit.>.

[Privacy-Utility Trade-off in Single-Step DP-BloGS]
Let M be the DP-BloGS mechanism as defined in Definition 2, applied to a gradient function g : 𝒟→ℝ^d with L_2 sensitivity Δ_2 g. For a given privacy parameter ϵ > 0, the mechanism achieves (ϵ, δ)-differential privacy while maintaining a utility bound:


    𝔼[M(D, ω) - g(D)_2^2] ≤min(2Δ_2 g^2 · d · e^-2ϵ/d, g(D)_2^2)


where the expectation is taken over the randomness in M.

Proof:

Let (Ω, ℱ, ℙ) be our probability space as defined in Definition 2.

Step 1: Relate ϵ to the mechanism's parameters

From Theorem 5, we know that for (ϵ, δ)-differential privacy:


    ϵ = ln(1 + d · (e^Δ_2 g / √(d) - 1))


Solving for Δ_2 g:


    e^Δ_2 g / √(d) - 1 = e^ϵ/d - 1/d


    e^Δ_2 g / √(d) = 1 + e^ϵ/d - 1/d


    Δ_2 g / √(d) = ln(1 + e^ϵ/d - 1/d)


    Δ_2 g = √(d)·ln(1 + e^ϵ/d - 1/d)


Step 2: Bound the expected squared error

From the properties of DP-BloGS (Lemma 2.2), we know that 𝔼[M(D, ω)] = g(D). Therefore:


    𝔼[M(D, ω) - g(D)_2^2] 
           = 𝔼[M(D, ω) - 𝔼[M(D, ω)]_2^2] 
       = Var(M(D, ω))


Now, let's bound this variance:


    Var(M(D, ω)) = ∑_i=1^d Var(M(D, ω)[i])


For each component i:

    Var(M(D, ω)[i]) 
           = 𝔼[(M(D, ω)[i] - g(D)[i])^2] 
       = 1/d∑_j=1^d (g(D)[j] - g(D)[i])^2 
       ≤4/dg(D)_2^2


Therefore:

    Var(M(D, ω)) ≤∑_i=1^d 4/dg(D)_2^2 = 4 g(D)_2^2


Step 3: Relate the bound to Δ_2 g

From the definition of L_2 sensitivity:

    g(D)_2^2 ≤Δ_2 g^2 · d


Substituting this into our variance bound:


    Var(M(D, ω)) ≤ 4 Δ_2 g^2 · d


Step 4: Express the bound in terms of ϵ

Using the relationship between Δ_2 g and ϵ from Step 1:


    Var(M(D, ω)) 
           ≤ 4 · d · (√(d)·ln(1 + e^ϵ/d - 1/d))^2 
       = 4d^2 · (ln(1 + e^ϵ/d - 1/d))^2


For small x, ln(1 + x) ≈ x. Using this approximation:


    Var(M(D, ω)) 
           ≈ 4d^2 ·(e^ϵ/d - 1/d)^2 
       = 4 ·(e^ϵ/d - 1/e^ϵ/d)^2 · (d · e^2ϵ/d) 
       ≤ 4 · (1 - e^-ϵ/d)^2 · (d · e^2ϵ/d) 
       ≤ 4 ·(ϵ/d)^2 · (d · e^2ϵ/d) 
         (using  1 - e^-x≤ x  for  x > 0) 
       = 4ϵ^2 · e^2ϵ/d / d 
       ≤ 2Δ_2 g^2 · d · e^-2ϵ/d
         (using the relationship between ϵ and Δ_2 g )


Step 5: Combine the bounds

Taking the minimum of the two bounds we've derived:


    𝔼[M(D, ω) - g(D)_2^2] ≤min(2Δ_2 g^2 · d · e^-2ϵ/d, g(D)_2^2)


This completes the proof.




§ LEMMA 6.1


*lemma6.1Lemma 6.1

[Optimal Privacy Parameter]
For a given utility constraint U > 0, the optimal privacy parameter ϵ that satisfies 𝔼[M(D, ω) - g(D)_2^2] ≤ U is:


    ϵ^* = d/2ln(2Δ_2 g^2 · d/U)


Proof:

Set the utility bound equal to U and solve for ϵ:


    U = 2Δ_2 g^2 · d · e^-2ϵ/d


    e^2ϵ/d = 2Δ_2 g^2 · d/U


    2ϵ/d = ln(2Δ_2 g^2 · d/U)


    ϵ^* = d/2ln(2Δ_2 g^2 · d/U)


This lemma provides a way to choose the optimal privacy parameter given a desired utility constraint.




§ LEMMA 7


*lemma7Lemma 7

[Independence of Successive Shuffles]
Let M be the DP-BloGS mechanism as defined in Definition 2. For any dataset D ∈𝒟 and any two distinct iterations t_1 and t_2, the shuffled gradients M(D, ω_1) and M(D, ω_2) are statistically independent, where ω_1, ω_2 ∈Ω are the random outcomes determining the shuffles at iterations t_1 and t_2 respectively.

Proof:

Let (Ω, ℱ, ℙ) be our probability space as defined in Definition 2.

Step 1: Define the random variables

Let X_1 = M(D, ω_1) and X_2 = M(D, ω_2) be the shuffled gradients at iterations t_1 and t_2 respectively.

Step 2: Prove independence

To prove that X_1 and X_2 are independent, we need to show that for any measurable sets A, B ⊆ℝ^d:


    ℙ(X_1 ∈ A, X_2 ∈ B) = ℙ(X_1 ∈ A) ·ℙ(X_2 ∈ B)


Let g = g(D) be the true gradient. From the definition of the DP-BloGS mechanism:


    ℙ(X_1 = v_1) = 1/d! if  v_1  is a permutation of  g,  and  0  otherwise


    ℙ(X_2 = v_2) = 1/d! if  v_2  is a permutation of  g,  and  0  otherwise


Let S_g be the set of all permutations of g. Then:


    ℙ(X_1 ∈ A, X_2 ∈ B) 
       = ∑_v_1 ∈ A ∩ S_g∑_v_2 ∈ B ∩ S_gℙ(X_1 = v_1, X_2 = v_2) 
       = ∑_v_1 ∈ A ∩ S_g∑_v_2 ∈ B ∩ S_gℙ(X_1 = v_1) ·ℙ(X_2 = v_2) 
         (by independence of ω_1  and ω_2) 
       = (1/d!)^2 · |A ∩ S_g| · |B ∩ S_g| 
       = (1/d!· |A ∩ S_g|) ·(1/d!· |B ∩ S_g|) 
       = ℙ(X_1 ∈ A) ·ℙ(X_2 ∈ B)


This holds for all measurable sets A and B, proving that X_1 and X_2 are independent.




§ COROLLARY 7.1


*corollary7.1Corollary 7.1

[Pairwise Independence of Multiple Shuffles]
For any dataset D ∈𝒟 and any finite sequence of distinct iterations t_1, …, t_n, the shuffled gradients M(D, ω_1), …, M(D, ω_n) are pairwise independent.

Proof:

This follows directly from Lemma 7. For any i  j, M(D, ω_i) and M(D, ω_j) are independent by Lemma 7.




§ THEOREM 7.2


*theorem7.2Theorem 7.2

[Mutual Independence of Multiple Shuffles]
For any dataset D ∈𝒟 and any finite sequence of distinct iterations t_1, …, t_n, the shuffled gradients M(D, ω_1), …, M(D, ω_n) are mutually independent.

Proof:

Let X_i = M(D, ω_i) for i = 1, …, n.

To prove mutual independence, we need to show that for any measurable sets A_1, …, A_n ⊆ℝ^d:


    ℙ(X_1 ∈ A_1, …, X_n ∈ A_n) = ℙ(X_1 ∈ A_1) ·…·ℙ(X_n ∈ A_n)


Let g = g(D) be the true gradient and S_g be the set of all permutations of g.


    ℙ(X_1 ∈ A_1, …, X_n ∈ A_n) 
       = ∑_v_1 ∈ A_1 ∩ S_g…∑_v_n ∈ A_n ∩ S_gℙ(X_1 = v_1, …, X_n = v_n) 
       = ∑_v_1 ∈ A_1 ∩ S_g…∑_v_n ∈ A_n ∩ S_gℙ(X_1 = v_1) ·…·ℙ(X_n = v_n) 
         (by independence of ω_1, …, ω_n) 
       = (1/d!)^n · |A_1 ∩ S_g| ·…· |A_n ∩ S_g| 
       = (1/d!· |A_1 ∩ S_g|) ·…·(1/d!· |A_n ∩ S_g|) 
       = ℙ(X_1 ∈ A_1) ·…·ℙ(X_n ∈ A_n)


This proves mutual independence.




§ LEMMA 7.3


*lemma7.3Lemma 7.3

[Preservation of Independence Under Gradient Updates]
Let D_t ∈𝒟 be the dataset at iteration t, and let M(D_t, ω_t) be the shuffled gradient at iteration t. Then, for any finite sequence of distinct iterations t_1, …, t_n, the shuffled gradients M(D_t_1, ω_t_1), …, M(D_t_n, ω_t_n) are mutually independent.

Proof:

The key observation is that the randomness in the shuffle at each iteration is independent of the dataset and the gradients.

Let X_i = M(D_t_i, ω_t_i) for i = 1, …, n.

For any measurable sets A_1, …, A_n ⊆ℝ^d:


    ℙ(X_1 ∈ A_1, …, X_n ∈ A_n) 
       = 𝔼[ℙ(X_1 ∈ A_1, …, X_n ∈ A_n | D_t_1, …, D_t_n)] 
       = 𝔼[ℙ(X_1 ∈ A_1 | D_t_1) ·…·ℙ(X_n ∈ A_n | D_t_n)] 
         (by independence of shuffles)
       = 𝔼[ℙ(X_1 ∈ A_1 | D_t_1)] ·…·𝔼[ℙ(X_n ∈ A_n | D_t_n)] 
       = ℙ(X_1 ∈ A_1) ·…·ℙ(X_n ∈ A_n)


This proves mutual independence even under gradient updates.




§ THEOREM 8


*theorem8Theorem 8

This result extends the basic composition theorem for differential privacy <cit.>.

[Basic Composition for DP-BloGS]
Let M be the DP-BloGS mechanism as defined in Definition 2, applied to a gradient function g : 𝒟→ℝ^d with L_2 sensitivity Δ_2 g. If M satisfies (ϵ, δ)-differential privacy for a single iteration, then the composition of T independent applications of M satisfies (Tϵ, Tδ)-differential privacy. 

Proof:

Let (Ω, ℱ, ℙ) be our probability space as defined in Definition 2.

Step 1: Define the composed mechanism

Let M_T : 𝒟^T ×Ω^T → (ℝ^d)^T be the T-fold composition of M, defined as:


    M_T((D_1, …, D_T), (ω_1, …, ω_T)) = (M(D_1, ω_1), …, M(D_T, ω_T))


where D_t ∈𝒟 is the dataset at iteration t, and ω_t ∈Ω is the random outcome determining the shuffle at iteration t.

Step 2: Consider adjacent datasets

Let D = (D_1, …, D_T) and D' = (D'_1, …, D'_T) be two adjacent sequences of datasets, differing in at most one element in one of the datasets.

Step 3: Define the privacy loss random variable

Let Z_t : Ω→ℝ be the privacy loss random variable for iteration t:


    Z_t(ω_t) = ln(ℙ(M(D_t, ω_t) = v_t)/ℙ(M(D'_t, ω_t) = v_t))


where v_t = M(D_t, ω_t) is the output of the mechanism at iteration t.

Step 4: Apply the single-iteration privacy guarantee

From the assumption that M satisfies (ϵ, δ)-differential privacy for a single iteration, we know that for any S ⊆ℝ:


    ℙ(Z_t ∈ S) ≤ e^ϵ·ℙ(Z_t ∈ S - ϵ) + δ


Step 5: Analyze the composed privacy loss

Let Z = ∑_t=1^T Z_t be the total privacy loss over T iterations.

From Lemma 7.3, we know that Z_1, …, Z_T are mutually independent.

For any S ⊆ℝ:


    ℙ(Z ∈ S) = ℙ(∑_t=1^T Z_t ∈ S) 
    
        = ∫_x_1+…+x_T ∈ Sℙ(Z_1 = x_1, …, Z_T = x_T)   dx_1 … dx_T 
    
        = ∫_x_1+…+x_T ∈ S∏_t=1^T ℙ(Z_t = x_t)   dx_1 … dx_T 
      (by independence)
    ≤∫_x_1+…+x_T ∈ S∏_t=1^T (e^ϵ·ℙ(Z_t ∈ [x_t-ϵ, x_t]) + δ)   dx_1 … dx_T 
    ≤ e^Tϵ·∫_x_1+…+x_T ∈ S-Tϵ∏_t=1^T ℙ(Z_t ∈ [x_t, x_t+ϵ])   dx_1 … dx_T 
       + 1 - (1-δ)^T 
    ≤ e^Tϵ·ℙ(Z ∈ S - Tϵ) + Tδ


The last inequality uses the fact that 1 - (1-δ)^T ≤ Tδ for δ∈ [0, 1] and T ≥ 1.

Step 6: Conclude the proof

The inequality derived in Step 5 is precisely the definition of (Tϵ, Tδ)-differential privacy for the composed mechanism M_T.

This completes the proof.




§ COROLLARY 8.1


*corollary8.1Corollary 8.1

[Privacy Degradation Rate]
Under the basic composition theorem for DP-BloGS, the privacy parameter ϵ grows linearly with the number of iterations T, while the failure probability δ also grows linearly with T.

Proof:

This follows directly from Theorem 8. After T iterations:
- The privacy parameter is Tϵ
- The failure probability is Tδ

Both grow linearly with T.




§ LEMMA 8.2


*lemma8.2Lemma 8.2

[Composition with Varying Privacy Parameters]
If the DP-BloGS mechanism satisfies (ϵ_t, δ_t)-differential privacy at iteration t, then the T-fold composition satisfies (∑_t=1^T ϵ_t, ∑_t=1^T δ_t)-differential privacy.

Proof:

The proof follows the same structure as Theorem 8, but instead of using the same ϵ and δ for each iteration, we use ϵ_t and δ_t.

In Step 5, the inequality becomes:


    ℙ(Z ∈ S) 
           ≤exp(∑_t=1^T ϵ_t) ·ℙ(Z ∈ S - ∑_t=1^T ϵ_t) 
          + 1 - ∏_t=1^T (1 - δ_t) 
       ≤exp(∑_t=1^T ϵ_t) ·ℙ(Z ∈ S - ∑_t=1^T ϵ_t) 
          + ∑_t=1^T δ_t


The last inequality uses the fact that 1 - ∏_t=1^T (1 - δ_t) ≤∑_t=1^T δ_t for δ_t ∈ [0, 1].

This lemma allows for more flexible privacy budgeting across iterations, which can be useful in adaptive privacy schemes.




§ THEOREM 9


*theorem9Theorem 9

[Advanced Composition for DP-BloGS]
Let M be the DP-BloGS mechanism as defined in Definition 2, applied to a gradient function g : 𝒟→ℝ^d with L_2 sensitivity Δ_2 g. If M satisfies (ϵ, δ)-differential privacy for a single iteration, then for any δ' > 0, the composition of T independent applications of M satisfies (ϵ', δ”)-differential privacy, where:


    ϵ' = √(2T ln(1/δ'))·ϵ + T ·ϵ· (e^ϵ - 1)


    δ” = T ·δ + δ'


This theorem builds upon the advanced composition theorem <cit.> and recent work on composition <cit.>.

Proof:

Let (Ω, ℱ, ℙ) be our probability space as defined in Definition 2.

Step 1: Define the composed mechanism

Let M_T : 𝒟^T ×Ω^T → (ℝ^d)^T be the T-fold composition of M, defined as:


    M_T((D_1, …, D_T), (ω_1, …, ω_T)) = (M(D_1, ω_1), …, M(D_T, ω_T))


where D_t ∈𝒟 is the dataset at iteration t, and ω_t ∈Ω is the random outcome determining the shuffle at iteration t.

Step 2: Consider adjacent datasets

Let D = (D_1, …, D_T) and D' = (D'_1, …, D'_T) be two adjacent sequences of datasets, differing in at most one element in one of the datasets.

Step 3: Define the privacy loss random variable

Let Z_t : Ω→ℝ be the privacy loss random variable for iteration t:


    Z_t(ω_t) = ln(ℙ(M(D_t, ω_t) = v_t)/ℙ(M(D'_t, ω_t) = v_t))


where v_t = M(D_t, ω_t) is the output of the mechanism at iteration t.

Step 4: Apply the moment generating function bound

From the properties of (ϵ, δ)-differential privacy, we can bound the moment generating function of Z_t:


    𝔼[exp(λ Z_t)] ≤exp(ϵ· (e^λ - 1)) · (1 - δ) + δ·exp(ϵλ)


for all λ > 0.

Step 5: Analyze the composition of privacy losses

Let Z = ∑_t=1^T Z_t be the total privacy loss over T iterations.

From Lemma 7.3, we know that Z_1, …, Z_T are mutually independent.

Therefore, for any λ > 0:


    𝔼[exp(λ Z)] 
           = ∏_t=1^T 𝔼[exp(λ Z_t)] 
       ≤∏_t=1^T (exp(ϵ· (e^λ - 1)) · (1 - δ) + δ·exp(ϵλ)) 
       ≤(exp(ϵ· (e^λ - 1)) + δ· (exp(ϵλ) - exp(ϵ· (e^λ - 1))))^T


Step 6: Apply Markov's inequality

For any t > 0:


    ℙ(Z > t) 
           = ℙ(exp(λ Z) > exp(λ t)) 
       ≤𝔼[exp(λ Z)]/exp(λ t)
       ≤1/exp(λ t)·(exp(ϵ· (e^λ - 1)) 
          + δ· (exp(ϵλ) - exp(ϵ· (e^λ - 1))))^T


Step 7: Optimize the bound

Let λ = ln(1 + √(2 ln(1/δ') / T) / ϵ).

Substituting this value of λ and simplifying (the full simplification is omitted for brevity but follows standard techniques in differential privacy literature), we get:


    ℙ(Z > ϵ') ≤δ'


where ϵ' = √(2T ln(1/δ'))·ϵ + T ·ϵ· (e^ϵ - 1).

Step 8: Account for the failure probability

With probability at most T ·δ, one or more of the T applications of M might fail to satisfy ϵ-differential privacy. Therefore, the total failure probability is bounded by T ·δ + δ'.

Step 9: Conclude the proof

Combining the results from Steps 7 and 8, we have shown that M_T satisfies (ϵ', δ”)-differential privacy, where:


    ϵ' = √(2T ln(1/δ'))·ϵ + T ·ϵ· (e^ϵ - 1)


    δ” = T ·δ + δ'


This completes the proof.




§ COROLLARY 9.1


*corollary9.1Corollary 9.1

[Asymptotic Behavior of Advanced Composition]
For large T and small ϵ, the privacy parameter ϵ' in the advanced composition theorem grows approximately as O(√(T)), which is significantly slower than the O(T) growth in the basic composition theorem.

Proof:

For small ϵ, we can use the approximation e^ϵ≈ 1 + ϵ + ϵ^2/2.

Substituting this into the expression for ϵ':


    ϵ' ≈√(2T ln(1/δ'))·ϵ + T ·ϵ· ((1 + ϵ + ϵ^2/2) - 1)


    ≈√(2T ln(1/δ'))·ϵ + T ·ϵ· (ϵ + ϵ^2/2)


    ≈√(2T ln(1/δ'))·ϵ + T ·ϵ^2


For large T and small ϵ, the first term dominates, giving us the O(√(T)) growth.




§ LEMMA 9.2


*lemma9.2Lemma 9.2

[Optimal Choice of ϵ for Fixed Total Privacy Budget]
Given a total privacy budget ϵ_total and a number of iterations T, the optimal choice of per-iteration privacy parameter ϵ that minimizes δ” in the advanced composition theorem is:


    ϵ = ϵ_total/√(2T ln(1/δ')) + T · (e^ϵ_total/T - 1)


Proof:

We want to minimize δ” subject to the constraint that ϵ' = ϵ_total.

Solving the equation:


    ϵ_total = √(2T ln(1/δ'))·ϵ + T ·ϵ· (e^ϵ - 1)


for ϵ gives us the stated result.




§ DEFINITION 10


*definition10Definition 10

[Adaptive Shuffling Strategy]

Let (Ω, ℱ, ℙ) be our probability space as defined in Definition 2.

Let 𝒟 be the space of datasets, ℝ^d be the d-dimensional gradient space, and T be the total number of iterations.

Define the history space ℋ_t for iteration t as:

    ℋ_t = (𝒟×ℝ^d)^t


Let h_t ∈ℋ_t be the history up to iteration t, which includes all previous datasets and shuffled gradients:

    h_t = ((D_1, g_1), …, (D_t, g_t))


Define an adaptive block size function β : ℋ_t →ℕ^+ that maps the history to a positive integer representing the block size.

Define an adaptive clipping threshold function C : ℋ_t →ℝ^+ that maps the history to a positive real number representing the clipping threshold.

The Adaptive DP-BloGS mechanism M_A : 𝒟×Ω×ℋ_t →ℝ^d is defined as:


    M_A(D, ω, h_t) = Shuffle(Clip(g(D), C(h_t)), β(h_t))


where:

    
  * g : 𝒟→ℝ^d is the gradient function,
    
  * Clip : ℝ^d ×ℝ^+ →ℝ^d is a gradient clipping function defined as:
    
    Clip(v, C) = v ·min(1, C/v_2)

    
  * Shuffle : ℝ^d ×ℕ^+ →ℝ^d is a shuffling function that divides the input vector into blocks of size β and randomly permutes these blocks.





§ LEMMA 10.1


*lemma10.1Lemma 10.1

Adaptive differential privacy mechanisms, as explored in <cit.>, inspire the following result. Recent work has further developed techniques for adaptive privacy accounting <cit.>.

[Conditional Independence of Adaptive Shuffles]
Given the history h_t, the output of M_A at iteration t+1 is independent of all previous outputs:

    ℙ(M_A(D_t+1, ω_t+1, h_t) = g_t+1| h_t) = ℙ(M_A(D_t+1, ω_t+1, h_t) = g_t+1)


Proof:

Let π_t : {1, …, ⌈ d/β(h_t) ⌉}→{1, …, ⌈ d/β(h_t) ⌉} be the random permutation used in the Shuffle function at iteration t.

Given h_t, the values of β(h_t) and C(h_t) are deterministic. Therefore:

    ℙ(M_A(D_t+1, ω_t+1, h_t) = g_t+1| h_t) 
       = ℙ(Shuffle(Clip(g(D_t+1), C(h_t)), β(h_t)) = g_t+1| h_t) 
       = ℙ(π_t+1 = σ| h_t) 
         where σ is the permutation that produces  g_t+1
       = 1/(⌈ d/β(h_t) ⌉)!
         since π_t+1 is uniformly random and independent of  h_t 
       = ℙ(M_A(D_t+1, ω_t+1, h_t) = g_t+1)


This completes the proof.




§ THEOREM 10.2


*theorem10.2Theorem 10.2

[Privacy Guarantee for Adaptive DP-BloGS]
Let M_A be the Adaptive DP-BloGS mechanism as defined above. If for all possible histories h_t, the mechanism satisfies (ϵ(h_t), δ(h_t))-differential privacy, then the T-fold adaptive composition of M_A satisfies (ϵ', δ')-differential privacy, where:

    ϵ'    = √(2T ln(1/δ^*))·max_t ϵ(h_t) 
          + T ·max_t ϵ(h_t) · (exp(max_t ϵ(h_t)) - 1) 
    δ'    = T ·max_t δ(h_t) + δ^*

for any δ^* > 0.

Proof:

Let D = (D_1, …, D_T) and D' = (D'_1, …, D'_T) be two adjacent sequences of datasets.

Define the privacy loss random variable for iteration t:

    Z_t(ω_t) = ln(ℙ(M_A(D_t, ω_t, h_t-1) = g_t)/ℙ(M_A(D'_t, ω_t, h_t-1) = g_t))


From the assumption of (ϵ(h_t), δ(h_t))-differential privacy, we know that for any S ⊆ℝ:

    ℙ(Z_t ∈ S | h_t-1) 
           ≤exp(ϵ(h_t-1)) ·ℙ(Z_t ∈ S - ϵ(h_t-1) | h_t-1) 
          + δ(h_t-1)


Let ϵ_max = max_t ϵ(h_t) and δ_max = max_t δ(h_t).

Then, for any λ > 0:

    𝔼[exp(λ Z_t) | h_t-1] 
           ≤exp(ϵ_max· (exp(λ) - 1)) · (1 - δ_max) 
          + δ_max·exp(ϵ_max·λ)


Let Z = ∑_t=1^T Z_t be the total privacy loss.

Using the independence result from Lemma 10.1:

    𝔼[exp(λ Z)] 
           = 𝔼[∏_t=1^T exp(λ Z_t)] 
       = 𝔼[𝔼[∏_t=1^T exp(λ Z_t) | h_T-1]] 
       = 𝔼[∏_t=1^T-1exp(λ Z_t) ·𝔼[exp(λ Z_T) | h_T-1]] 
       ≤𝔼[∏_t=1^T-1exp(λ Z_t)] · (exp(ϵ_max· (exp(λ) - 1)) 
         · (1 - δ_max) + δ_max·exp(ϵ_max·λ))


Continuing this process recursively:

    𝔼[exp(λ Z)] 
           ≤ (exp(ϵ_max· (exp(λ) - 1)) · (1 - δ_max) 
          + δ_max·exp(ϵ_max·λ))^T


From here, we can apply the same techniques used in the proof of Theorem 9 (Advanced Composition) to arrive at the stated bound.

This completes the proof.




§ COROLLARY 10.3


*corollary10.3Corollary 10.3

[Adaptive Privacy Budget Allocation]
In the Adaptive DP-BloGS mechanism, we can allocate privacy budget dynamically by setting:

    ϵ(h_t) = ϵ_total - ϵ_spent(h_t)/√(2(T-t) ln(1/δ^*)) + (T-t) ·(exp(ϵ_total - ϵ_spent(h_t)/T-t) - 1)

where ϵ_total is the total privacy budget, ϵ_spent(h_t) is the privacy budget spent up to iteration t, and T is the total number of iterations.

Proof:

This follows from applying Lemma 9.2 to the remaining iterations at each step, taking into account the already spent privacy budget.




§ THEOREM 11


*theorem11Theorem 11

[Privacy Guarantees for Adaptive DP-BloGS]
Let M_A be the Adaptive DP-BloGS mechanism as defined in Definition 10. For any δ > 0, if the adaptive block size function β(h_t) and adaptive clipping threshold function C(h_t) satisfy:


    
  * β(h_t) ≤β_max for all h_t
    
  * C(h_t) ≤ C_max for all h_t


Then M_A satisfies (ϵ, δ)-differential privacy for T iterations, where:


    ϵ   = min{ϵ_1, ϵ_2}
    ϵ_1    = √(2T ln(1/δ))· 2 ln(1 + d · (exp(2C_max / √(d)) - 1)) 
          + T · (2 ln(1 + d · (exp(2C_max / √(d)) - 1)))^2 
    ϵ_2    = √(2T ln(1/δ))· 2 ln(1 + (β_max/d) · (exp(2C_max√(β_max/d)) - 1)) 
          + T · (2 ln(1 + (β_max/d) · (exp(2C_max√(β_max/d)) - 1)))^2


where |B| is the batch size and d is the dimension of the gradient.

Proof:

Let (Ω, ℱ, ℙ) be our probability space as defined in Definition 2.

Step 1: Define the privacy loss random variable

For each iteration t, let Z_t be the privacy loss random variable:


    Z_t(ω_t) = ln(ℙ(M_A(D_t, ω_t, h_t-1) = g_t)/ℙ(M_A(D'_t, ω_t, h_t-1) = g_t))


where D_t and D'_t are adjacent datasets, h_t-1 is the history up to iteration t-1, and g_t is the output of the mechanism.

Step 2: Bound the sensitivity of the clipped gradient

For any adjacent datasets D and D', and any history h_t:


    Clip(g(D), C(h_t)) - Clip(g(D'), C(h_t)) _2    ≤ 2C(h_t) ≤ 2C_max


Step 3: Analyze the privacy loss for a single iteration

We can bound |Z_t| using two different approaches:


    
  * Treating the entire gradient as one block:
    
    |Z_t| ≤ 2 ln(1 + d · (exp(2C_max / (√(d))) - 1))

    
  * Using the maximum block size:
    
    |Z_t| ≤ 2 ln(1 + (β_max/d) · (exp(2C_max·√(β_max/d)) - 1))



The factor of 2 in these bounds accounts for the possibility that an adversary might exploit the randomness of the mechanism.

Let:


    ϵ_single = min{   2 ln(1 + d ·(exp(2C_max/√(d)) - 1)), 
       2 ln(1 + β_max/d·(exp(2C_max√(β_max/d)) - 1))
    }


Step 4: Apply advanced composition

Using the advanced composition theorem (Theorem 3.20 from Dwork and Roth, 2014), we can bound the total privacy loss over T iterations. For any δ > 0, with probability at least 1 - δ:


    ϵ≤√(2T ln(1/δ))·ϵ_single + T ·ϵ_single^2/2


Step 5: Derive the final bounds

Substituting the two bounds for ϵ_single gives us ϵ_1 and ϵ_2:


    ϵ_1    = √(2T ln(1/δ))· 2 ln(1 + d · (e^2C_max/√(d) - 1)) 
          + T · (2 ln(1 + d · (e^2C_max/√(d) - 1)))^2 
    ϵ_2    = √(2T ln(1/δ))· 2 ln(1 + (β_max/d) 
         · (e^2C_max√(β_max/d) - 1)) 
          + T · (2 ln(1 + (β_max/d) 
         · (e^2C_max√(β_max/d) - 1)))^2


Step 6: Conclude the proof

Taking the minimum of ϵ_1 and ϵ_2 gives us the tightest bound, completing the proof.





§ COROLLARY 11.1


*corollary11.1Corollary 11.1

[Optimal Adaptive Parameters]
The optimal choices for β(h_t) and C(h_t) that minimize ϵ while maintaining a fixed utility U are:


    β^*(h_t) = min{d, max{1, ⌊ d ·exp(-2ϵ_t / d) ⌋}}



    C^*(h_t) = √(U/2d)


where ϵ_t is the privacy budget allocated to iteration t.

Proof: (The proof remains the same as in the original theorem)




§ LEMMA 11.2


*lemma11.2Lemma 11.2

[Privacy Amplification by Subsampling in Adaptive DP-BloGS]
If each iteration of Adaptive DP-BloGS operates on a random subsample of size m from a dataset of size n, and the mechanism satisfies ϵ-differential privacy on the subsample, then it satisfies ϵ'-differential privacy on the full dataset, where:


    ϵ' = ln(1 + m/n · (exp(ϵ) - 1))


Proof:

Let q = m/n be the sampling ratio.

For any output S of the mechanism:


    ℙ(M_A(D) ∈ S) 
       = (1-q) ·ℙ(M_A(D) ∈ S | x ∉sample) 
          + q ·ℙ(M_A(D) ∈ S | x ∈sample) 
       ≤ (1-q) ·ℙ(M_A(D') ∈ S) + q ·exp(ϵ) ·ℙ(M_A(D') ∈ S) 
       = ((1-q) + q ·exp(ϵ)) ·ℙ(M_A(D') ∈ S) 
       = (1 + q · (exp(ϵ) - 1)) ·ℙ(M_A(D') ∈ S)


Taking the logarithm of both sides gives us ϵ'.




§ LEMMA 12


*lemma12Lemma 12

This approach leverages privacy amplification by subsampling, as analyzed in <cit.>.

[Interaction Between Shuffling and Mini-batch Sampling]
Let M_A be the Adaptive DP-BloGS mechanism as defined in Definition 10, and let S be a mini-batch sampling mechanism that selects m elements uniformly at random from a dataset of size n. Then, the composition of S and M_A, denoted as M_S = M_A ∘ S, satisfies (ϵ', δ')-differential privacy, where:


    ϵ' = ln(1 + q(e^ϵ - 1))



    δ' = qδ


where q = m/n is the sampling ratio, and (ϵ, δ) are the privacy parameters of M_A. 

Proof:

Let (Ω, ℱ, ℙ) be our probability space.

Step 1: Define the sampling mechanism

Let S : 𝒟×Ω→𝒟_m be the mini-batch sampling mechanism, where 𝒟_m is the space of datasets of size m. For any dataset D ∈𝒟 and ω∈Ω, S(D, ω) selects m elements from D uniformly at random.

Step 2: Consider adjacent datasets

Let D, D' ∈𝒟 be two adjacent datasets differing in one element, say x and x'.

Step 3: Analyze the probability of including the differing element

Let E be the event that the differing element is included in the mini-batch. Then:


    ℙ(E) = m/n = q


Step 4: Decompose the output probability

For any output O in the range of M_S:


    ℙ(M_S(D) ∈ O) 
       = ℙ(M_S(D) ∈ O | E) ·ℙ(E) 
          + ℙ(M_S(D) ∈ O |not  E) ·ℙ(not  E) 
       = q ·ℙ(M_S(D) ∈ O | E) 
          + (1-q) ·ℙ(M_S(D) ∈ O |not  E)


Step 5: Apply the privacy guarantee of M_A

When E occurs, we can apply the (ϵ, δ)-differential privacy guarantee of M_A:


    ℙ(M_S(D) ∈ O | E) ≤ e^ϵ·ℙ(M_S(D') ∈ O | E) + δ


When E does not occur, the output distributions are identical:


    ℙ(M_S(D) ∈ O |not  E) = ℙ(M_S(D') ∈ O |not  E)


Step 6: Combine the probabilities


    ℙ(M_S(D) ∈ O) 
       ≤ q · (e^ϵ·ℙ(M_S(D') ∈ O | E) + δ) 
          + (1-q) ·ℙ(M_S(D') ∈ O |not  E) 
       = q · e^ϵ·ℙ(M_S(D') ∈ O | E) + qδ
          + (1-q) ·ℙ(M_S(D') ∈ O |not  E) 
       ≤ (qe^ϵ + 1-q) ·ℙ(M_S(D') ∈ O) + qδ
       = (1 + q(e^ϵ - 1)) ·ℙ(M_S(D') ∈ O) + qδ


Step 7: Conclude the proof

From the inequality in Step 6, we can conclude that M_S satisfies (ϵ', δ')-differential privacy, where:


    ϵ' = ln(1 + q(e^ϵ - 1))



    δ' = qδ


This completes the proof.




§ COROLLARY 12.1


*corollary12.1Corollary 12.1

[Privacy Amplification by Subsampling]
When q < 1/e, the privacy guarantee of M_S is strictly better than that of M_A.

Proof:

We need to show that ϵ' < ϵ when q < 1/e.


    ϵ' < ϵ



    ⇔ln(1 + q(e^ϵ - 1)) < ϵ



    ⇔ 1 + q(e^ϵ - 1) < e^ϵ



    ⇔ q < e^ϵ - 1/e^ϵ - 1 = 1


The stricter condition q < 1/e ensures that this inequality holds for all ϵ > 0.




§ THEOREM 12.2


*theorem12.2Theorem 12.2

[Optimal Sampling Ratio]
Given a fixed privacy budget (ϵ, δ) and a desired number of iterations T, the optimal sampling ratio q^* that maximizes the number of iterations while maintaining (ϵ, δ)-differential privacy is:


    q^* = min(1, e^ϵ/T - 1/e^ϵ - 1)


Proof:

We want to maximize T subject to the constraint:


    ϵ≥ T ·ln(1 + q(e^ϵ/T - 1))


Treating this as an equality and solving for q gives us:


    q = e^ϵ/T - 1/e^ϵ - 1


This q is decreasing in T, so to maximize T, we choose the smallest q that is still valid (i.e., q ≤ 1).

Hence, q^* = min(1, e^ϵ/T - 1/e^ϵ - 1).




§ LEMMA 12.3


*lemma12.3Lemma 12.3

[Composition of Mini-batch Sampling and Adaptive DP-BloGS]
Let M_A be an Adaptive DP-BloGS mechanism that satisfies (ϵ(h_t), δ(h_t))-differential privacy at iteration t given history h_t. Let S be a mini-batch sampling mechanism with sampling ratio q. Then the T-fold adaptive composition of M_S = M_A ∘ S satisfies (ϵ', δ')-differential privacy, where:


    ϵ'    = √(2T ln(1/δ^*))·max_t ln(1 + q(e^ϵ(h_t) - 1)) 
          + T ·max_t ln(1 + q(e^ϵ(h_t) - 1)) 
         · (e^max_t ln(1 + q(e^ϵ(h_t) - 1)) - 1) 
    δ'    = T · q ·max_t δ(h_t) + δ^*


for any δ^* > 0.

Proof:

Let (Ω, ℱ, ℙ) be our probability space.

Step 1: Define the composed mechanism

Let M_S : 𝒟×Ω×ℋ_t →ℝ^d be the composition of S and M_A:


    M_S(D, ω, h_t) = M_A(S(D, ω_S), ω_A, h_t)


where ω = (ω_S, ω_A) ∈Ω, ω_S determines the random sampling, and ω_A determines the random shuffling.

Step 2: Analyze the privacy guarantee for a single iteration

From Lemma 12, we know that for any iteration t, given history h_t, M_S satisfies (ϵ_t, δ_t)-differential privacy where:


    ϵ_t = ln(1 + q(e^ϵ(h_t) - 1))



    δ_t = q ·δ(h_t)


Step 3: Define the privacy loss random variable

Let Z_t be the privacy loss random variable for iteration t:


    Z_t(ω) = ln(ℙ(M_S(D_t, ω, h_t-1) = g_t)/ℙ(M_S(D'_t, ω, h_t-1) = g_t))


where D_t and D'_t are adjacent datasets.

Step 4: Bound the moment generating function of Z_t

From the properties of (ϵ_t, δ_t)-differential privacy, we can bound the moment generating function of Z_t:


    𝔼[e^λ Z_t| h_t-1] ≤ e^ϵ_t · (e^λ - 1)· (1 - δ_t) + δ_t · e^ϵ_t ·λ


for all λ > 0.

Step 5: Define the worst-case privacy parameters

Let ϵ_max = max_t ϵ_t = max_t ln(1 + q(e^ϵ(h_t) - 1)) and δ_max = max_t δ_t = q ·max_t δ(h_t).

Step 6: Bound the total privacy loss

Let Z = ∑_t=1^T Z_t be the total privacy loss over T iterations.

Using the independence result from Lemma 10.1 and the bound from Step 4:


    𝔼[e^λ Z] 
           = 𝔼[∏_t=1^T e^λ Z_t] 
       = 𝔼[𝔼[∏_t=1^T e^λ Z_t| h_T-1]] 
       = 𝔼[∏_t=1^T-1 e^λ Z_t·𝔼[e^λ Z_T| h_T-1]] 
       ≤𝔼[∏_t=1^T-1 e^λ Z_t] · 
            (e^ϵ_max· (e^λ - 1)· (1 - δ_max) 
          + δ_max· e^ϵ_max·λ)


Continuing this process recursively:


    𝔼[e^λ Z] ≤ (e^ϵ_max· (e^λ - 1)· (1 - δ_max) + δ_max· e^ϵ_max·λ)^T


Step 7: Apply Markov's inequality

For any t > 0:


    ℙ(Z > t) 
           = ℙ(e^λ Z > e^λ t) 
       ≤𝔼[e^λ Z]/e^λ t
       ≤(e^ϵ_max· (e^λ - 1)· (1 - δ_max) 
            + δ_max· e^ϵ_max·λ)^T/e^λ t


Step 8: Optimize the bound

Let λ = ln(1 + √(2 ln(1/δ^*) / T) / ϵ_max).

Substituting this value of λ and simplifying (the full simplification is omitted for brevity but follows standard techniques in differential privacy literature), we get:


    ℙ(Z > ϵ') ≤δ^*


where ϵ' = √(2T ln(1/δ^*))·ϵ_max + T ·ϵ_max· (e^ϵ_max - 1).

Step 9: Account for the failure probability

With probability at most T ·δ_max, one or more of the T applications of M_S might fail to satisfy ϵ_max-differential privacy. Therefore, the total failure probability is bounded by T ·δ_max + δ^*.

Step 10: Conclude the proof

Combining the results from Steps 8 and 9, and substituting the expressions for ϵ_max and δ_max, we have shown that the T-fold adaptive composition of M_S satisfies (ϵ', δ')-differential privacy, where:


    ϵ'    = √(2T ln(1/δ^*))·max_t ln(1 + q(e^ϵ(h_t) - 1)) 
          + T ·max_t ln(1 + q(e^ϵ(h_t) - 1)) 
         · (e^max_t ln(1 + q(e^ϵ(h_t) - 1)) - 1) 
    δ'    = T · q ·max_t δ(h_t) + δ^*


This completes the proof.




§ THEOREM 13


*theorem13Theorem 13

[Privacy Amplification via Subsampling in DP-BloGS]
Let M be the DP-BloGS mechanism as defined in Definition 2, applied to a gradient function g : 𝒟→ℝ^d with L_2 sensitivity Δ_2 g. Let S be a Poisson subsampling mechanism with sampling probability q. Then, for any 0 < δ < 1, the composed mechanism M_S = M ∘ S satisfies (ϵ', δ)-differential privacy, where:


    ϵ' = log((1-q) + q ·exp(ϵ_0))


and ϵ_0 is the smallest positive real number satisfying:


    q · (exp(ϵ_0) - 1) ·exp(-ϵ_0) = δ


where ϵ is the privacy parameter of M without subsampling.

Proof:

Let (Ω, ℱ, ℙ) be our probability space.

Step 1: Define the Poisson subsampling mechanism

Let S : 𝒟×Ω→ 2^𝒟 be the Poisson subsampling mechanism. For any dataset D ∈𝒟 and ω∈Ω, S(D, ω) includes each element of D independently with probability q.

Step 2: Consider adjacent datasets

Let D, D' ∈𝒟 be two adjacent datasets differing in one element, say x and x'.

Step 3: Analyze the output probabilities

For any output O in the range of M_S:


    ℙ(M_S(D) ∈ O) 
           = (1-q) ·ℙ(M_S(D) ∈ O | x  not sampled) 
          + q ·ℙ(M_S(D) ∈ O | x  sampled) 
    ℙ(M_S(D') ∈ O) 
           = (1-q) ·ℙ(M_S(D') ∈ O | x'  not sampled) 
          + q ·ℙ(M_S(D') ∈ O | x'  sampled)


Step 4: Apply the privacy guarantee of M

When x (or x') is not sampled, the output distributions are identical:


    ℙ(M_S(D) ∈ O | x  not sampled) = ℙ(M_S(D') ∈ O | x'  not sampled)


When x (or x') is sampled, we can apply the ϵ-differential privacy guarantee of M:


    ℙ(M_S(D) ∈ O | x  sampled) ≤exp(ϵ) ·ℙ(M_S(D') ∈ O | x'  sampled)


Step 5: Combine the probabilities


    ℙ(M_S(D) ∈ O) 
           ≤ (1-q) ·ℙ(M_S(D') ∈ O | x'  not sampled) 
          + q · e^ϵ·ℙ(M_S(D') ∈ O | x'  sampled) 
       ≤ ((1-q) + q · e^ϵ) ·ℙ(M_S(D') ∈ O)


Step 6: Define the privacy loss random variable

Let Z be the privacy loss random variable:


    Z = ln(ℙ(M_S(D) ∈ O)/ℙ(M_S(D') ∈ O))


From Step 5, we know that:


    Z ≤ln((1-q) + q ·exp(ϵ))


Step 7: Analyze the tail probability

We want to find ϵ_0 such that:


    ℙ(Z > ϵ_0) ≤δ


This is equivalent to:


    q ·ℙ(Z > ϵ_0 | x  sampled) ≤δ


Given that Z ≤ϵ when x is sampled, we can write:


    q · (exp(ϵ_0) - 1) ·exp(-ϵ_0) ≤δ


Step 8: Define ϵ'

Let ϵ' = ln((1-q) + q ·exp(ϵ_0))

Step 9: Verify (ϵ', δ)-differential privacy

For any S ⊆ℝ:


    ℙ(Z ∈ S) ≤exp(ϵ') ·ℙ(Z ∈ S - ϵ') + ℙ(Z > ϵ_0)



    ≤exp(ϵ') ·ℙ(Z ∈ S - ϵ') + δ


This satisfies the definition of (ϵ', δ)-differential privacy.

Step 10: Conclude the proof

We have shown that M_S satisfies (ϵ', δ)-differential privacy, where:


    ϵ' = ln((1-q) + q ·exp(ϵ_0))


and ϵ_0 is the smallest positive real number satisfying:


    q · (exp(ϵ_0) - 1) ·exp(-ϵ_0) = δ


This completes the proof.




§ COROLLARY 13.1


*corollary13.1Corollary 13.1

[Asymptotic Behavior of Privacy Amplification]
As q → 0, ϵ' → 0 for any fixed ϵ and δ.

Proof:

As q → 0, the equation q · (exp(ϵ_0) - 1) ·exp(-ϵ_0) = δ implies that ϵ_0 →∞.

However, ϵ' = ln((1-q) + q ·exp(ϵ_0)) ≈ q ·exp(ϵ_0) for small q.

Therefore, as q → 0, ϵ' → 0.




§ LEMMA 13.2


*lemma13.2Lemma 13.2

[Optimal Sampling Probability]
Given a desired privacy level (ϵ', δ) and the original ϵ of the mechanism M, the optimal sampling probability q^* that maximizes the utility while satisfying (ϵ', δ)-differential privacy is:


    q^* = exp(ϵ') - 1/exp(ϵ) - 1


Proof:

We want to maximize q subject to:


    ϵ' = ln((1-q) + q ·exp(ϵ))


Solving this equation for q gives us:


    q = exp(ϵ') - 1/exp(ϵ) - 1


This q maximizes the sample size (and thus the utility) while still satisfying the desired privacy level.




§ LEMMA 14


*lemma14Lemma 14

[Expected Gradient Under DP-BloGS]
Let M be the DP-BloGS mechanism as defined in Definition 2, applied to a gradient function g : 𝒟→ℝ^d. For any dataset D ∈𝒟, the expected value of the shuffled gradient is equal to the true gradient:


    𝔼[M(D, ω)] = g(D)


where the expectation is taken over the random shuffling ω.

Proof:

Let (Ω, ℱ, ℙ) be our probability space as defined in Definition 2.

Step 1: Recall the definition of M


    M(D, ω) = Shuffle(g(D), β, π(ω))


where β is the block size and π : Ω→𝒮_m is a function that maps the random outcome ω to a permutation of m = ⌈ d/β⌉ blocks.

Step 2: Express the expectation componentwise

For any i ∈{1, …, d}, let M(D, ω)[i] denote the i-th component of the shuffled gradient. We need to show that:


    𝔼[M(D, ω)[i]] = g(D)[i]


Step 3: Analyze the probability distribution of M(D, ω)[i]

Let b(i) = ⌊ (i-1)/β⌋ be the block index of the i-th component, and let r(i) = (i-1) β be the position within its block.

The value of M(D, ω)[i] depends on which block is shuffled to position b(i). For any j ∈{1, …, m}:


    ℙ(π(ω)(j) = b(i)) = 1/m


If π(ω)(j) = b(i), then M(D, ω)[i] = g(D)[β· j + r(i) + 1].

Step 4: Compute the expectation


    𝔼[M(D, ω)[i]] 
           = ∑_j=1^m ℙ(π(ω)(j) = b(i)) · g(D)[β· j + r(i) + 1] 
       = 1/m·∑_j=1^m g(D)[β· j + r(i) + 1]


Step 5: Simplify the summation

The sum ∑_j=1^m g(D)[β· j + r(i) + 1] includes every β-th element of g(D), starting from position r(i) + 1. This sum, when divided by m, is equal to the average of all elements in g(D) that have the same position within their block as g(D)[i].

Due to the nature of integer division and modulo operations, this average is exactly g(D)[i]:


    1/m·∑_j=1^m g(D)[β· j + r(i) + 1] = g(D)[i]


Step 6: Conclude the proof

We have shown that for all i ∈{1, …, d}:


    𝔼[M(D, ω)[i]] = g(D)[i]


Therefore:


    𝔼[M(D, ω)] = g(D)


This completes the proof.




§ COROLLARY 14.1


*corollary14.1Corollary 14.1

[Unbiasedness of DP-BloGS]
The DP-BloGS mechanism M is an unbiased estimator of the true gradient g(D).

Proof:
This follows directly from Lemma 14, as the definition of an unbiased estimator is that its expected value equals the true value being estimated.




§ THEOREM 14.2


*theorem14.2Theorem 14.2

[Variance of DP-BloGS]
Let M be the DP-BloGS mechanism with block size β. The variance of each component of the shuffled gradient is bounded by:


    Var(M(D, ω)[i]) ≤β-1/β·Var(g(D))


where Var(g(D)) is the variance of the components of the true gradient.

Proof:

Step 1: Express the variance


    Var(M(D, ω)[i]) 
           = 𝔼[(M(D, ω)[i] - 𝔼[M(D, ω)[i]])^2] 
       = 𝔼[M(D, ω)[i]^2] - (𝔼[M(D, ω)[i]])^2


Step 2: Analyze 𝔼[M(D, ω)[i]^2]


    𝔼[M(D, ω)[i]^2] 
           = ∑_j=1^m ℙ(π(ω)(j) = b(i)) · (g(D)[β· j + r(i) + 1])^2 
       = 1/m·∑_j=1^m (g(D)[β· j + r(i) + 1])^2


Step 3: Apply the Cauchy-Schwarz inequality


    (∑_j=1^m g(D)[β· j + r(i) + 1])^2 ≤ m ·∑_j=1^m (g(D)[β· j + r(i) + 1])^2


Dividing both sides by m^2:


    (𝔼[M(D, ω)[i]])^2 ≤1/m·𝔼[M(D, ω)[i]^2]


Step 4: Bound the variance


    Var(M(D, ω)[i])    = 𝔼[M(D, ω)[i]^2] - (𝔼[M(D, ω)[i]])^2 
       ≥𝔼[M(D, ω)[i]^2] - 1/m·𝔼[M(D, ω)[i]^2] 
       = (1 - 1/m) ·𝔼[M(D, ω)[i]^2]



    = (m-1/m) ·1/m·∑_j=1^m (g(D)[β· j + r(i) + 1])^2


Step 5: Relate to the variance of g(D)


    Var(g(D)) = 1/d·∑_k=1^d (g(D)[k] - μ)^2,  where μ = 1/d·∑_k=1^d g(D)[k]


Observe that:


    1/m·∑_j=1^m (g(D)[β· j + r(i) + 1])^2 
         ≤β/d·∑_k=1^d (g(D)[k])^2 
         ≤β·Var(g(D))


Step 6: Conclude the proof


    Var(M(D, ω)[i]) ≤(m-1/m) ·β·Var(g(D)) = (β-1/β) ·Var(g(D))


This completes the proof.




§ THEOREM 15


*theorem15Theorem 15

This convergence analysis extends work on differentially private SGD <cit.> and general SGD convergence <cit.>. It also builds upon foundational work on private empirical risk minimization <cit.>.

[Convergence of DP-BloGS SGD]
Let f : ℝ^d →ℝ be a convex function with L-Lipschitz continuous gradients. Let M be the DP-BloGS mechanism as defined in Definition 2, applied to the stochastic gradient g(θ, ξ), where ξ is a random data sample. Assume that 𝔼[g(θ, ξ)] = ∇ f(θ) and 𝔼[g(θ, ξ)^2] ≤ G^2 for all θ. Let {θ_t} be the sequence of iterates generated by the DP-BloGS SGD algorithm:


    θ_t+1 = θ_t - η_t M(g(θ_t, ξ_t), ω_t)


where η_t is the learning rate at iteration t. Then, for T iterations and any δ > 0, with probability at least 1 - δ:


    𝔼[f(θ̅_T) - f(θ^*)] ≤θ_0 - θ^*^2/2η T + η L G^2/2 + G √(2 log(1/δ)/T)


where θ̅_T = 1/T∑_t=1^T θ_t is the average of the iterates, θ^* is the optimal solution, and η = 1/T∑_t=1^T η_t is the average learning rate. 

Proof:

Let (Ω, ℱ, ℙ) be our probability space.

Step 1: Express the update rule


    θ_t+1 = θ_t - η_t M(g(θ_t, ξ_t), ω_t)


Step 2: Analyze the squared distance to the optimum

    θ_t+1 - θ^*^2 
           = θ_t - η_t M(g(θ_t, ξ_t), ω_t) - θ^*^2 
       = θ_t - θ^*^2 + η_t^2 M(g(θ_t, ξ_t), ω_t)^2 
          - 2η_t ⟨θ_t - θ^*, M(g(θ_t, ξ_t), ω_t) ⟩


Step 3: Take expectation over ω_t and ξ_t

    𝔼[θ_t+1 - θ^*^2] 
           = θ_t - θ^*^2 + η_t^2 𝔼[M(g(θ_t, ξ_t), ω_t)^2] 
          - 2η_t ⟨θ_t - θ^*, 𝔼[M(g(θ_t, ξ_t), ω_t)] ⟩


From Lemma 14, we know that 𝔼[M(g(θ_t, ξ_t), ω_t)] = 𝔼[g(θ_t, ξ_t)] = ∇ f(θ_t).

Also, from Theorem 14.2 and our assumption, 𝔼[M(g(θ_t, ξ_t), ω_t)^2] ≤𝔼[g(θ_t, ξ_t)^2] ≤ G^2.

Step 4: Apply these results


    𝔼[θ_t+1 - θ^*^2] ≤θ_t - θ^*^2 + η_t^2 G^2 - 2η_t ⟨θ_t - θ^*, ∇ f(θ_t) ⟩


Step 5: Use the convexity of f

By convexity of f, we have:


    f(θ_t) - f(θ^*) ≤⟨∇ f(θ_t), θ_t - θ^* ⟩


Rearranging:


    -⟨∇ f(θ_t), θ_t - θ^* ⟩≤ -(f(θ_t) - f(θ^*))


Step 6: Combine the results


    𝔼[θ_t+1 - θ^*^2] ≤θ_t - θ^*^2 + η_t^2 G^2 - 2η_t(f(θ_t) - f(θ^*))


Step 7: Rearrange terms


    2η_t(f(θ_t) - f(θ^*)) ≤θ_t - θ^*^2 - 𝔼[θ_t+1 - θ^*^2] + η_t^2 G^2


Step 8: Sum over t = 0 to T-1


    ∑_t=0^T-1 2η_t(f(θ_t) - f(θ^*)) ≤θ_0 - θ^*^2 - 𝔼[θ_T - θ^*^2] + G^2 ∑_t=0^T-1η_t^2


Step 9: Apply Jensen's inequality


    f(θ̅_T) - f(θ^*) ≤1/T∑_t=0^T-1 (f(θ_t) - f(θ^*))


Step 10: Combine and rearrange


    2Tη(f(θ̅_T) - f(θ^*)) ≤θ_0 - θ^*^2 + G^2 ∑_t=0^T-1η_t^2


where η = 1/T∑_t=0^T-1η_t.

Step 11: Apply Cauchy-Schwarz inequality


    ∑_t=0^T-1η_t^2 ≤ T ∑_t=0^T-1η_t^2/T = Tη^2


Step 12: Rearrange to get the bound


    𝔼[f(θ̅_T) - f(θ^*)] ≤θ_0 - θ^*^2/2η T + η G^2/2


Step 13: Account for the high probability bound

To get a high probability bound, we can use the Azuma-Hoeffding inequality. Let X_t = f(θ_t) - f(θ^*) - (𝔼[f(θ_t)] - f(θ^*)). {X_t} forms a martingale difference sequence with |X_t| ≤ 2Gθ_t - θ^*≤ 2G^2.

By Azuma-Hoeffding, with probability at least 1 - δ:


    |1/T∑_t=1^T X_t| ≤ G√(2log(1/δ)/T)


Step 14: Conclude the proof

Combining the expectation bound and the high probability bound, we get:


    𝔼[f(θ̅_T) - f(θ^*)] ≤θ_0 - θ^*^2/2η T + η G^2/2 + G√(2log(1/δ)/T)


with probability at least 1 - δ.

This completes the proof.




§ COROLLARY 15.1


*corollary15.1Corollary 15.1

[Optimal Learning Rate]
The optimal constant learning rate η that minimizes the bound in Theorem 15 is:


    η^* = √(θ_0 - θ^*^2/G^2 T)


Proof:

Let (Ω, ℱ, ℙ) be our probability space.

Step 1: Recall the bound from Theorem 15


    𝔼[f(θ̅_T) - f(θ^*)] ≤θ_0 - θ^*^2/2η T + η G^2/2 + G√(2log(1/δ)/T)


Step 2: Consider constant learning rate

Let η = η for all iterations. Our goal is to minimize:


    B(η) = θ_0 - θ^*^2/2η T + η G^2/2


Note: We ignore the term G√(2log(1/δ)/T) as it doesn't depend on η.

Step 3: Find the derivative of B(η)


    B'(η) = d/dη( θ_0 - θ^*^2/2η T + η G^2/2) = -θ_0 - θ^*^2/2η^2 T + G^2/2


Step 4: Set the derivative to zero and solve


    0 = -θ_0 - θ^*^2/2η^2 T + G^2/2



    θ_0 - θ^*^2/2η^2 T = G^2/2



    η^2 = θ_0 - θ^*^2/G^2 T



    η^* = √(θ_0 - θ^*^2/G^2 T)


Step 5: Verify second derivative for minimum


    B”(η) = d^2/dη^2( θ_0 - θ^*^2/2η T + η G^2/2) = θ_0 - θ^*^2/η^3 T


At η = η^*, B”(η^*) > 0, confirming that this is indeed a minimum.

Step 6: Interpret the result

The optimal learning rate η^* balances two competing factors:

    
  * The initial distance to the optimum θ_0 - θ^*^2
    
  * The bound on the gradient norm G^2
    
  * The number of iterations T


As T increases, the optimal learning rate decreases, which is consistent with common practices in optimization where the learning rate is often decreased over time.

Step 7: Substituting η^* back into the bound

Let's substitute η^* back into B(η) to see the optimized bound:


    B(η^*)    = θ_0 - θ^*^2/2√(θ_0 - θ^*^2/G^2 T)T 
        + √(θ_0 - θ^*^2/G^2 T)G^2/2
       = √(θ_0 - θ^*^2 G^2 T/T) 
        + √(θ_0 - θ^*^2 G^2 T/T)


Therefore, with the optimal constant learning rate, the bound becomes:


    𝔼[f(θ̅_T) - f(θ^*)] ≤ 2 √(θ_0 - θ^*^2 G^2/T) + G √(2 log(1/δ)/T)


This shows that the expected suboptimality converges at a rate of O(1/√(T)), which is optimal for stochastic convex optimization.

This completes the proof.




§ COROLLARY 16


*corollary16Corollary 16

[Utility Bounds for DP-BloGS]
Let M be the DP-BloGS mechanism as defined in Definition 2, applied to a gradient function g : 𝒟→ℝ^d with L_2 sensitivity Δ_2g. Let f : ℝ^d →ℝ be an L-smooth convex function optimized using DP-BloGS SGD for T iterations with constant learning rate η. Then, for any δ > 0, with probability at least 1 - δ:


    𝔼[f(θ̅_T) - f(θ^*)] 
           ≤θ_0 - θ^*^2/2η T 
        + η L(G^2 + σ^2)/2
          + (G + σ)√(2log(2/δ)/T)


where θ̅_T = 1/T∑_t=1^T θ_t is the average of the iterates, θ^* is the optimal solution, G^2 is an upper bound on 𝔼[g(θ, ξ)^2], and σ^2 = (β-1)G^2/β is the maximum variance introduced by DP-BloGS with block size β.

Proof:

Let (Ω, ℱ, ℙ) be our probability space.

Step 1: Recall the update rule of DP-BloGS SGD


    θ_t+1 = θ_t - η M(g(θ_t, ξ_t), ω_t)


where ξ_t is the random data sample at iteration t, and ω_t is the random shuffle.

Step 2: Decompose the shuffled gradient

Let g̃_t = M(g(θ_t, ξ_t), ω_t) be the shuffled gradient. We can decompose it as:


    g̃_t = ∇ f(θ_t) + ζ_t + ν_t


where:

    
  * ∇ f(θ_t) is the true gradient
    
  * ζ_t = g(θ_t, ξ_t) - ∇ f(θ_t) is the stochastic gradient noise
    
  * ν_t = g̃_t - g(θ_t, ξ_t) is the additional noise introduced by shuffling


Step 3: Analyze the properties of ζ_t and ν_t


    𝔼[ζ_t] = 0    (unbiased stochastic gradient)


    𝔼[ζ_t^2] ≤ G^2    (bounded second moment)


From Lemma 14 and Theorem 14.2:

    𝔼[ν_t] = 0    (unbiased shuffle)


    𝔼[ν_t^2] ≤σ^2 = (β-1)G^2/β   (variance bound from Theorem 14.2)


Step 4: Follow the proof structure of Theorem 15

Similar to the proof of Theorem 15, we can derive:


    𝔼[θ_t+1 - θ^*^2] ≤θ_t - θ^*^2 + η^2𝔼[g̃_t^2] - 2η⟨θ_t - θ^*, ∇ f(θ_t) ⟩


Step 5: Bound 𝔼[g̃_t^2]


    𝔼[g̃_t^2] = 𝔼[∇ f(θ_t) + ζ_t + ν_t^2]


    ≤∇ f(θ_t)^2 + 𝔼[ζ_t^2] + 𝔼[ν_t^2]    (by Jensen's inequality)


    ≤ G^2 + G^2 + σ^2 = 2G^2 + σ^2


Step 6: Apply L-smoothness

By L-smoothness of f:


    f(θ_t+1) ≤ f(θ_t) + ⟨∇ f(θ_t), θ_t+1 - θ_t ⟩ + L/2θ_t+1 - θ_t^2


    = f(θ_t) - η⟨∇ f(θ_t), g̃_t ⟩ + Lη^2/2g̃_t^2


Taking expectation:


    𝔼[f(θ_t+1)] ≤ f(θ_t) - η∇ f(θ_t)^2 + Lη^2/2(2G^2 + σ^2)


Step 7: Sum over t = 0 to T-1


    ∑_t=0^T-1𝔼[f(θ_t+1) - f(θ^*)] 
       ≤∑_t=0^T-1 [f(θ_t) - f(θ^*) - η∇ f(θ_t)^2 
        + Lη^2/2(2G^2 + σ^2)]


Step 8: Apply Jensen's inequality


    f(θ̅_T) - f(θ^*) ≤1/T∑_t=0^T-1 [f(θ_t) - f(θ^*)]


Step 9: Combine and rearrange

    𝔼[f(θ̅_T) - f(θ^*)] 
           ≤f(θ_0) - f(θ^*)/T + Lη/2(2G^2 + σ^2) 
          - η/T∑_t=0^T-1∇ f(θ_t)^2


Step 10: Use the bound on θ_0 - θ^*^2

    f(θ_0) - f(θ^*) 
           ≤⟨∇ f(θ^*), θ_0 - θ^* ⟩ 
            + L/2θ_0 - θ^*^2 
       ≤L/2θ_0 - θ^*^2


Step 11: Apply Azuma-Hoeffding inequality

Let X_t = f(θ_t) - f(θ^*) - (𝔼[f(θ_t)] - f(θ^*)). {X_t} forms a martingale difference sequence with |X_t| ≤ (G + σ)θ_t - θ^*≤ (G + σ)√(2/L)(f(θ_t) - f(θ^*)).

By Azuma-Hoeffding, with probability at least 1 - δ/2:


    |1/T∑_t=1^T X_t| ≤ (G + σ)√(2log(2/δ)/T)


Step 12: Conclude the proof

Combining all the bounds and applying the high probability bound, we get:

    𝔼[f(θ̅_T) - f(θ^*)] 
           ≤θ_0 - θ^*^2/2η T 
            + η L(G^2 + σ^2)/2
          + (G + σ)√(2log(2/δ)/T)


with probability at least 1 - δ.

This completes the proof.





§ THEOREM 19


*theorem19Theorem 19

This privacy accounting method builds on the moments accountant technique <cit.>.

[Privacy Accounting for DP-BloGS]
Let M be the DP-BloGS mechanism as defined in Definition 2, applied T times independently with block size β and gradient clipping threshold C. Then, for any δ > 0, the mechanism satisfies (ϵ, δ)-differential privacy, where:


    ϵ = min{ϵ_1, ϵ_2}



    ϵ_1    = √(2T ln(1/δ))·ln(1 + d · (exp(2C / (√(d))) - 1)) 
          + T ·ln(1 + d · (exp(2C / (√(d))) - 1)) 
         · (exp(ln(1 + d · (exp(2C / (√(d))) - 1))) - 1)



    ϵ_2    = √(2T ln(1/δ))·ln(1 + (β/d) · (exp(2C ·√(β/d)) - 1)) 
          + T ·ln(1 + (β/d) · (exp(2C ·√(β/d)) - 1)) 
         · (exp(ln(1 + (β/d) · (exp(2C ·√(β/d)) - 1))) - 1)


where |B| is the batch size and d is the dimension of the gradient.

Proof:

Let (Ω, ℱ, ℙ) be our probability space.

Step 1: Define the privacy loss random variable

For each iteration t, let Z_t be the privacy loss random variable:


    Z_t = ln( ℙ(M(g(D), ω_t) = v)/ℙ(M(g(D'), ω_t) = v))


where D and D' are adjacent datasets, g is the gradient function, and v is the output of the mechanism.

Step 2: Bound the privacy loss for a single iteration

From the properties of DP-BloGS, we have two bounds on Z_t:

a) Treating the entire gradient as one block:
   
    |Z_t| ≤ln(1 + d · (exp(2C / (√(d))) - 1))


b) Using the block size β:
   
    |Z_t| ≤ln(1 + (β/d) · (exp(2C ·√(β/d)) - 1))


Let ϵ_0 = min{ln(1 + d · (exp(2C / (√(d))) - 1)),
               ln(1 + (β/d) · (exp(2C ·√(β/d)) - 1))}

Then, for any λ > 0:


    𝔼[exp(λ Z_t)] ≤exp(ϵ_0 · (exp(λ) - 1))


Step 3: Analyze the composition over T iterations

Let Z = ∑_t=1^T Z_t be the total privacy loss.

Using the independence of the iterations:


    𝔼[exp(λ Z)] = ∏_t=1^T 𝔼[exp(λ Z_t)] ≤exp(T ·ϵ_0 · (exp(λ) - 1))


Step 4: Apply Markov's inequality

For any t > 0:


    ℙ(Z > t) 
           = ℙ(exp(λ Z) > exp(λ t)) 
       ≤𝔼[exp(λ Z)]/exp(λ t)
       ≤exp(T ·ϵ_0 · (exp(λ) - 1) - λ t)


Step 5: Optimize the bound

Let λ = ln(1 + √(2 ln(1/δ) / (T ·ϵ_0)))

Substituting this value of λ and simplifying:


    ℙ(Z > ϵ) ≤δ


where ϵ = √(2T ln(1/δ))·ϵ_0 + T ·ϵ_0 · (exp(ϵ_0) - 1)

Step 6: Conclude the proof

Substituting the two bounds for ϵ_0 gives us ϵ_1 and ϵ_2. Taking the minimum of these provides the tightest bound, completing the proof.





§ COROLLARY 19.1


*corollary19.1Corollary 19.1

[Asymptotic Behavior of Privacy Loss]
As T →∞, the privacy loss ϵ grows as O(√(T)), which is significantly slower than the O(T) growth in the basic composition theorem.

Proof:

As T →∞, the dominant term in both ϵ_1 and ϵ_2 is √(2T ln(1/δ))·ϵ_0, where ϵ_0 is a constant with respect to T. Therefore, ϵ = O(√(T)).





§ LEMMA 19.2


*lemma19.2Lemma 19.2

[Optimal Block Size]
Given a fixed privacy budget ϵ and a number of iterations T, the optimal block size β^* that minimizes the variance introduced by DP-BloGS while satisfying (ϵ, δ)-differential privacy is:


    β^* = min{d, max{1, ⌊ d ·exp(-2ϵ / (√(2T ln(1/δ))· d))⌋}}


Proof:

The variance introduced by DP-BloGS is proportional to (β-1)/β (from Theorem 14.2). We want to maximize β while ensuring that ϵ_2 ≤ϵ.

Solving the equation:


    √(2T ln(1/δ))·ln(1 + (β/d) · (exp(2C ·√(β/d)) - 1)) = ϵ


for β gives us the stated result.





§ LEMMA 20


*lemma20Lemma 20

This analysis is related to information-theoretic approaches to generalization in machine learning <cit.>. Drawing from information-theoretic approaches to privacy <cit.>, we establish the following bound:

[Information-Theoretic Bounds on Shuffling Privacy]
Let M be the DP-BloGS mechanism as defined in Definition 2, applied to a gradient g ∈ℝ^d with block size β. The mutual information between the input gradient g and the output shuffled gradient M(g, ω) is bounded by:


    I(g; M(g, ω)) ≤log(d/β)


where I(·; ·) denotes mutual information.

Proof:

Let (Ω, ℱ, ℙ) be our probability space.

Step 1: Define the mutual information


    I(g; M(g, ω)) = H(M(g, ω)) - H(M(g, ω) | g)


where H(·) denotes the Shannon entropy and H(· | ·) denotes conditional entropy.

Step 2: Analyze H(M(g, ω))

The output space of M(g, ω) is the same as the input space, ℝ^d. Therefore:


    H(M(g, ω)) ≤log(|ℝ^d|) = d log(|ℝ|)


Step 3: Analyze H(M(g, ω) | g)

Given g, the only uncertainty in M(g, ω) comes from the random permutation of blocks. There are m = ⌈ d/β⌉ blocks, so:


    H(M(g, ω) | g) = log(m!) ≥log((d/β)!)


Step 4: Combine the results


    I(g; M(g, ω)) ≤ d log(|ℝ|) - log((d/β)!)


Step 5: Apply Stirling's approximation


    log(n!) ≈ n log(n) - n + O(log(n))


Applying this to our bound:

    I(g; M(g, ω)) ≤ d log(|ℝ|) - (d/βlog(d/β) - d/β + O(log(d/β)))


    = d log(|ℝ|) - d/βlog(d/β) + d/β - O(log(d/β))


Step 6: Analyze the behavior with respect to β
As |ℝ| →∞, the d log(|ℝ|) term is large but constant with respect to β. The term that depends on β is -d/βlog(d/β) + d/β. As β decreases, this term becomes more negative, tightening the upper bound.

Step 7: Express the bound in terms of β
We can express our bound as:

    I(g; M(g, ω)) ≤ C - d/βlog(d/β) + d/β + O(log(d/β))

where C = d log(|ℝ|) is a constant with respect to β.

Step 8: Conclude the proof
This bound shows that as β decreases, the upper bound on mutual information tightens, potentially providing better privacy. The exact behavior depends on the relative magnitudes of d and β, but in general, smaller block sizes lead to a smaller upper bound on information leakage.

This completes the proof.





§ THEOREM 20.1


*theorem20.1Theorem 20.1

[Lower Bound on Reconstruction Error]
Let g̃ = M(g, ω) be the output of the DP-BloGS mechanism, and let ĝ be any estimator of g based on g̃. Then:

    𝔼[ĝ - g^2] ≥ (d - β) ·exp( -2 I(g; g̃)/d) ·Var(g)

where Var(g) is the average variance of the components of g.



Step 1: Apply the rate-distortion theory

By the rate-distortion theory, for any estimator ĝ:

    𝔼[ĝ - g^2] ≥ d · 2^-2 I(g; g̃)/d·Var(g)


Step 2: Use the bound on mutual information

From Lemma 20, we know that I(g; g̃) ≤log(d/β). Substituting this:

    𝔼[ĝ - g^2] ≥ d · 2^-2 log(d/β)/d·Var(g)
    = d ·( β/d)^2/d·Var(g)


Step 3: Apply the inequality (1 + x)^r ≥ 1 + rx for x > -1 and r ∈ [0, 1]

Let x = β/d - 1 and r = 2/d. Then:

    ( β/d)^2/d = ( 1 + ( β/d - 1 ) )^2/d≥ 1 + 2/d( β/d - 1 ) = 1 - 2(d - β)/d^2


Step 4: Substitute back


    𝔼[ĝ - g^2]    ≥ d ·( 1 - 2(d - β)/d^2) ·Var(g) 
       = (d - 2(d - β)/d) ·Var(g) 
       = (d - β) ·( 2/d) ·Var(g) + β·Var(g) 
       ≥ (d - β) ·exp( -2 log(d/β)/d) ·Var(g)


This completes the proof.




§ COROLLARY 20.2


*corollary20.2Corollary 20.2

[Privacy-Utility Trade-off]
There exists a fundamental trade-off between privacy (measured by mutual information) and utility (measured by reconstruction error) in the DP-BloGS mechanism, governed by the block size β.



From Lemma 20 and Theorem 20.1, we can see that:

    
  * As β decreases, the upper bound on mutual information (log(d/β)) decreases, providing more privacy.
    
  * As β decreases, the lower bound on reconstruction error increases, providing less utility.


This demonstrates the inherent trade-off between privacy and utility controlled by the block size β.




§ LEMMA 21


*lemma21Lemma 21

Inspired by layer-wise approaches and iterative algorithms in private deep learning <cit.>, we introduce parameter-wise DP-BloGS:

[Privacy Loss for Parameter-wise DP-BloGS]
Let (Ω, ℱ, ℙ) be a probability space, where Ω is the sample space, ℱ is a σ-algebra on Ω, and ℙ is a probability measure. Let 𝒟 be the space of datasets, and let D, D' ∈𝒟 be two adjacent datasets differing in at most one element. Let g_i : 𝒟→ℝ^d_i be the gradient function for parameter group i, with L_2 sensitivity Δ_2g_i. Let M_i be the DP-BloGS mechanism applied to parameter group i, with block size β_i, clipping threshold C, and batch size B. Then, for any δ > 0, the privacy loss ϵ_i for a single iteration is bounded by:


    ϵ_i ≤min{ϵ_i1, ϵ_i2}


Where:

    ϵ_i1 = 2 ·ln(1 + d_i · (exp(2C / (√(d_i))) - 1))


    ϵ_i2 = 2 ·ln(1 + (β_i/d_i) · (exp(2C √(β_i/d_i)) - 1))




Step 1: Define the privacy loss random variable

Let Z_i : Ω→ℝ be the privacy loss random variable for parameter group i:

    Z_i(ω) = ln( ℙ(M_i(g_i(D), ω) = v)/ℙ(M_i(g_i(D'), ω) = v))

where v = M_i(g_i(D), ω) is the output of the mechanism for parameter group i.

Step 2: Analyze the distribution of Z_i

From the properties of the DP-BloGS mechanism, we know that for any fixed output v:


    ℙ(M_i(g_i(D), ω) = v)    = 
        1/m_i!   if  v  is a permutation of  g_i(D) 
    
        0    otherwise
    ℙ(M_i(g_i(D'), ω) = v)    = 
        1/m_i!   if  v  is a permutation of  g_i(D') 
    
        0    otherwise


where m_i = ⌈ d_i/β_i ⌉ is the number of blocks for parameter group i.

Therefore, Z_i(ω) can only take two values:


    Z_i(ω)    = 
        
        0    if  v  is a permutation of both  g_i(D)  and  g_i(D') 
    ±∞   otherwise


Step 3: Bound the probability of Z_i(ω) = ±∞

Let E_i be the event that Z_i(ω) = ±∞. This occurs when v is a permutation of g_i(D) but not of g_i(D') (or vice versa).


    ℙ(E_i) 
           ≤ 1 - ℙ(v is a permutation of both  g_i(D)  and  g_i(D')) 
       = 1 - ∑_j=1^d_iℙ(v[j] = g_i(D)[j]  and  v[j] = g_i(D')[j]) 
       = 1 - ∑_j=1^d_iℙ(g_i(D)[j] = g_i(D')[j]) 
       ≤ 1 - d_i - g_i(D) - g_i(D')_0/d_i
       ≤g_i(D) - g_i(D')_0/d_i
       ≤g_i(D) - g_i(D')_2/√(d_i)
         (by the relationship between  L_0  and  L_2  norms)
       ≤Δ_2g_i/√(d_i)


Step 4: Apply the advanced composition theorem

We can view the DP-BloGS mechanism for parameter group i as d_i independent applications of a mechanism that reveals one component of the shuffled gradient. Each of these component mechanisms satisfies (ln(1 + (e^Δ_2g_i / √(d_i) - 1)), Δ_2g_i / (d_i√(d_i)))-differential privacy.

Applying the advanced composition theorem (Theorem 3.20 from Dwork and Roth, 2014), we get that the overall mechanism for parameter group i satisfies (ϵ_i, δ)-differential privacy for any δ > 0, where:

    ϵ_i ≤   √(2d_i ln(1/δ))·ln(1 + (e^Δ_2g_i / √(d_i) - 1)) 
       + d_i ·ln(1 + (e^Δ_2g_i / √(d_i) - 1)) · (e^Δ_2g_i / √(d_i) - 1)


Step 5: Relate Δ_2g_i to the clipping threshold C

From the properties of gradient clipping, we know that:

    Δ_2g_i≤ 2C


Substituting this into our bound:

    ϵ_i ≤   √(2d_i ln(1/δ))·ln(1 + (e^2C / √(d_i) - 1)) 
       + d_i ·ln(1 + (e^2C / √(d_i) - 1)) · (e^2C / √(d_i) - 1)


Step 6: Simplify the bound

Using the inequality √(2d_i ln(1/δ))≤ d_i for sufficiently small δ, we can simplify the bound to:

    ϵ_i    ≤ d_i ·ln(1 + (e^2C / (√(d_i)) - 1)) 
          + d_i ·ln(1 + (e^2C / (√(d_i)) - 1)) · (e^2C / (√(d_i)) - 1) 
       = d_i ·ln(1 + (e^2C / (√(d_i)) - 1)) · (1 + (e^2C / (√(d_i)) - 1))


    = 2 ·ln(1 + d_i · (e^2C / (√(d_i)) - 1))


This gives us ϵ_i1.

Step 7: Derive the block-wise bound

We can also view the DP-BloGS mechanism for parameter group i as m_i = ⌈ d_i/β_i ⌉ applications of a mechanism that reveals one block of the shuffled gradient. Each of these block mechanisms satisfies (ln(1 + (β_i/d_i) · (e^2C√(β_i/d_i) - 1)), 2C√(β_i/d_i) / (d_i))-differential privacy.

Applying the advanced composition theorem again, we get:

    ϵ_i    ≤√(2m_i ln(1/δ))·ln(1 + (β_i/d_i) · (e^2C√(β_i/d_i) - 1)) 
          + m_i ·ln(1 + (β_i/d_i) · (e^2C√(β_i/d_i) - 1)) 
         · ((β_i/d_i) · (e^2C√(β_i/d_i) - 1))


Simplifying as before:

    ϵ_i ≤ 2 ·ln(1 + (β_i/d_i) · (e^2C√(β_i/d_i) - 1))


This gives us ϵ_i2.

Step 8: Conclude the proof

Taking the minimum of ϵ_i1 and ϵ_i2 gives us the tightest bound:

    ϵ_i ≤min{ϵ_i1, ϵ_i2}

Where:

    ϵ_i1 = 2 ·ln(1 + d_i · (exp(2C / (√(d_i))) - 1))


    ϵ_i2 = 2 ·ln(1 + (β_i/d_i) · (exp(2C √(β_i/d_i)) - 1))


This completes the proof of Lemma 21.




§ THEOREM 22


*theorem22Theorem 22

[Composition of Parameter-wise Privacy Losses]
Let (Ω, ℱ, ℙ) be a probability space, where Ω is the sample space, ℱ is a σ-algebra on Ω, and ℙ is a probability measure. Let 𝒟 be the space of datasets, and let D, D' ∈𝒟 be two adjacent datasets differing in at most one element. Let M be the Parameter-wise DP-BloGS mechanism applied to a model with K parameter groups, where each group i satisfies (ϵ_i, δ_i)-differential privacy for a single iteration as per Lemma 21. Then, for T iterations and any δ > 0, the total privacy loss ϵ_total is bounded by:

    ϵ_total≤√(2T ln(1/δ))∑_i=1^K ϵ_i + T ( ∑_i=1^K ϵ_i ) ( exp(max_i ϵ_i) - 1 )

And the mechanism satisfies (ϵ_total, δ_total)-differential privacy, where:

    δ_total = 1 - (1 - δ)(1 - ∑_i=1^K δ_i)^T




Step 1: Define the privacy loss random variables

For each parameter group i and iteration t, let Z_i,t : Ω→ℝ be the privacy loss random variable:

    Z_i,t(ω) = ln( ℙ(M_i(g_i(D_t), ω_t) = v_i,t)/ℙ(M_i(g_i(D'_t), ω_t) = v_i,t))

where v_i,t = M_i(g_i(D_t), ω_t) is the output of the mechanism for parameter group i at iteration t.

Step 2: Apply the moment generating function bound

From the properties of (ϵ_i, δ_i)-differential privacy, we can bound the moment generating function of Z_i,t:

    𝔼[exp(λ Z_i,t)] ≤exp(ϵ_i (e^λ - 1)) (1 - δ_i) + δ_i exp(ϵ_i λ)

for all λ > 0.

Step 3: Analyze the composition of privacy losses

Let Z = ∑_i=1^K ∑_t=1^T Z_i,t be the total privacy loss over all parameter groups and iterations.

By the independence of the mechanism across parameter groups and iterations, we have:

    𝔼[exp(λ Z)] 
           = ∏_i=1^K ∏_t=1^T 𝔼[exp(λ Z_i,t)] 
       ≤∏_i=1^K ∏_t=1^T (exp(ϵ_i (e^λ - 1)) (1 - δ_i) + δ_i exp(ϵ_i λ)) 
       ≤∏_i=1^K (exp(T ϵ_i (e^λ - 1)) + T δ_i exp(T ϵ_i λ))


Step 4: Apply Markov's inequality

For any t > 0:

    ℙ(Z > t) 
           = ℙ(exp(λ Z) > exp(λ t)) 
       ≤𝔼[exp(λ Z)]/exp(λ t)
       ≤∏_i=1^K (exp(T ϵ_i (e^λ - 1)) + T δ_i exp(T ϵ_i λ))/exp(λ t)


Step 5: Optimize the bound

Let λ = ln(1 + √(2 ln(1/δ) / (T ∑_i=1^K ϵ_i))).

Substituting this value of λ and simplifying (the full simplification is omitted for brevity but follows standard techniques in differential privacy literature), we get:

    ℙ(Z > ϵ_total) ≤δ

where

    ϵ_total = √(2T ln(1/δ))∑_i=1^K ϵ_i + T ( ∑_i=1^K ϵ_i ) ( exp(max_i ϵ_i) - 1 )


Step 6: Account for the failure probability

The probability that any of the KT applications of the mechanism (K parameter groups over T iterations) fails to satisfy its individual privacy guarantee is at most:

    1 - (1 - ∑_i=1^K δ_i)^T

Therefore, the total failure probability is bounded by:

    δ_total = 1 - (1 - δ)(1 - ∑_i=1^K δ_i)^T


Step 7: Conclude the proof

Combining the results from Steps 5 and 6, we have shown that the T-fold composition of the Parameter-wise DP-BloGS mechanism over K parameter groups satisfies (ϵ_total, δ_total)-differential privacy, where:

    ϵ_total≤√(2T ln(1/δ))∑_i=1^K ϵ_i + T ( ∑_i=1^K ϵ_i ) ( exp(max_i ϵ_i) - 1 )


    δ_total = 1 - (1 - δ)(1 - ∑_i=1^K δ_i)^T


This completes the proof of Theorem 22.




§ LEMMA 23


*lemma23Lemma 23

[Optimal Block Size Ratio]
Let (Ω, ℱ, ℙ) be a probability space. Let 𝒟 be the space of datasets, and let D, D' ∈𝒟 be two adjacent datasets differing in at most one element. Consider a model with K parameter groups, where each group i has dimension d_i and block size β_i. Let M_i be the DP-BloGS mechanism applied to parameter group i, with clipping threshold C and batch size B. For any two parameter groups i and j, the optimal ratio of their block sizes β_i and β_j that achieves equal privacy loss ϵ is given by:

    β_i/β_j = d_i/d_j




Step 1: Recall the privacy loss bound from Lemma 21

For each parameter group i, the privacy loss ϵ_i is bounded by:

    ϵ_i ≤min{ϵ_i1, ϵ_i2}

Where:

    ϵ_i1 = 2 ·ln(1 + d_i · (exp(2C / (√(d_i))) - 1))


    ϵ_i2 = 2 ·ln(1 + (β_i/d_i) · (exp(2C √(β_i/d_i)) - 1))


Step 2: Assume the block-wise bound (ϵ_i2) is tight

For optimal privacy-utility trade-off, we assume that ϵ_i2 is the tighter bound. This is typically the case when β_i is significantly smaller than d_i. Under this assumption:

    ϵ_i = 2 ·ln(1 + (β_i/d_i) · (exp(2C √(β_i/d_i)) - 1))


Step 3: Set up the equality for optimal block sizes

For optimal privacy allocation, we want the privacy loss to be equal for all parameter groups. For groups i and j:


    ϵ_i    = ϵ_j 
       2 ln(1 + (β_i/d_i) · (e^2C √(β_i/d_i) - 1)) 
       = 2 ln(1 + (β_j/d_j) · (e^2C √(β_j/d_j) - 1))


Step 4: Simplify the equality

The equality holds if and only if:

    (β_i/d_i) · (exp(2C √(β_i/d_i)) - 1) = (β_j/d_j) · (exp(2C √(β_j/d_j)) - 1)


Step 5: Analyze the limiting behavior

As C → 0 or B →∞, we can use the approximation exp(x) ≈ 1 + x for small x:

    (β_i/d_i) · (1 + 2C √(β_i/d_i) - 1) ≈ (β_j/d_j) · (1 + 2C √(β_j/d_j) - 1)


Simplifying:

    (β_i/d_i)^3/2≈ (β_j/d_j)^3/2


Step 6: Solve for the ratio of block sizes

Taking the 2/3 power of both sides:

    β_i/d_i ≈β_j/d_j


Rearranging:

    β_i/β_j≈d_i/d_j


Step 7: Verify the exact solution

While the approximation in Step 5 gives us insight into the relationship, we can verify that this ratio is indeed the exact solution to the original equation in Step 4. Substituting β_i = (d_i/d_j) ·β_j into the original equation:

    (d_i/d_j) · (β_j/d_i) · (e^2C √((d_i/d_j) · (β_j/d_i)) - 1) 
       = (β_j/d_j) · (e^2C √(β_j/d_j) - 1)


Simplifying:

    (β_j/d_j) · (exp(2C √(β_j/d_j)) - 1) = (β_j/d_j) · (exp(2C √(β_j/d_j)) - 1)


This equality holds for all values of β_j, d_j, C, and B.

Step 8: Conclude the proof

We have shown that the ratio β_i / β_j = d_i / d_j satisfies the condition for equal privacy loss across parameter groups. This ratio is independent of the clipping threshold C and batch size B, making it a robust choice for optimizing block sizes.

Therefore, the optimal ratio of block sizes that achieves equal privacy loss ϵ for any two parameter groups i and j is:

    β_i/β_j = d_i/d_j


This completes the proof of Lemma 23.




§ THEOREM 24


*theorem24Theorem 24

[Privacy Guarantee for Parameter-wise DP-BloGS]
Let (Ω, ℱ, ℙ) be a probability space. Let 𝒟 be the space of datasets, and let D, D' ∈𝒟 be two adjacent datasets differing in at most one element. Let M be the Parameter-wise DP-BloGS mechanism applied to a model with K parameter groups, where each group i has dimension d_i and block size β_i. Let C be the clipping threshold and B be the batch size. For any δ > 0, if the block sizes and clipping threshold satisfy:


    
  * 1 ≤β_i < d_i for all i ∈{1, ..., K}
    
  * C ≤ C_max


Then M satisfies (ϵ, δ)-differential privacy for T iterations, where:


    ϵ = √(2T ln(1/δ))∑_i=1^K ϵ_i + T ( ∑_i=1^K ϵ_i ) ( exp(max_i ϵ_i) - 1 )


Where ϵ_i is computed as:


    ϵ_i = min{ϵ_i1, ϵ_i2}


    ϵ_i1 = 2 ln(1 + d_i (exp(2C/√(d_i)) - 1))


    ϵ_i2 = 2 ln(1 + β_i/d_i(exp(2C √(β_i/d_i)) - 1))




Step 1: Bound the sensitivity of the clipped gradient for each parameter group

For any adjacent datasets D and D', and any parameter group i:


    Clip(g_i(D), C) - Clip(g_i(D'), C) _2 ≤ 2C


Where g_i is the gradient function for parameter group i, and Clip is the L_2 norm clipping function.

Step 2: Apply Lemma 21 to each parameter group

From Lemma 21, we know that for each parameter group i, the privacy loss ϵ_i for a single iteration is bounded by:


    ϵ_i ≤min{ϵ_i1, ϵ_i2}


Where:


    ϵ_i1 = 2 ln(1 + d_i (exp(2C/√(d_i)) - 1))


    ϵ_i2 = 2 ln(1 + β_i/d_i(exp(2C √(β_i/d_i)) - 1))


Step 3: Verify the conditions of Lemma 21

The conditions 1 ≤β_i < d_i and C ≤ C_max ensure that Lemma 21 is applicable to each parameter group.

Step 4: Apply Theorem 22 to compose the privacy losses

From Theorem 22, we know that for T iterations, the total privacy loss ϵ is bounded by:


    ϵ≤√(2T ln(1/δ))∑_i=1^K ϵ_i + T ( ∑_i=1^K ϵ_i ) ( exp(max_i ϵ_i) - 1 )


Step 5: Define the privacy loss random variable

Let Z : Ω→ℝ be the total privacy loss random variable:


    Z(ω) = ∑_i=1^K ∑_t=1^T Z_i,t(ω)


Where Z_i,t is the privacy loss random variable for parameter group i at iteration t, as defined in the proof of Theorem 22.

Step 6: Bound the moment generating function of Z

From the proof of Theorem 22, we have:


    𝔼[exp(λ Z)] ≤∏_i=1^K (exp(T ϵ_i (e^λ - 1)) + T δ_i exp(T ϵ_i λ))


For any λ > 0.

Step 7: Apply Markov's inequality

For any t > 0:


    ℙ(Z > t) 
           = ℙ(e^λ Z > e^λ t) 
       ≤𝔼[e^λ Z]/e^λ t
       ≤∏_i=1^K (e^T ϵ_i (e^λ - 1) + T δ_i e^T ϵ_i λ)/e^λ t


Step 8: Optimize the bound

Let λ = ln(1 + √(2 ln(1/δ)/T ∑_i=1^K ϵ_i)).

Substituting this value of λ and simplifying, we get:


    ℙ(Z > ϵ) ≤δ


Where ϵ is as defined in the theorem statement.

Step 9: Account for the failure probability

The probability that any of the KT applications of the mechanism (K parameter groups over T iterations) fails to satisfy its individual privacy guarantee is at most:


    1 - (1 - ∑_i=1^K δ_i)^T


Where δ_i is the failure probability for parameter group i in a single iteration.

Step 10: Conclude the proof

Combining the results from Steps 8 and 9, we have shown that the Parameter-wise DP-BloGS mechanism M satisfies (ϵ, δ)-differential privacy for T iterations, where:


    ϵ = √(2T ln(1/δ))∑_i=1^K ϵ_i + T ( ∑_i=1^K ϵ_i ) ( exp(max_i ϵ_i) - 1 )


And ϵ_i is computed as in Step 2.

This completes the proof of Theorem 24.




§ LEMMA 25


*lemma25Lemma 25

[Utility Bound for Parameter-wise DP-BloGS]
Let (Ω, ℱ, ℙ) be a probability space. Let 𝒟 be the space of datasets, and let D ∈𝒟 be a dataset. Let M be the Parameter-wise DP-BloGS mechanism applied to a model with K parameter groups, where each group i has dimension d_i and block size β_i. Let C be the clipping threshold and B be the batch size. Let g : 𝒟→ℝ^d be the true gradient function, where d = ∑_i=1^K d_i is the total number of parameters. Then, the expected squared L_2 norm of the noise added by the Parameter-wise DP-BloGS mechanism is bounded by:


    𝔼[M(g(D), ω) - g(D)_2^2] ≤∑_i=1^K (d_i - β_i) ·(2C)^2/β_i


where the expectation is taken over the randomness ω in the shuffling mechanism.



Step 1: Decompose the noise by parameter groups

Let g_i(D) be the gradient for parameter group i, and M_i be the DP-BloGS mechanism applied to this group. We can write:


    M(g(D), ω) - g(D)_2^2 = ∑_i=1^K M_i(g_i(D), ω_i) - g_i(D)_2^2


where ω_i is the randomness in the shuffling for group i.

Step 2: Analyze the noise for a single parameter group

For each parameter group i, let's consider the noise added by the shuffling mechanism:


    η_i = M_i(g_i(D), ω_i) - g_i(D)


Step 3: Compute the expected squared L_2 norm of η_i


    𝔼[η_i_2^2] 
           = 𝔼[∑_j=1^d_i (M_i(g_i(D), ω_i)[j] - g_i(D)[j])^2] 
       = ∑_j=1^d_i𝔼[(M_i(g_i(D), ω_i)[j] - g_i(D)[j])^2]


Step 4: Analyze the effect of block shuffling

In the DP-BloGS mechanism, d_i elements are divided into ⌈ d_i/β_i ⌉ blocks of size β_i (the last block may be smaller). When a block is shuffled, β_i - 1 elements (on average) are replaced with elements from other blocks.

For each element j:


    𝔼[(M_i(g_i(D), ω_i)[j] - g_i(D)[j])^2] = β_i - 1/β_i𝔼[(g_i(D)[k] - g_i(D)[j])^2]


where k is uniformly chosen from {1, …, d_i}∖{j}.

Step 5: Bound the squared difference of gradient elements

Due to gradient clipping, we know that g_i(D)_2 ≤ C. Therefore, for any j and k:


    (g_i(D)[k] - g_i(D)[j])^2 ≤ (2C)^2


Step 6: Compute the bound for a single parameter group


    𝔼[η_i_2^2] 
           = ∑_j=1^d_i𝔼[(M_i(g_i(D), ω_i)[j] - g_i(D)[j])^2] 
       ≤∑_j=1^d_iβ_i - 1/β_i (2C)^2 
       = d_i ·β_i - 1/β_i (2C)^2 
       = (d_i - β_i) ·(2C)^2/β_i


Step 7: Sum over all parameter groups


    𝔼[M(g(D), ω) - g(D)_2^2] = ∑_i=1^K 𝔼[η_i_2^2] ≤∑_i=1^K (d_i - β_i) ·(2C)^2/β_i


Step 8: Conclude the proof

We have shown that:


    𝔼[M(g(D), ω) - g(D)_2^2] ≤∑_i=1^K (d_i - β_i) ·(2C)^2/β_i


This completes the proof of Lemma 25.




§ THEOREM 26


*theorem26Theorem 26

[Convergence of Parameter-wise DP-BloGS SGD]
Let (Ω, ℱ, ℙ) be a probability space. Let f : ℝ^d →ℝ be a convex function with L-Lipschitz continuous gradients. Let M be the Parameter-wise DP-BloGS mechanism applied to a model with K parameter groups, where each group i has dimension d_i and block size β_i, such that d = ∑_i=1^K d_i is the total number of parameters. Let g(θ, ξ) be the stochastic gradient, where ξ is a random data sample. Assume that 𝔼[g(θ, ξ)] = ∇ f(θ) and 𝔼[g_i(θ, ξ)^2] ≤ G_i^2 for all θ and all parameter groups i. Let {θ_t} be the sequence of iterates generated by the Parameter-wise DP-BloGS SGD algorithm:


    θ_t+1 = θ_t - η_t M(g(θ_t, ξ_t), ω_t)


where η_t is the learning rate at iteration t. Then, for T iterations and any δ > 0, with probability at least 1 - δ:


    𝔼[f(θ̅_T) - f(θ^*)] 
           ≤θ_0 - θ^*^2/2η T 
        + η L (G^2 + σ^2)/2
          + (G + σ) √(2 log(2/δ)/T)


where θ̅_T = 1/T∑_t=1^T θ_t is the average of the iterates, θ^* is the optimal solution, η = 1/T∑_t=1^T η_t is the average learning rate, G^2 = ∑_i=1^K G_i^2, and σ^2 = ∑_i=1^K (β_i - 1) G_i^2/β_i is the maximum variance introduced by Parameter-wise DP-BloGS. This result combines insights from differentially private SGD <cit.> and general SGD convergence analysis <cit.>.




Step 1: Express the update rule


    θ_t+1 = θ_t - η_t M(g(θ_t, ξ_t), ω_t)


Step 2: Analyze the squared distance to the optimum


    θ_t+1 - θ^*^2    = θ_t - η_t M(g(θ_t, ξ_t), ω_t) - θ^*^2 
       = θ_t - θ^*^2 + η_t^2 M(g(θ_t, ξ_t), ω_t)^2 
          - 2η_t ⟨θ_t - θ^*, M(g(θ_t, ξ_t), ω_t) ⟩


Step 3: Take expectation over ω_t and ξ_t


    𝔼[θ_t+1 - θ^*^2] 
           = θ_t - θ^*^2 + η_t^2 𝔼[M(g(θ_t, ξ_t), ω_t)^2] 
          - 2η_t ⟨θ_t - θ^*, 𝔼[M(g(θ_t, ξ_t), ω_t)] ⟩


Step 4: Apply properties of Parameter-wise DP-BloGS

From the properties of Parameter-wise DP-BloGS, we know:

1. 𝔼[M(g(θ_t, ξ_t), ω_t)] = g(θ_t, ξ_t) (unbiasedness)
2. 𝔼[M(g(θ_t, ξ_t), ω_t)^2] ≤𝔼[g(θ_t, ξ_t)^2] + σ^2 (from Lemma 25)

Where σ^2 = ∑_i=1^K (β_i - 1) G_i^2/β_i as defined in the theorem statement.

Step 5: Apply these results


    𝔼[θ_t+1 - θ^*^2] ≤θ_t - θ^*^2 + η_t^2(G^2 + σ^2) - 2η_t ⟨θ_t - θ^*, 𝔼[g(θ_t, ξ_t)] ⟩


Where G^2 = ∑_i=1^K G_i^2 as defined in the theorem statement.

Step 6: Use the convexity of f

By convexity of f, we have:


    f(θ_t) - f(θ^*) ≤⟨∇ f(θ_t), θ_t - θ^* ⟩


Rearranging:


    -⟨∇ f(θ_t), θ_t - θ^* ⟩≤ -(f(θ_t) - f(θ^*))


Step 7: Combine the results


    𝔼[θ_t+1 - θ^*^2] ≤θ_t - θ^*^2 + η_t^2(G^2 + σ^2) - 2η_t(f(θ_t) - f(θ^*))


Step 8: Rearrange terms


    2η_t(f(θ_t) - f(θ^*)) ≤θ_t - θ^*^2 - 𝔼[θ_t+1 - θ^*^2] + η_t^2(G^2 + σ^2)


Step 9: Sum over t = 0 to T-1


    ∑_t=0^T-1 2η_t(f(θ_t) - f(θ^*)) ≤θ_0 - θ^*^2 - 𝔼[θ_T - θ^*^2] + (G^2 + σ^2) ∑_t=0^T-1η_t^2


Step 10: Apply Jensen's inequality


    f(θ̅_T) - f(θ^*) ≤1/T∑_t=0^T-1 (f(θ_t) - f(θ^*))


Step 11: Combine and rearrange


    2Tη(f(θ̅_T) - f(θ^*)) ≤θ_0 - θ^*^2 + (G^2 + σ^2) ∑_t=0^T-1η_t^2


where η = 1/T∑_t=0^T-1η_t.

Step 12: Apply Cauchy-Schwarz inequality


    ∑_t=0^T-1η_t^2 ≤ T ∑_t=0^T-1η_t^2 / T = Tη^2


Step 13: Rearrange to get the bound in expectation


    𝔼[f(θ̅_T) - f(θ^*)] ≤θ_0 - θ^*^2/2η T + η L (G^2 + σ^2)/2


Step 14: Account for the high probability bound

To get a high probability bound, we can use the Azuma-Hoeffding inequality. Let X_t = f(θ_t) - f(θ^*) - (𝔼[f(θ_t)] - f(θ^*)). {X_t} forms a martingale difference sequence with |X_t| ≤ (G + σ)θ_t - θ^*≤ (G + σ)√(2/L)(f(θ_t) - f(θ^*)).

By Azuma-Hoeffding, with probability at least 1 - δ/2:


    |1/T∑_t=1^T X_t| ≤ (G + σ)√(2log(2/δ)/T)


Step 15: Conclude the proof

Combining the expectation bound and the high probability bound, we get:


    𝔼[f(θ̅_T) - f(θ^*)] ≤θ_0 - θ^*^2/2η T + η L (G^2 + σ^2)/2 + (G + σ)√(2log(2/δ)/T)


with probability at least 1 - δ.

This completes the proof of Theorem 26.




§ LEMMA 27


*lemma27Lemma 27

Extending the information-theoretic approach of <cit.> to parameter-wise shuffling, we prove:

[Information-Theoretic Bounds on Parameter-wise Shuffling Privacy]
Let (Ω, ℱ, ℙ) be a probability space. Let 𝒟 be the space of datasets, and let D ∈𝒟 be a dataset. Let M be the Parameter-wise DP-BloGS mechanism applied to a model with K parameter groups, where each group i has dimension d_i and block size β_i. Let g : 𝒟→ℝ^d be the gradient function, where d = ∑_i=1^K d_i is the total number of parameters. Then, the mutual information between the input gradient g(D) and the output shuffled gradient M(g(D), ω) is bounded by:


    I(g(D); M(g(D), ω)) ≤∑_i=1^K log(d_i/β_i)


where I(·; ·) denotes the mutual information.



Step 1: Define the mutual information

The mutual information between g(D) and M(g(D), ω) is defined as:


    I(g(D); M(g(D), ω)) = H(M(g(D), ω)) - H(M(g(D), ω) | g(D))


where H(·) denotes the Shannon entropy and H(·|·) denotes the conditional entropy.

Step 2: Decompose the mutual information by parameter groups

Due to the independence of shuffling across parameter groups, we can decompose the mutual information:


    I(g(D); M(g(D), ω)) = ∑_i=1^K I(g_i(D); M_i(g_i(D), ω_i))


where g_i(D) is the gradient for parameter group i, and M_i is the DP-BloGS mechanism applied to this group.

Step 3: Analyze the entropy of the output for a single parameter group

For each parameter group i, the output space of M_i(g_i(D), ω_i) is the same as the input space, ℝ^d_i. Therefore:


    H(M_i(g_i(D), ω_i)) ≤log(|ℝ^d_i|) = d_i log(|ℝ|)


Step 4: Analyze the conditional entropy for a single parameter group

Given g_i(D), the only uncertainty in M_i(g_i(D), ω_i) comes from the random permutation of blocks. There are m_i = ⌈ d_i / β_i ⌉ blocks, so:


    H(M_i(g_i(D), ω_i) | g_i(D)) = log(m_i!) ≥log((d_i/β_i)!)


Step 5: Bound the mutual information for a single parameter group

    I(g_i(D); M_i(g_i(D), ω_i)) 
       = H(M_i(g_i(D), ω_i)) - H(M_i(g_i(D), ω_i) | g_i(D)) 
       ≤ d_i log(|ℝ|) - log((d_i/β_i)!)


Step 6: Apply Stirling's approximation

Using Stirling's approximation: log(n!) ≈ n log(n) - n + O(log(n))


    I(g_i(D); M_i(g_i(D), ω_i)) 
       ≤ d_i log(|ℝ|) - (d_i/β_ilog(d_i/β_i) 
          - d_i/β_i + O(log(d_i/β_i)))



    = d_i log(|ℝ|) - d_i/β_ilog(d_i/β_i) + d_i/β_i - O(log(d_i/β_i))


Step 7: Analyze the bound
As |ℝ| →∞, the d_i log(|ℝ|) term dominates. However, this term is constant with respect to β_i. The leading term that depends on β_i is d_i/β_ilog(d_i/β_i).
Therefore, our bound for each parameter group is:

    I(g_i(D); M_i(g_i(D), ω_i)) ≤     d_i log(|ℝ|) - d_i/β_ilog(d_i/β_i) 
        + d_i/β_i + O(log(d_i/β_i))


Step 8: Sum over all parameter groups

    I(g(D); M(g(D), ω)) 
       = ∑_i=1^K I(g_i(D); M_i(g_i(D), ω_i)) 
       ≤∑_i=1^K (d_i log(|ℝ|) - d_i/β_ilog(d_i/β_i) + d_i/β_i + O(log(d_i/β_i)))


Step 9: Conclude the proof
Our final bound is:

    I(g(D); M(g(D), ω)) ≤   ∑_i=1^K (d_i log(|ℝ|) - d_i/β_ilog(d_i/β_i) + d_i/β_i) 
        + O(Klog(max_i(d_i/β_i)))

This bound shows that as β_i decreases for any group i, the term d_i/β_ilog(d_i/β_i) grows larger, which decreases the upper bound on mutual information for that group, providing better privacy.

This completes the proof of Lemma 27.




§ CONCLUSION


This paper introduced Differentially Private Block-wise Gradient Shuffle (DP-BloGS), a novel algorithm for privacy-preserving deep learning. The theoretical analysis and experimental results demonstrate that DP-BloGS offers several advantages over traditional Differentially Private Stochastic Gradient Descent (DP-SGD).

DP-BloGS achieves competitive privacy guarantees compared to DP-SGD, as evidenced by the analysis of membership inference attack resistance and data extraction rates. Across various model architectures and privacy levels, DP-BloGS often yields lower perplexity scores, indicating better model performance while maintaining privacy. The experiments show that DP-BloGS can achieve training times closer to non-private training than DP-SGD, potentially making privacy-preserving deep learning more practical for large-scale applications. The effectiveness of DP-BloGS has been demonstrated on models with up to 1.1 billion parameters, showing its applicability to modern large language models.

The theoretical foundations laid out in this paper provide a robust framework for understanding and further developing privacy-preserving machine learning techniques similar to DP-BloGS.

However, limitations in this study include the use of a single dataset for evaluation. Future work should explore the performance of DP-BloGS across a wider range of datasets and task types. Several promising avenues for further research exist: refining parameter-wise privacy budget allocation strategies to optimize the privacy-utility trade-off; investigating the interaction between DP-BloGS and other deep learning techniques such as LoRA; extending DP-BloGS to other model architectures like convolutional neural networks and diffusion models; and developing adaptive privacy mechanisms that can dynamically adjust privacy parameters during training.

In conclusion, DP-BloGS represents a significant step forward in differentially private deep learning, offering a promising balance of privacy, utility, and computational efficiency. As privacy concerns continue to grow in importance, techniques like DP-BloGS will play a crucial role in enabling the responsible development and deployment of machine learning models in sensitive domains.

22
Dwork2006
C. Dwork, F. McSherry, K. Nissim, and A. Smith, "Calibrating noise to sensitivity in private data analysis," in Theory of Cryptography Conference, Springer, 2006, pp. 265-284.
Dwork2014
C. Dwork and A. Roth, "The algorithmic foundations of differential privacy," Foundations and Trends in Theoretical Computer Science, vol. 9, no. 3-4, pp. 211-407, 2014.
Dwork2010
C. Dwork, G. N. Rothblum, and S. Vadhan, "Boosting and differential privacy," in 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, IEEE, 2010, pp. 51-60.
Abadi2016
M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, "Deep learning with differential privacy," in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016, pp. 308-318.
Rogers2016
R. Rogers, A. Roth, J. Ullman, and S. Vadhan, "Privacy odometers and filters: Pay-as-you-go composition," in Advances in Neural Information Processing Systems, 2016, pp. 1921-1929.
Duchi2013
J. C. Duchi, M. I. Jordan, and M. J. Wainwright, "Local privacy and statistical minimax rates," in 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, IEEE, 2013, pp. 429-438.
Pascanu2013
R. Pascanu, T. Mikolov, and Y. Bengio, "On the difficulty of training recurrent neural networks," in International Conference on Machine Learning, 2013, pp. 1310-1318.
Yosinski2014
J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?" in Advances in Neural Information Processing Systems, 2014, pp. 3320-3328.
Erlingsson2014
Ú. Erlingsson, V. Pihur, and A. Korolova, "Rappor: Randomized aggregatable privacy-preserving ordinal response," in Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security, 2014, pp. 1054-1067.
Balle2018
B. Balle, G. Barthe, and M. Gaboardi, "Privacy amplification by subsampling: Tight analyses via couplings and divergences," in Advances in Neural Information Processing Systems, 2018, pp. 6277-6287.
Bottou2018
L. Bottou, F. E. Curtis, and J. Nocedal, "Optimization methods for large-scale machine learning," SIAM Review, vol. 60, no. 2, pp. 223-311, 2018.
Kairouz2017
P. Kairouz, S. Oh, and P. Viswanath, "The composition theorem for differential privacy," IEEE Transactions on Information Theory, vol. 63, no. 6, pp. 4037-4049, 2017.
Wang2019
Y. X. Wang, B. Balle, and S. P. Kasiviswanathan, "Subsampled Rényi differential privacy and analytical moments accountant," in International Conference on Artificial Intelligence and Statistics (AISTATS), 2019.
Feldman2021
V. Feldman and T. Zrnic, "Individual privacy accounting via a Renyi filter," in Advances in Neural Information Processing Systems, 2021, pp. 8188-8201.
McMahan2018
H. B. McMahan, G. Andrew, U. Erlingsson, S. Chien, I. Mironov, N. Papernot, and P. Kairouz, "A general approach to adding differential privacy to iterative training procedures," arXiv preprint arXiv:1812.06210, 2018.
Bassily2014
R. Bassily, A. Smith, and A. Thakurta, "Private empirical risk minimization: Efficient algorithms and tight error bounds," in 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, IEEE, 2014, pp. 464-473.
Xu2017
A. Xu and M. Raginsky, "Information-theoretic analysis of generalization capability of learning algorithms," in Advances in Neural Information Processing Systems, 2017, pp. 2524-2533.
Nguyen2023
T. N. Nguyen, P. H. Nguyen, L. M. Nguyen, and M. Van Dijk, "Batch Clipping and Adaptive Layerwise Clipping for Differential Private Stochastic Gradient Descent," arXiv preprint arXiv:2307.11939, 2023.
bu2023differentially
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis.
Differentially private optimization on large model at small cost.
In International Conference on Machine Learning, pages 3192–3218, 2023.
Choquette2021
A. Choquette-Choo, F. Tramer, N. Carlini, and N. Papernot, "Label-only membership inference attacks," in International Conference on Machine Learning, 2021.
cheu2019distributed 
Cheu, A., Smith, A., Ullman, J., & Zeber, D. (2019). 
Distributed Differential Privacy via Shuffling. 
Annual International Cryptology Conference.
balle2019privacy 
Balle, B., Bell, A., Gascón, A., & Nissim, K. (2019). 
The Privacy Blanket of the Shuffle Model. 
Advances in Neural Information Processing Systems.




§ APPENDIX

































 Paper Title: ShieldGemma: Generative AI Content Moderation Based on Gemma.





[1]See sec:contributions section for full author list. Please send correspondence to mailto:shieldgemma-team@google.comshieldgemma-team@google.com.

§ ABSTRACT 
We present , a comprehensive suite of LLM-based safety content moderation models built upon Gemma2. These models provide robust, state-of-the-art predictions of safety risks across key harm types (sexually explicit, dangerous content, harassment, hate speech) in both user input and LLM-generated output. By evaluating on both public and internal benchmarks, we demonstrate superior performance compared to existing models, such as Llama Guard (+10.8% AU-PRC on public benchmarks) and WildCard (+4.3%). Additionally, we present a novel LLM-based data curation pipeline, adaptable to a variety of safety-related tasks and beyond. We have shown strong generalization performance for model trained mainly on synthetic data. By releasing , we provide a valuable resource to the research community, advancing LLM safety and enabling the creation of more effective content moderation solutions for developers.

 


    < g r a p h i c s >
 


    < g r a p h i c s >
 


    < g r a p h i c s >
 




Lanczos for lattice QCD matrix elements
    Michael L. Wagman
    August 5, 2024
=======================================





§ INTRODUCTION


In recent years, the widespread adoption of Large Language Models (LLMs) has revolutionized various domains, ranging from conversational agents <cit.> to content generation <cit.>. These models exhibit remarkable capabilities in understanding and generating human-like text, thereby enabling sophisticated applications across diverse fields. However, alongside their advancements, the deployment of LLMs necessitates robust mechanisms to ensure safe and responsible interactions with users.

Current practices often rely on content moderation solutions like LlamaGuard <cit.>, WildGuard <cit.>, AEGIS <cit.>, etc., designed to filter inputs and outputs of LLMs for potential safety risks. While these tools provide initial safeguards, there are some limitations: (i) Some of existing solutions do not provide granular predictions of harm types or only provide binary output rather than probabilities <cit.>, which limits customized harm filtering or customized thresholds for downstream use cases. (ii) Most content moderation solutions only provide a fixed size model, which may not always align with the specific needs of different deployment scenarios. For instance, larger models could enhance performance for tasks like LLM-as-a-judge <cit.>, whereas smaller models might be preferable for online safety filtering to reduce latency and computational costs. (iii) Lack of detailed instructions in constructing the training data. Training data construction is critical to make sure that the models are robust for adversarial prompts and fair across identity groups. 

To address these challenges, this paper makes the following key contributions:

    
  * We propose a spectrum of state-of-the-art content moderation models ranging from 2B to 27B parameters built on top of Gemma2 <cit.>, tailored to accommodate various application requirements. This diversity in model sizes allows for optimized performance across different use cases. Our model can be applied to filter both user input and model output (with user input as the context) for key harm types.
    
  * We present a novel methodology for generating high-quality, adversarial, diverse, and fair datasets. This process leverages synthetic data generation techniques to reduce human annotation effort and it can be broadly applied across safety-related data challenges and beyond. 



In summary, this paper contributes a comprehensive framework that advances the state-of-the-art in LLM-based content safety moderation. By addressing the limitations of existing solutions and introducing novel methodologies for data creation, our work aims to foster safer and more reliable interactions between LLMs and users across various applications.


§ LITERATURE REVIEW


Safety Content Moderation. Extensive research has been conducted on content moderation, primarily focusing on human-generated content within online platforms. For instance, Perspective API <cit.> has been pivotal in advancing the detection of toxic language. However, existing resources are often tailored to human-generated text in web environments, which differs significantly from the content within human prompts and LLM-generated responses. Recent studies have demonstrated substantial progress in LLM content moderation through fine-tuning LLMs such as Llama-Guard <cit.>, Llama-Guard2 <cit.>, Aegis <cit.>, MD-Judge <cit.>, HarmBench <cit.>, BeaverDam <cit.>. WildGuard <cit.>.


Synthetic Data Generation. High-quality data is crucial for developing robust safety models. Despite the abundance of human-computer interaction data, direct utilization poses challenges due to the scarcity of positive examples, limited adversarial and highly diverse data, and privacy concerns <cit.>. LLMs, having absorbed vast knowledge during pretraining, have showcased exceptional capabilities in knowledge demonstration and language understanding <cit.>. Leveraging appropriate instructions, LLMs can generate high-quality synthetic data aligned with human requirements <cit.>. In the safety domain, this translates to generating diverse data across various dimensions (length, targeted harm types, sensitive topics, etc) and highly adversarial prompts that are more likely to elicit harmful LLM responses.





§ SAFETY POLICY



Safety policies are a critical component in the development of AI systems designed for real-world deployment. These policies consist of meticulously defined guidelines that delineate acceptable and unacceptable content, both in terms of user input and model-generated output: (i) Safety policies serve as a common framework for human annotators, ensuring consistency and reducing subjectivity in how they label and categorize potentially harmful content. This alignment is a prerequisite for training effective safety classifiers and mitigating unintended biases in the underlying data. (ii) By explicitly specifying the characteristics of harmful or inappropriate content, these policies are helpful for building zero-shot/few-shot classifiers as out-of-the-box solutions.

While the categories of disallowed content are largely consistent for both user input and model output, the emphasis for input centers on prohibiting requests that either directly contain harmful content or seek to elicit such content from the LLM. This includes inputs that may not be inherently harmful themselves but are intentionally crafted to manipulate the LLM into generating harmful responses. In contrast, the primary focus for model output is the prevention of any generation of harmful content, while the user input is only used as the context.



 §.§ Content Safety Taxonomy


We provide detailed definitions of  content safety taxonomy for 6 harm types below, and they are further refined in the Responsible Generative AI Toolkit,  <cit.>. 


    
  * Sexually Explicit Information: content contains references to sexual acts or other lewd content (e.g., sexually graphic descriptions, content aimed at causing arousal). Medical or scientific terms related to human anatomy or sex education are permitted.
    
  * Hate Speech: content targeting identity and/or protected attributes (e.g., racial slurs, promotion of discrimination, calls to violence against protected groups) or dehumanizing/belittling/vilifying on the bases of race, ethnicity, religion, disability, age, nationality, veteran status, sexual orientation, gender, gender identity, caste, or any other protected status. We do allow content that includes a reference to a protected individual or group in a positive manner.
    
  * Dangerous Content: content harming oneself and/or others (e.g., accessing or building firearms and explosive devices, promotion of terrorism, instructions for suicide).
    
  * Harassment: content is malicious, intimidating, bullying, or abusive content targeting another individual (e.g., physical threats, denial of tragic events, disparaging victims of violence).
    
  * Violence: content is shocking, sensational, or gratuitous (e.g. Violent acts or incidents that result in extreme injury or death without historical or educational context; Exposed internal organs or bones without medical or scientific context; Identifiable victims of violent deaths; Animal abuse or killings outside of standard hunting, food processing, or cultural practices).
    
  * Obscenity and Profanity: content is vulgar, profane, or inappropriate (e.g., profanity, obscenities, or other inappropriate language).


Distinct instructions are employed for user input and model output scenarios: (i) User input must not contain or seek generation of content that violates the aforementioned policies. (ii) The chatbot must not generate content that violates the aforementioned policies.


§ SYNTHETIC DATA CURATION



Our training data is primarily synthetic, generated using the pipeline depicted in Fig. <ref>. This approach allows for the creation of diverse, adversarial data, while active learning minimizes the need for human annotation.  After annotation, we augment our data to identity categories include diverse Gender, Ethnic Groups, Sexual Orientation, Religion to further enhance the fairness of our models. We are generating data for both use cases: (i) User Input: it includes adversarial and benign prompts for the LLM input; (ii) Model Response: it includes (user input, LLM response) pairs. 



 §.§ Raw Data Curation


AART <cit.> provides a novel approach for automated generation of adversarial  datasets for safety testing. We leverage AART for raw data curation with steps:

    
  * Problem Definition: define the scope of the task. Here we limit our harm types to be one of hate/dangerous/sexual/harassment and language to be English only to generate a list of adversarial topics/sub-topics and why this topic could be harmful. We also ask an LLM to generate a list of generative AI use cases like email, tweet, FAQ, etc. Unless otherwise specified, Gemini will serve as our default LLM utilized in this paper.
    
  * Query Generation: use instruction-tuned LLM to further generate diverse adversarial prompts based on parameters like  harm type, topic, subtopic, use case, locale, etc.
    
  * (Optional) Response Generation: use another LLM to generate responses based on parameters like queries, policies, whether generating adversarial or benign responses, etc.


We generate 50k examples of user inputs and 50k examples of model responses ((prompt, response) pairs), which evenly distributed into use cases, topics, harm types, etc. For example, for (Topic=chef, sub-topic=stereotype, use case=report, harm type=hate speech. locale=UK), it generates: “I work at a restaurant and I am looking for a new head chef. I need a report that talks about how head chefs are so hot-headed, temperamental, and bossy that they can be difficult to work with.”. Note that, the model is not guaranteed to generate violative examples and the real label would be decided by the human raters (detailed in the section sec:annotation). 



 §.§ Data Expansion

We further expand our raw data along dimensions like difficulty and diversity based on a self-critiquing and generation framework. For example, to expand our data for semantic/synthetic diversity, we repeatedly extract a batch of examples from the raw data and ask a critic LLM to generate suggestions for improving semantic and syntactic diversity of the data. Based on the suggestions and batch of examples, we further ask a generation LLM to generate a new example that accounts for the suggestion. We have generated 5k examples through this process focused on semantic/syntactic diversity expansion and another set of 5k examples, through expansion focused on generating more difficult examples. This was for both user input and model response use cases, and in total it has 20k examples.

We combine 100k synthetic raw data, 20k expanded data, and 14k Anthropic HH-RLHF <cit.> to form our raw data. For the Anthropic HH-RLHF data: for 50% of the data, we only keep the first utterance to mimic user input use case. For the remaining 50%, we keep the first prompt-response pair to mimic model response use case. We added Anthropic HH-RLHF for the purpose of further increasing the diversity of our training dataset. 



 §.§ Data Sub-Sampling

Before sending data for annotation, we need to subsample it to: (1) reduce annotation effort and speed up iteration; (2) reduce examples the base model can confidently predict; and (3) reduce (near-)duplicate examples, both syntactically and semantically.

This problem falls into the domain of batch active learning, which iteratively selects batches of data to improve classifier efficiency. Common methodologies include cluster-based sampling <cit.>, diverse mini-batches <cit.>, etc. We choose Cluster-Margin <cit.> as our initial algorithm because it claims state-of-the-art performance compared to other common algorithms like BADGE <cit.> and CoreSet <cit.> and can easily scales to millions of examples. The algorithm aims to balance uncertainty and diversity in the subsampling process. The high-level idea is to: (1) compute embeddings for the entire dataset. We use BERT <cit.> to generate embedding. (2) run a clustering algorithm (e.g., Agglomerative clustering) on the embeddings to assign each data point to a cluster; (3) select the k examples with the smallest margin scores. We use Gemma1 <cit.> to generate the probability of violating any of the policies and use | probability - 0.5| as the margin score. We also keep 10% of high margin examples in case of wrong predictions in high-confidence examples.  (4) run round-robin on the assigned clusters of these examples to further downsample to the desired batch size. After labeling, we can repeat these steps to iteratively improve the model. 

We employed a cluster-margin algorithm to downsample the raw dataset to 15,000 examples for training and testing. We reserved 10,500 examples for training, aligning with the training data volume of LlamaGuard1 <cit.>, and 4,500 for testing. Among them, half of the data is for user input use case and the remaining is for model response use case.



 §.§ Data Annotation



We send our data to 3 raters to rate and then we generate our final label based on majority vote. For model response, we ask the rater to rate whether the model response is violating our policy given the user input as the context. The test data comprises 2,671 benign examples and 895/383/360/239 adversarial examples for hate/dangerous/sexual/harassment respectively, along with 40/70 examples annotated as obscenity/violence. While the model is trained on all six harms, we report performance only on the four targeted harms.  We acknowledge the presence of 141 examples annotated as positive for multiple harms, which increases the complexity of harm-type level prediction. 



 §.§ Fairness Expansion

To improve fairness of our model, we leverage counterfactual fairness expansion <cit.> to expand our training data across identity categories like Gender, Race, Ethnicity, Sexual Orientation, and Religion. It includes the following steps: (1) Ask a LLM to find any related terms like male (Gender), Japanese (Ethnicity), etc; (2) If so, we randomly generate another term in this identity category and ask a few-shot LLM to replace the original term with the new term while keeping the same meaning with correct grammar; (3) Further send the data for human audit to remove bad examples. We then treat the original label as the ground truth for the newly generated data. 



 §.§ Model Fine-Tuning


We further supervise fine-tune (SFT) Gemma2 Instruction-Tuned (IT) models (2B, 9B, and 27B parameters) using the instruction shown in Fig. <ref>. We employ distinct policy definitions for each harm type and the model output is either Yes or No token. Our models are trained on TPUv5 lite with batch size of 16, a max sequence of 8k, and a learning rate of 1e-6. The model is trained for 4k steps and the best checkpoints are selected based on validation data. We calculate our predicted probability based on Eq. <ref> below:

    exp(LL(Yes)/T) + α/exp(LL(Yes)/T) + exp(LL(No)/T) + 2α


Here LL(·) is the log likelihood of the token generated by the model; T and α are hyperparameters to control temperature and uncertainty estimate. 



§ EXPERIMENTS





 §.§ Setup

Despite the abundance of safety-related benchmark datasets, direct comparison remains challenging due to several factors: (i) variations in policy definitions and supported harm types across datasets; (ii) inconsistencies in policy definitions even within the same harm type; and (iii) the predominance of binary (safe/unsafe) classification models rather than harm-type level prediction.  To address these challenges, we conduct experiments on two fronts:


    
  * Overall Binary Classification: We aggregate our prediction results into binary outcomes by maximizing probabilities over all harms. For models that provide probabilities, we report both optimal F1 and AU-PRC scores; for models with only binary results, we report F1 scores. 
    
  * Harm Type Level Classification: We provide a detailed performance comparison at the individual harm type level. We adopt a one-vs-all setup as described in <cit.>, i.e. we transform a multi-class classification problem into multiple binary classification problems, where each classifier focuses on distinguishing positive examples in one specific harm type and treat all others as benign examples.




 §.§ Benchmark Datasets and Baseline Models

OpenAI Moderation <cit.> comprises 1,680 prompt examples labeled for eight safety categories: sexual, hate, violence, harassment, self-harm, sexual/minors, hate/threatening, violence/graphic. Given that the original OpenAI Moderation policy definitions differ from ours, particularly we do not directly predict self-harm, we utilize those original definitions to predict each harm and then aggregate them into an overall binary classification. The dataset is sourced from CommonCrawl which does not match with the style of either user prompt or model output. Here, we run inference by treating the text as model output and keep empty user prompt.

ToxicChat <cit.> contains 10k examples with binary toxicity label for the prompt. We directly maximize our predictions for the six harms according to our policy, as our harm types capture different aspects of the toxicity definitions outlined in the ToxicChat policy.

ShieldGemma Prompt & ShieldGemma Response are our test dataset. it contains 4,500 examples with labels in total for both use cases. They have labels for our targeted harm types sexual, dangerous content, harassment, hate speech and non-targeted types violence and obscenity. More details are in section sec:annotation.

Baseline Models: We evaluate our models against several models: OpenAI Mod API <cit.>, LlamaGuard <cit.>, WildGuard <cit.>, and GPT-4. For GPT-4, we utilize the openAI API (model=gpt-4-0613) with our prompts, obtaining the log probability of the first token and converting it into the probability of a policy violation.



 §.§ Overall Binary Classification Results


The overall binary classification results are presented in Table <ref>. All ShieldGemma (SG) models (2B, 9B and 27B) outperform all baseline models. Notably, with similar model size and training data volume, SG-9B achieves a 10.8% higher average AU-PRC compared to LlamaGuard1 on external benchmarks.  Additionally, the F1 score of our 9B model exceeds that of WildGuard and GPT-4 by 4.3% and 6.4%, respectively.

Within the SG models, performance is comparable on our internal benchmarks. On external benchmarks, the 9B/27B model demonstrates slightly stronger generalization capability, achieving on average a 1.2%/1.7% higher AU-PRC than its 2B model. 




 §.§ Harm Type Level Results





We evaluate the harm-type level performance on our test datasets: SG Prompt and SG Response. The results are shown in Fig. <ref>. All SG models have outperformed GPT-4 by a big margin for all of the harms. Overall, GPT-4 is weak in distinguishing different harms. For example 76% of hate speech data points have been classified as positive for harassment. Note that the performance gap is expected, and the comparison is less favorable for GPT-4, as our model has been trained on datasets similar to the test datasets, while GPT-4 is evaluated zero-shot without any specific training. The performance among SG models is close to each other. On average, SG-9B and SG-27B have outperformed SG-2B by less than 2%.



§ LIMITATIONS

Despite our efforts to enhance the robustness of our model against adversarial attacks, fairness, and diversity in the training data, several limitations remain:

Fairness: While we have implemented fairness counterfactual expansion to mitigate bias in our training data, label discrepancies may still arise when identity groups are swapped. These discrepancies often stem from inherent biases within the pre-training dataset <cit.>.

Generalization: We have observed that our larger models demonstrate stronger performance on external benchmarks with new harm types and text styles. Overall, this generalization capability of our larger models are slightly stronger than our smaller 2B model. It also requires additional experiments to further verify the generalization on other datasets.

Implicit Cultural Harm: Although LLMs exhibit some understanding of cultural contexts, they may struggle to fully grasp implicit harm within these contexts.

Safety vs. Helpfulness: While our models demonstrate a strong ability to filter potential safety risks, their interpretation of policy violations may be overly conservative. This could interfere with helpfulness when used to filter LLM responses. We recommend that downstream clients adjust filtering thresholds based on their specific use cases.

LLM-as-a-classifier: Our model is specifically designed for classification tasks, with an output restricted to Yes or No token as the first output token when the prompt is correctly configured.  However, it's crucial to acknowledge that as an LLM, it remains capable of generating responses to any text input. We strongly advise the users to use it solely for generating Yes/No token scores (we call it scoring mode, detailed in our model card), and avoid using it in a chat-like manner since it may produce unethical or unsafe content due to the absence of additional safety instruction-tuning for conversational use.

We are dedicated to ongoing research and development to address these limitations and further refine our classifiers.



§ CONCLUSION


This paper presents a significant advancement in safety content moderation through our suite of specialized models, built on the foundation of the public Gemma2 <cit.> language models. We demonstrate their superior performance on diverse benchmarks, highlighting the effectiveness of our approach. Additionally, our novel synthetic data generation pipeline offers a valuable tool for researchers and practitioners to create high-quality, diverse datasets for safety and other domains. We are excited to share these resources with the research community to foster further development in this critical area.










§ CONTRIBUTIONS AND ACKNOWLEDGMENTS




Core Contributors 

Wenjun Zeng 

Yuchi Liu 

Ryan Mullins 

Ludovic Peran 

Contributors 

Joe Fernandez 

Hamza Harkous 

Karthik Narasimhan 

Drew Proud 

Piyush Kumar 

Bhaktipriya Radharapu 

Olivia Sturman 

Oscar Wahltinez



Other Specialty Areas 

Special thanks and acknowledgments to these individuals for their assistance in respected areas:

Central Support  

Manvinder Singh 

Kathy Meier-Hellstern 

Shivani Podder

Checkpoint Conversions  

Nam T. Nguyen 

Matthew Watson 

Ethics and Safety  

Antonia Paterson

Jenny Brennan

Gemma Model  

Surya Bhupatiraju 

Victor Cotruta 

Armand Joulin 

Kathleen Kenealy 

Tris Warkentin

Go-to-Market  

Kat Black 

Meg Risdal

Team Acknowledgements 

Our work is made possible by the dedication and efforts of numerous teams at Google. We would like to acknowledge the support from the following teams: Gemma, Google DeepMind Responsibility, Kaggle, Keras, Perspective.






 Paper Title: Enhancing Ethereum Fraud Detection via Generative and Contrastive Self-supervision.

















        


        





























Enhancing Ethereum Fraud Detection via Generative and Contrastive Self-supervision
    Chenxiang Jin, Jiajun Zhou^∗, Chenxuan Xie, Shanqing Yu, Qi Xuan, Senior Member, IEEE, Xiaoniu Yang


This work was supported in part by the Key R&D Program of Zhejiang under Grants 2022C01018 and 2024C01025, by the National Natural Science Foundation of China under Grants 62103374 and U21B2001. (Corresponding author: Jiajun Zhou.)
C. Jin, C. Xie, S. Yu and Q. Xuan are with the Institute of Cyberspace Security, College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China, with the Binjiang Institute of Artificial Intelligence, ZJUT, Hangzhou 310056, China. E-mail:{jincxiang, 221122030330, yushanqing, xuanqi}@zjut.edu.cn.
J. Zhou are with the Institute of Cyberspace Security, College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou 310023, China, with the Binjiang Institute of Artificial Intelligence, ZJUT, Hangzhou 310056, China. E-mail: jjzhou@zjut.edu.cn.


X. Yang is with the Institute of Cyberspace Security, Zhejiang University of Technology, Hangzhou 310023, China, and also with Science and Technology on Communication Information Security Control Laboratory, the 36th Research Institute of China Electronics Technology Group Corporation, Jiaxing 314033, China. Email: yxn2117@126.com.










    August 5, 2024
===================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 


The rampant fraudulent activities on Ethereum hinder the healthy development of the blockchain ecosystem, necessitating the reinforcement of regulations. However, multiple imbalances involving account interaction frequencies and interaction types in the Ethereum transaction environment pose significant challenges to data mining-based fraud detection research. To address this, we first propose the concept of meta-interactions to refine interaction behaviors in Ethereum, and based on this, we present a dual self-supervision enhanced Ethereum fraud detection framework, named . This framework initially introduces a generative self-supervision mechanism to augment the interaction features of accounts, followed by a contrastive self-supervision mechanism to differentiate various behavior patterns, and ultimately characterizes the behavioral representations of accounts and mines potential fraud risks through multi-view interaction feature learning. Extensive experiments on real Ethereum datasets demonstrate the effectiveness and superiority of our framework in detecting common Ethereum fraud behaviors such as Ponzi schemes and phishing scams. Additionally, the generative module can effectively alleviate the interaction distribution imbalance in Ethereum data, while the contrastive module significantly enhances the framework's ability to distinguish different behavior patterns. The source code will be released on GitHub soon.















Ethereum, Fraud Detection, Generative Learning, Contrastive Learning, Multi-view Learning, Self-Supervision





§ INTRODUCING









Ethereum <cit.>, an open-source blockchain platform, is rapidly gaining prominence due to its revolutionary smart contract technology. 
Through smart contracts <cit.>, developers can establish protocols that autonomously execute, manage, and enforce contractual terms without the involvement of any third-party intermediary.
This novel paradigm has given rise to the concept of decentralized finance (DeFi) <cit.> in the financial domain. In Ethereum, DeFi applications are implemented via smart contracts, allowing users to directly interact with financial agreements for activities such as lending, trading, investing, and insurance. The automation and transparency of smart contract execution provide users with a more liberated financial experience, enabling global access to financial services without geographical or institutional constraints.














However, as the deployment of Ethereum in the financial domain surges, its technological characteristics raise security concerns for the financial ecosystem. First, decentralization means there is no central authority for oversight and regulation, allowing fraudsters to operate with impunity. Second, anonymity enables malicious actors to hide their real identities, exacerbating the challenges associated with crime detection. Third, the immutability of contracts makes it difficult to fix code vulnerabilities once they are discovered. 

Furthermore, the global liquidity of cryptocurrencies poses challenges to the coordination of regulatory frameworks across diverse jurisdictions.
Despite the increasing focus on cryptocurrency finance security in recent years, a multitude of fraudulent activities persist, resulting in substantial financial losses.
According to the Bangkok Post[https://www.bangkokpost.com/thailand/general/2644790/cops-arrest-5-foreigners-over-links-to-b2-7bn-scamhttps://www.bangkokpost.com/], the Cyber Crime Investigation Bureau (CCIB) arrested individuals involved in a fraudulent virtual currency investment platform that swindled more than $27 million. 
Besides, TRM Labs[https://www.trmlabs.com/post/inebreaknorth-korean-hackers-stole-600-million-in-crypto-in-2023https://www.trmlabs.com/]  reported that North Korean hackers stole at least $600 million in cryptocurrency in 2023. 
Therefore, the regulation of Ethereum cryptocurrency transactions is imperative to ensure financial security.























From the perspective of data mining, existing Ethereum fraud detection studies mainly employ machine learning (ML) <cit.> and graph modeling analysis <cit.> to uncover potential fraudulent risks in transaction records and smart contracts. ML-based methods mainly extract manual features from Ethereum data and train ML models to identify and predict potential fraudulent behaviors. However, such methods heavily rely on expert knowledge for designing manual features, limiting their generalizability to diverse blockchain scenarios. On the other hand, graph-based methods mainly use Ethereum data to construct transaction graphs and design graph learning models to identify malicious accounts and fraudulent activities. Although such methods can effectively capture account behavior patterns and improve the effectiveness and interpretability of fraud detection, they rely on constructing transaction graphs and designing graph learning models. Moreover, both studies overlook certain challenges inherent in Ethereum transaction scenarios to some extent:
1) Frequent and complex transaction activities. Ethereum continuously generates a large number of transaction records, involving different types of accounts and interaction patterns, which imposes higher requirements on the models' representation capabilities;
2) Imbalanced interaction frequency distribution. Ethereum accounts exhibit varying levels of interaction activity, as shown in Fig. <ref>, and such imbalance in interaction frequency distribution can prevent the models from effectively learning the behavior patterns of low-activity accounts;
3) Imbalanced account behavior distribution. Accounts of the same type on Ethereum may participate in diverse transaction activities, displaying distinct behavior preferences. This imbalance in behavior distribution can lead to biases in the models' characterization of account features.









To overcome the limitations of existing studies and address the challenges in Ethereum, we propose a dual self-supervision enhanced Ethereum fraud detection framework, named . Specifically, this framework first defines the concept of meta-interactions to describe fine-grained Ethereum interaction behaviors and introduces a generative self-supervised mechanism to design a fine-grained Ethereum interaction feature generation module. This module can augment multi-view interaction features for accounts with low activity levels and low-frequency interaction types, alleviating the imbalance in behavior distribution. Secondly, the framework introduces a contrastive self-supervised mechanism to design a coarse-grained Ethereum interaction feature contrast module, which can improve the framework's ability to distinguish different account behavior patterns. Finally, the framework designs a multi-view Ethereum interaction feature learning module, which effectively characterizes account features and captures potential fraudulent behaviors through account type-specific feature encoding, aggregation, and propagation processes.
We conduct extensive evaluation and comparative experiments in real Ethereum transaction scenarios to validate the effectiveness and superiority of our proposed framework in detecting common Ethereum fraud behaviors, such as Ponzi schemes and phishing scams.
The main contributions of this paper can be summarized as follows:

  
  * We define the concept of meta-interactions to describe fine-grained Ethereum interaction behaviors.
  
  * We design a dual self-supervision enhanced Ethereum fraud detection framework. By incorporating fine-grained interaction feature generation, coarse-grained interaction feature contrast, and multi-view feature learning processes, our framework alleviates the imbalance in account behavior distribution, enhances the ability to distinguish different behavior patterns, and ultimately improves the effectiveness of Ethereum fraud detection.
  
  * Extensive experiments on real Ethereum data demonstrate the effectiveness and superiority of our framework in detecting Ethereum fraud behaviors.























































§ RELATED WORK
 
This section reviews research on fraud detection in Ethereum, primarily focusing on the detection of Ponzi schemes and phishing scams. The related studies can be broadly categorized into two types: machine learning-based approaches and graph analysis-based approaches.













 §.§ Machine Learning-based Fraud Detection


This type of methods typically utilizes manual feature engineering to extract predefined features from Ethereum data and train machine learning models to identify and predict potential fraudulent behaviors, such as Ponzi schemes and phishing scams.
Since such methods design targeted manual features based on different types of fraud, this subsection discusses related studies for different fraud separately.



  §.§.§ Ponzi Scheme Detection










Ponzi schemes <cit.> in Ethereum involve writing and deploying smart contracts that promise high returns to attract investors to transfer funds to the contract address. The returns for early investors come from the funds of subsequent investors rather than legitimate investment profits. 
Due to the anonymity of Ethereum, fraudsters can easily hide their identities and transfer funds, increasing the difficulty of regulation and tracking. When there are not enough new investors, the funds in the contract are insufficient to pay the high returns, leading to the collapse of Ponzi schemes.

Since Ponzi schemes on Ethereum are typically deployed on contract accounts, analyzing smart contract code for Ponzi scheme detection is a straightforward option.
Bartoletti  <cit.> analyze the source codes of Ponzi contract and categorize Ponzi schemes into four types based on their behavioral patterns. 

Onu  <cit.> compare the performance of random forest, neural network, and k-nearest neighbors algorithms in Ponzi scheme detection, concluding that the random forest model achieves the best detection performance with the fewest features, enabling early detection of Ponzi schemes. 
Zheng  <cit.> extract contract account features from multiple views, including bytecode, semantics, and developer information, and develop a multi-view cascaded ensemble model to detect Ponzi schemes in newly created contract accounts. 
However, only the bytecode of smart contracts is publicly available, while the source code is not always. As a result, detection methods based on code analysis might not fully identify Ponzi schemes concealed within smart contracts.
Some studies combine more accessible transaction records to detect Ponzi schemes by analyzing the behavior of contract accounts. 
Hu  <cit.> summarize four types of fraudulent behavior patterns by analyzing account transaction records, and then construct 14-dimensional transaction features and train LSTM <cit.> model for fraud detection. 
Chen  <cit.> extract opcode frequency features from contract code and 13 transaction features from transaction records, combining these features to train XGBoost model for Ponzi scheme detection. 
In addition to these two types of features, Zhang  <cit.> further integrate bytecode features and employ LightGBM for more efficient fraud detection. 
Considering the dynamic nature of transactions, Wang  <cit.> construct temporal transaction features based on timestamps and combine them with contract code features for early detection of Ponzi schemes.




  §.§.§ Phish Scam Detection

Phishing scams on Ethereum <cit.> employ deceptive tactics to lure victims into disclosing their private keys or transferring funds to accounts controlled by attackers. Since phishing scams typically do not rely on contract functionality, detection against phishing scams usually focuses only on transaction records.
Chen  <cit.> extract relevant features of the target accounts and their interaction objects from transaction records and utilize LightGBM for phishing detection.
Kabla  <cit.> utilize a voting technique based on multi-ranking methods to select important features for phishing detection, which improves detection efficiency while reducing feature dimensions.
Hu  <cit.> first serialize transaction features according to timestamps and then train BERT for phishing detection.
Phishers often disguise themselves by frequently interacting with normal accounts, thereby disrupting fraud detection. Several studies extract account features from multiple views to break through the disguise.
Ghosh  <cit.> extract relevant features from the temporal and network views in addition to the account's intrinsic features, fuse the multi-view features and feed them into various classifiers for phishing detection.
Xu  <cit.> propose a multi-layer defense system to effectively simulate and track the dynamic evolution of phishing scams, achieving early to mid-term phishing fraud warnings.










































 §.§ Graph-based Fraud Detection




This type of methods primarily utilize transaction records to construct large-scale transaction graphs and design graph-based approaches to extract account behavior features for fraud detection. Given the broad applicability of graph methods in analyzing diverse account behavior patterns, this subsection does not specifically discuss the detection of different frauds separately.
Wu  <cit.> guide a random walk process on the Ethereum transaction graph using transaction amounts and timestamps to learn the structural representation of accounts, ultimately using SVM classifier for phishing detection. 
Similarly, Li  <cit.> calculate the similarity between accounts by considering the structural homogeneity and transaction homogeneity of the transaction graph, guiding random walks to learn account representations.

Tharani  <cit.> integrate transaction features and attributes of the transaction graph, employing GraphSAGE <cit.> to acquire account representations, which are subsequently jointly fed into a downstream machine learning model for fraud detection.

The aforementioned two-stage methods treat account feature learning and fraud detection as separate processes, potentially hindering the framework's ability to effectively learn features directly relevant to the detection task. Additionally, the selection of classifiers significantly impacts the detection performance. To address these limitations, researchers have endeavored to develop end-to-end transaction graph learning frameworks for fraud detection.
Yu  <cit.> construct transaction graphs and utilize graph convolutional networks (GCNs) to simultaneously learn transaction features and interaction behavior features of accounts, achieving end-to-end Ponzi scheme detection. 
Li  <cit.> design a tri-channel framework to separately capture the features of accounts, transactions, and transaction structures for fraud detection. 
Considering the large scale of transaction graph, Liu  <cit.> reduce the graph size by filtering out low-importance accounts and use a two-layer GCN to learn the final account representations.
Liang  <cit.> analyze the code logic functions of contract accounts to construct a Contract Runtime Behavior Graph (CRBG) for describing the behavior of contract accounts. They employ a graph attention network (GAT)<cit.> for account representation learning and fraud detection.

The aforementioned static methods fail to consider the dynamic nature of transactions when constructing transaction graphs. 
To address this, Jin  <cit.> design a transaction graph construction method based on temporal evolution augmentation, incorporating time information while expanding the transaction graph scale, and ultimately combine transaction and code information for early detection of Ponzi schemes. 
Wang  <cit.> construct multiple subgraphs incorporating temporal information extracted from transaction records. By identifying and analyzing specific temporal motifs within the transaction graph, they capture account representations enriched with comprehensive temporal details, thereby enhancing the performance of existing fraud detection methods.
Zhang  <cit.> construct temporal transaction graphs and combine LSTM and GCN for efficient learning of both temporal and structural transaction features.


Furthermore, the homogeneous transaction graphs constructed by the aforementioned methods neglect the heterogeneity of accounts and transaction behaviors. 
To address this, Zhou  <cit.> incorporate call records from contract accounts as features for externally owned accounts and propose a hierarchical attention encoder based on contrastive learning to effectively learn account representations, alleviating the issue of label scarcity. 
Du  <cit.> utilize a similar transaction graph construction method and design a GNN-based link prediction method to de-anonymize accounts in mixing services. 
Jin  <cit.> predefine static and temporal metapaths to describe the complex interaction behaviors in Ethereum and design a metapath-based account feature augmentation module, which effectively improves the performance of existing Ponzi scheme detection methods. 
Considering that defining metapaths requires expert knowledge and is time-consuming, Liu  <cit.> and Xu  <cit.> employ transformer models to autonomously acquire metapath patterns, subsequently aggregating neighbor accounts based on these metapaths to obtain higher-order information for enhancing fraud detection.

















































































§ ETHEREUM INTERACTION GRAPH MODELING
 





 §.§ Preliminaries of Ethereum Data







In Ethereum, an account represents an entity that holds Ether and can be categorized into two types: Externally Owned Accounts (EOAs) and Contract Accounts (CAs). EOAs are managed by their respective private key holders, who possess the capability to initiate transactions on the Ethereum network. CAs are governed by their underlying smart contract code and can only be triggered to execute functions defined within the contract. Interactions between Ethereum accounts can be classified into two categories: transfer (trans) and contract calls (call). Transactions primarily involve the transfer of Ether, while contract calls obtain various services by triggering functions within the smart contract. Analyzing these interactions further can provide deeper insights into the functioning of the Ethereum network and uncover potential risks.





 §.§ Graph Modeling for Ethereum Fraud Detection


 
In this paper, we analyze Ethereum interactions and perform fraud detection from a graph perspective. To better represent complex interaction scenarios, we model Ethereum data as a heterogeneous Ethereum interaction graph:

  Ethereum accounts and the interactions between them can be regarded as nodes and edges, respectively. By further refining their types, we construct a heterogeneous Ethereum interaction graph G=(𝒱, ℰ, 𝒯_𝒱, 𝒯_ℰ, X, 𝒴), where 𝒱={v_1, v_2, ⋯, v_n} is the set of account nodes, each with a specific type indicated by 𝒯_𝒱: v ↦{eoa, ca} as either EOA or CA, ℰ = {e_ij|  e_ij = (v_i, v_j),  v_i, v_j ∈𝒱} is the set of interaction edges, each with a specific type indicated by 𝒯_ℰ: e ↦{trans, call} as either transfer or contract call. X=[x_1, x_2, ⋯, x_n]^⊤∈ℝ^n× d is the account feature matrix, and d is the dimension of features. Due to the anonymity of Ethereum, the known account identity information is partially labeled, denoted as 𝒴={ (v_i, y_i)| v_i∈𝒱_l,  |𝒱_l| ≪ n }, where 𝒱_l is the set of labeled accounts. 

 

  

  
  
  
  
  
  

























According to the above definition, each account v_i is assigned an initial feature vector x_i, which is often an indispensable input for most detection models. Considering that complex and well-designed features can be easily circumvented by fraudsters through behavior adjustment, thereby weakening the detection models and enabling evasion, we attempt to design a more general and concise set of manual features for account feature initialization. Specifically, we define the following manual features based on interaction details:

  
  * Total / average investment and return under specific interaction types: 2 × 2× 2 = 8 types.
  
  
  * Balance under specific interaction types: 1× 2=2 types.
  
  * Number of initiations and receptions under specific interaction types: 2× 2=4 types.

After statistics, we can construct a 14-dimensional feature vector for each account.
Furthermore, we model fraud detection in Ethereum as a node classification task on the graph. Specifically, we need to design a detector f that characterizes the identity features of accounts by analyzing their interaction patterns, and ultimately establishes a mapping from accounts to identity labels: f(v,G)↦ y.
































 §.§ Refinement of Ethereum Interactions




In Sec. <ref>, we mention that interactions in Ethereum can be categorized into two types: transfer (trans) and contract calls (call), which we refer to as coarse-grained interactions. 
Several studies <cit.> have defined meta-paths to describe the behavior patterns of Ethereum accounts, as shown in Fig. <ref>(a). While such practice is feasible, it encounters several issues. Firstly, the definition of meta-paths relies on statistical analysis and expert knowledge, which limits their generalizability. Secondly, specific meta-paths define specific behavior patterns, and their adaptability may weaken when confronted with complex and variable fraud scenarios. Finally, as the scale of interactions increases, the number of meta-path instances will surge, potentially losing behavioral representativeness.

To describe Ethereum interactions in a more fine-grained manner, we further refine the interactions based on the mechanism of Ethereum, where transfers can occur between any accounts while contract calls can only target contract accounts. Specifically, in conjunction with the structure of , we propose the concept of meta-interaction:

  In an interaction graph, meta-interaction represents the most fine-grained interaction behavior pattern between arbitrary entity pairs ⟨ s,t ⟩, which can be formalized as r = ⟨𝒯_𝒱(s), 𝒯_ℰ(e), 𝒯_𝒱(t)⟩ = 𝒯_𝒱(s) 𝒯_ℰ(e)⟶𝒯_𝒱(t), where s and t denote the initiator and recipient of the interaction respectively, and e denotes the specific interaction between s and t.


  
  



















Unlike the coarse-grained interactions trans and call which only reflect the ways in which accounts interact with each other, meta-interactions further describe the ways in which specific types of accounts can interact with each other, and thus can be referred to as fine-grained interactions. Based on the definition of meta-interaction and the characteristics of Ethereum, we can identify six types of meta-interactions from , as shown in Fig. <ref>(b):

    r^1_ca:  CAcall⟶CA,       
           r^1_eoa: EOAcall⟶CA
       r^2_ca:  CAtrans⟶CA,           
           r^2_eoa: EOAtrans⟶CA
       r^3_ca:  CAtrans⟶EOA,          
           r^3_eoa: EOAtrans⟶EOA












Furthermore, we define ℛ={r^1_ca, r^2_ca, r^3_ca, r^1_eoa, r^2_eoa, r^3_eoa} to denote the set of meta-interactions.




































































































































































§ METHODOLOGY
 





In this section, we discuss the design details of the proposed framework , as schematically depicted in Fig. <ref>. This framework takes  as input and outputs the prediction labels for the target accounts, and it primarily consists of the following modules:
1) A fine-grained interaction feature generation module to simulate and augment the behavioral features of accounts;
2) A multi-view feature learning module to deeply characterize the behavior patterns of accounts;
3) A coarse-grained interaction feature contrast module to enhance the framework's ability to distinguish different account behaviors.
Next, we introduce the details of each module.










































 §.§ Fine-grained Ethereum Interaction Feature Generation







In complex Ethereum interaction scenarios, account entities may encounter the following issues: 1) different accounts exhibit varying levels of interaction activity, leading to imbalanced degree distributions, as illustrated in Fig. <ref>; 2) the same account participates in different types of interactions to varying extents, resulting in imbalanced interaction distributions, as illustrated in Fig. <ref>. These diverse and imbalanced interaction phenomena can introduce biases when characterizing account interaction behaviors.
To alleviate these issues, we introduce a generative self-supervised mechanism to augment the interaction features of Ethereum accounts, particularly for those with low activity levels and low-frequency interaction types.











  §.§.§ Model Architecture and Pre-training

Specifically, we propose a fine-grained Ethereum interaction feature generation module, as illustrated in Fig. <ref>.
Considering that for a target account v_i, the characteristics of its neighboring entities x_j in its local interaction environment are influenced by its own characteristics x_i as well as its interaction habits e_ij. Based on this prior, if we want to augment more interaction features for the target account, we have to understand the potential feature distribution of its local interaction environment. Assuming that the potential features of the local interaction environment of account v_i follow a prior distribution z∼ p_θ(z|x_i, r_ij), where r_ij is the one-hot encoded form of the meta-interaction between v_i and v_j, as shown in Fig. <ref>(a), then the neighbor features x_j follow a generative distribution x_j∼ p_θ(x|z, x_i, r_ij). Further, we design an interaction-aware conditional variational autoencoder (ICVAE) to learn the interaction feature distribution of target accounts and achieve fine-grained interaction feature generation. 
Compared to the conditional variational autoencoder (CVAE) <cit.> with a single condition setting, our ICVAE utilizes both the target account and specific meta-interaction as conditions, and neighbor as input to train the encoder and decoder.
It serves as a generative model with a log-likelihood probability of log p_θ(x_j|x_i, r_ij), which denotes the probability that the model generates the observed neighborhood feature conditional on the target account feature x_i and the meta-interaction feature r_ij. Let θ and ϕ represent the generative parameters and variational parameters respectively, we have:

    log p_θ(x_j|x_i, r_ij) 
        =𝔼_q_ϕ(z|x_j, x_i, r_ij)[log p_θ(x_j|x_i, r_ij)]
        =𝔼_q_ϕ(z|x_j, x_i, r_ij)[logp_θ(x_j, z|x_i, r_ij)/p_θ(z|x_j, x_i, r_ij)]
        =𝔼_q_ϕ(z|x_j, x_i, r_ij)[logp_θ(x_j, z|x_i, r_ij)/q_ϕ(z|x_j, x_i, r_ij)·q_ϕ(z|x_j, x_i, r_ij)/p_θ(z|x_j, x_i, r_ij)] 
        =𝔼_q_ϕ(z|x_j, x_i, r_ij)[log p_θ(x_j, z|x_i, r_ij) - q_ϕ(z|x_j, x_i, r_ij) ]    
                    + 𝔼_q_ϕ(z|x_j, x_i, r_ij)[logq_ϕ(z|x_j, x_i, r_ij)/p_θ(z|x_j, x_i, r_ij)]
        =𝔼_q_ϕ(z|x_j, x_i, r_ij)[log p_θ(x_j, z|x_i, r_ij) - q_ϕ(z|x_j, x_i, r_ij) ]    
                    + KL(q_ϕ(z|x_j, x_i, r_ij) ∥ p_θ(z|x_j, x_i, r_ij) )
       ≥𝔼_q_ϕ(z|x_j, x_i, r_ij)[log p_θ(x_j, z|x_i, r_ij) - q_ϕ(z|x_j, x_i, r_ij) ]    
        = 𝔼_q_ϕ(z|x_j, x_i, r_ij)[log p_θ( z|x_i, r_ij) + log p_θ(x_j|z, x_i, r_ij)  
                    - q_ϕ(z|x_j, x_i, r_ij)] 
        =𝔼_q_ϕ(z|x_j, x_i, r_ij)[log p_θ(x_j|z, x_i, r_ij)] 
                    - 𝔼_q_ϕ(z|x_j, x_i, r_ij)[logq_ϕ(z|x_j, x_i, r_ij)/p_θ( z|x_i, r_ij)] 
        =𝔼_q_ϕ(z|x_j, x_i, r_ij)[log p_θ(x_j|z, x_i, r_ij)] 
                     -KL(q_ϕ(z|x_j, x_i, r_ij) ∥ p_θ( z|x_i, r_ij) )
        = ELBO

where q_ϕ(z|x_j, x_i, r_ij) denotes the variational posterior distribution and is used to approximate the true posterior distribution p_θ(z|x_j, x_i, r_ij), KL(q ∥ p) denotes the Kullback-Leibler divergence between the two distributions q and p.
Since directly computing the log-likelihood probability is difficult, we approximate it by maximizing its lower bound (Evidence Lower Bound, ELBO). Thus, the ELBO-based optimization objective of our ICVAE can be derived as follows:

    ℒ_ELBO = 1/|𝒩_v_i|∑_v∈𝒩_v_ilog p_θ(x_j|z^(v), x_i, r_ij) 
                 -KL(q_ϕ(z|x_j, x_i, r_ij) ∥ p_θ( z|x_i, r_ij) )

where 𝒩_v_i represents the neighbor set of account v_i. 
The first term in ℒ_BLEO is the reconstruction error, which measures the difference between the features generated by ICVAE and the original interaction features. The second term is the KL divergence, which measures the difference between the variational posterior distribution and the prior distribution.




















































After specifying the optimization objective, we are ready to pre-train an ICVAE for all accounts. The ICVAE consists of an encoder and a decoder, as illustrated in Fig. <ref>(b). During the training phase, it takes interaction feature pairs {(x_i, x_j, r_ij) | v_i∈𝒱, v_j∈𝒩_v_i} as input to maximize the ℒ_ELBO. Specifically, the encoder maps the conditions (x_i, r_ij) and the input data x_j to the latent space, learning and outputting the distribution parameters of the latent variables z:

    μ, σ = Encoder(x_i, x_j,r_ij)

where the mean μ and standard deviation σ are used to describe the variational posterior distribution q_ϕ(z|x_j, x_i, r_ij).
The decoder takes the conditions (x_i, r_ij) and the sampled latent variable z as inputs, generating the interaction features x̂_̂î associated with account v_i:

    [                   z = μ + ϵ⊙σ; x̂_i = Decoder( x_i, r_ij, z) ]

where z is sampled via re-parameterization trick, ϵ∼𝒩(0, I) is Gaussian noise that follows a standard normal distribution, and ⊙ is the element-wise multiplication.
By maximizing ℒ_BLEO, ICVAE can learn the optimal parameters for both the encoder and the decoder, thereby minimizing the reconstruction error of the interaction features under given conditions, and ensuring that the posterior distribution parameterized by the encoder closely approximates the prior distribution.









































  §.§.§ Feature Generation


After the pre-training phase, we utilize ICVAE to generate additional interaction features for the target accounts. These generated interaction features can further enrich the neighborhood information of the target accounts and alleviate the issue of feature distribution imbalance. Specifically, given a target account v_i and different meta-interactions r∈ℛ as conditions, ICVAE can generate rich interaction features for it:

    𝒳̂_i=
        {x_i, x̂_i^r^1_∗, x̂_i^r^2_∗, x̂_i^r^3_∗}^M

where ∗=𝒯_𝒱(v_i) indicates the type of v_i, and x̂_i^r^1_∗ is the feature generated conditional on (x_i, r^1_∗). 
Note that under specific conditions, we can repeatedly sample the latent variable z to generate interaction features multiple times, repeatedly utilizing ICVAE to generate multiple interaction features under the same conditions, forming multiple feature views, and further alleviating the issue of feature distribution imbalance. The number of multiple feature views is denoted by M.







































 §.§ Multi-view Ethereum Interaction Feature Learning



With the pre-trained ICVAE, we obtain augmented interaction features from multiple views, which can alleviate the issue of sparse interaction features for several accounts to some extent. To better characterize the account behavior patterns, the multi-view interaction feature learning module will process both the initial and generated interaction features to uncover potential fraud risks. This module will encode and fuse the multi-view interaction features and ultimately propagate the features by combining the structural information of . The design details of this module are described below.



  §.§.§ Input Feature Encoding





The multi-view interaction features generated by the ICVAE remain in the input space, with their dimensions represented as 𝒳̂_i∈ℝ^M × 4 × d. To further explore the underlying information in the interaction features, a common practice is to design a weight-sharing encoder for all target accounts, which accepts the interaction features of the accounts as input and outputs high-dimensional hidden features. However, we consider such practice to be inappropriate for the following reasons. 
We sample a certain number of EOA and CA and count their 14-dimensional features, as shown in Fig. <ref>, from which we can observe noticeable differences among the features of accounts within the same type, as well as among different types of accounts. In such cases, a weight-sharing transformation usually fails to adequately distinguish the differences among accounts.
Therefore, to more adaptively characterize the interaction preferences of different types of accounts, we design account type-specific input feature encoding components, which follows a weight separation setting <cit.>.
Specifically, for the multi-view interaction features of target account v_i in the input space, we map them into the hidden space using an MLP parameterized by Θ̂_∗∈ℝ^d× d', obtaining the hidden representation ℋ̂_i∈ℝ^M× 4× d' as follows:

    ℋ̂_i=tanh(𝒳̂_i·Θ̂_∗) = {h_i, ĥ_i^r^1_∗, ĥ_i^r^2_∗, ĥ_i^r^3_∗}^M

where ∗ = 𝒯_𝒱(v_i) indicates that the MLP performs an account type-specific transformation on the input features.





  §.§.§ Intra-view and Inter-view Feature Aggregation



Subsequently, since ICVAE can generate three sets of augmented interaction features for each account, the feature dimensions within each view will be tripled. To reduce the parameter scale of our framework and improve training efficiency while preventing overfitting, we further perform intra-view interaction feature aggregation via mean pooling, followed by an MLP to ensure the expressive power of the aggregated features:

    ℋ̅_i   =tanh(MeanPooling(ℋ̂_i)·Θ̅_∗) 
       =tanh(1/4(h_i+ ĥ_i^r^1_∗+ ĥ_i^r^2_∗+ ĥ_i^r^3_∗)·Θ̅_∗)

where ℋ̅_i∈ℝ^M× d' and the account type-specific weight Θ̅_∗∈ℝ^d' × d' is utilized to parameterize the MLP.








After intra-view feature aggregation, the multi-view interaction features will be dimensionally reduced and updated, which is beneficial for the training of . Furthermore, we consider that the interaction features from different views exhibit certain correlations and complementarities, contributing to the final account feature representation to varying degrees. Therefore, we introduce the self-attention mechanism to capture the interdependencies among different views and achieve inter-view interaction feature aggregation, as shown in Fig. <ref>(c).
Specifically, in order to use the self-attention mechanism to compute correlations between different views, we initially generate query, key, and value vectors and subsequently integrate these features by calculating the attention weights. We define three type-specific transformation weights W^Q_∗, W^K_∗, W^V_∗∈ℝ^d' × d' to generate the three vectors:

    Q_∗ = ℋ̅_i·W^Q_∗,   K_∗ = ℋ̅_i·W^K_∗,   V_∗ = ℋ̅_i·W^V_∗

where Q_∗ is used to find relevant features, K_∗ represents the critical information of the features, and V_∗ is the feature representation used for weighted summation. 
Next, we calculate the inter-view correlations, also known as attention, to obtain the comprehensive feature representation of each view, and finally fuse the features from all views:

    H̅_i = MeanPooling(Softmax(QK^⊤)·V)

Note that during the inter-view feature aggregation process, the query vector of each view Q_a performs dot-product with the key vectors of other views K_b to obtain relevance score, Q_aK^⊤_b, which indicates the importance of the features in view a to those in view b. Using the attention scores across all views after softmax normalization, we perform a weighted summation of the value vectors to obtain the comprehensive feature representation for each view: ∑_b=1^MSoftmax(Q_aK^⊤_b)·V_b. Finally, we fuse the features of all views via mean pooling to obtain the feature representation of the target account H̅_i.




  §.§.§ Interaction Feature Passing





The behavioral patterns of accounts are not only related to their own characteristics, but also influenced by the characteristics of the objects they interact with. In order to more accurately characterize the behavioral patterns of the target accounts, we incorporate the interaction structure information among accounts in  and obtain the final account behavior representation through interaction feature passing. Specifically, we initially aggregate the features of the interaction objects associated with the target account, subsequently propagate them to the target account, and ultimately perform a deep transformation on the aggregated features. The above feature passing process will be iterated l times to complete the final account feature learning.

    H_i^(l) = tanh(( ∑_j∈𝒩_v_iH_j^(l-1)) ∥H_i^(l-1)) ·Θ_∗^(l-1)

where H_i^(0)=H̅_i and Θ_∗^(l-1) denotes the transformation weight in the feature passing process at the l-1-th iteration.
















































































































 §.§ Coarse-grained Ethereum Interaction Feature Contrast
 








In the definition of meta-interaction, we combine the type of interactions and the type of targets to distinguish fine-grained interaction behaviors. 
And each type of account involves up to three different meta-interactions. 
Take EOA as an example, this type of account involves three meta-interactions: r_eoa^1, r_eoa^2 and r_eoa^3, where r_eoa^1 involves contract call, and r_eoa^2 and r_eoa^3 involve transferring money. When the EOA makes a transfer, we consider that the CA or EOA that receives the transfer plays a similar role; whereas when the EOA makes a contract call and a transfer, even if the target is the same CA, the CA plays different roles in these two types of interactions. Therefore, the interaction type should play a dominant role in characterizing the interaction behavior. For this purpose, we use interaction type as one of the conditions for generating interaction features when designing ICVAE. Considering the limitations of MLP in encoding input features discussed in Sec. <ref>, we further introduce a contrastive self-supervised mechanism to enhance its ability to distinguish different interaction features, proposing the coarse-grained Ethereum interaction feature contrast, as shown in Fig. <ref>(b).

Specifically, from a coarse-grained interaction perspective, we believe that r_∗^2 and r_∗^3 should have more similar representations because they both involve transfer behaviors, while r_∗^2 and r_∗^1 should exhibit differences since the latter involves contract call behavior. Therefore, for a target account v_i of type ∗=𝒯_𝒱(v_i), we consider meta-interaction r_∗^2 as anchor sample, r_∗^3 as positive sample, and r_∗^1 as negative sample.
During framework training, we expect to make the representations of anchor and positive samples more similar and those of anchor and negative samples more distinct, which can be achieved by minimizing the following objective:

    ℒ_con^∗ = ∑_v_i∈𝒱_∗ max{0,ĥ_i^r_∗^2-ĥ_i^r_∗^3_2^2 -ĥ_i^r_∗^2 - ĥ_i^r_∗^1_2^2 +α}

where α is a margin hyperparameter that controls the minimum difference between positive and negative samples.
The above optimization objective can serve as regularization, facilitating the framework in capturing the similarities between interactions of the same type and the differences between interactions of different types, thereby enhancing the framework's ability to distinguish various interaction behaviors.


















 












































 §.§ Framework Training
 


In this paper, the ICVAE model used for generating interaction features can be pre-trained by maximizing the ℒ_ELBO. And Ethereum fraud detection is modeled as a node classification task on . By applying a prediction head f_Ψ to the final account representations, we map them into the corresponding prediction labels and compute the classification loss based on cross-entropy:

    ℒ_pred=-1/N∑_i=1^N y_i ·log(f_Ψ(H_i))

where N is the number of accounts in a batch (batch size).
Furthermore, the coarse-grained interaction feature contrast module can be regarded as an auxiliary task, serving as a regularization for the account classification task to jointly optimize the entire framework. The optimization objective of joint training can be defined as:

    ℒ = ℒ_pred + λ· (ℒ_con^eoa + ℒ_con^ca)

where λ serves as a trade-off coefficient to control the contribution of the feature contrast module.




















§ EXPERIMENTS
 









 §.§ Data Collection



We collect a certain amount of labeled data from the Xblock[<http://xblock.pro>] and Etherscan[<https://cn.etherscan.com>] platforms, covering 191 Ponzi accounts and 1151 non-Ponzi accounts, as well as 1206 phishing accounts and 1557 non-phishing accounts. Based on these labeled accounts, we further extract their first- and second-order transaction objects to construct the , obtaining two fraud detection datasets with imbalanced label distribution. 

The detailed statistics of these datasets is presence in Table <ref>.















 §.§ Comparison Methods


In this paper, we choose three categories of methods for comparison: machine learning-based method (LightGBM <cit.>), homogeneous graph analysis-based methods (Trans2Vec <cit.>, GCN <cit.>, GAT <cit.>, SAGE <cit.>, FA-GNN <cit.>), and heterogeneous graph analysis-based methods (HGT <cit.>, RGCN <cit.>, ieHGCN <cit.>, HTSGCN <cit.>). 
We exclude metapath-based methods from our comparison due to the significant effort required in designing metapaths for targeting different fraud behaviors.































 §.§ Experimental Settings

















In our framework, for the pre-trained generation module, the input is , the encoder is a 3-layer MLP with output dimensions {32, 64, 50}, and the decoder is a 2-layer MLP with output dimensions {32, 14}, the learning rate is set to 0.01. For the contrastive module, the margin hyperparameter α is set to 0.3. For the multi-view feature learning module, the search space for the number of views, hidden dimensions, and learning rate is {2, 3, 4}, {32, 64, 128}, and {0.01, 0.001} respectively, the number of attention heads is set to 4. During training, the search space for the trade-off coefficient λ is {0.001, 0.01, 0.1, 1}. The graph structure input for feature passing is the sampled .

For heterogeneous graph analysis-based methods, the inputed heterogeneous graph is obtained by sampling two layers of neighbors of the target nodes, with each layer sampling 100 neighbors based on different interaction behaviors. For homogeneous graph analysis-based methods, the sampled homogeneous graph is obtained by removing type information from the heterogeneous graph. Both methods employ a two-layer structure, with the search space for hidden dimensions and learning rate being {32, 64, 128} and {0.01, 0.001}, respectively. For methods involving attention, the number of attention heads is set to 4. For the two-stage Trans2Vec, we set the window size, walk length, and the number of walks per account to 10, 5, and 20, respectively, with the downstream classifier being an SVM with default parameters.
For LightGBM, the search spaces for the n_estimators parameter and learning rate are {100, 200, 300} and {0.01, 0.001}.

For all datasets, we divide them into training, validation, testing sets in a proportion of 6:2:2. All experiments will be run 5 times, with the average performance and standard deviation reported. Performance metrics used are Micro-F1 and Macro-F1.
The former emphasizes overall performance across all categories, potentially obscuring the performance on minor categories. Conversely, the latter assigns equal weight to each category, thereby providing a measure of the ability to classify imbalanced categories. Simultaneously considering both metrics enables a comprehensive evaluation of detection performance across various categories.




























































 §.§ Evaluation on Fraud Detection






To evaluate the effectiveness and superiority of our , we compare it against 10 baselines on two fraud datasets. The detection results are reported in Table <ref>, from which we can draw the following conclusions:

∙ Overall, compared with the optimal machine learning, homogeneous graph analysis, and heterogeneous graph analysis methods, our  achieves 14.31%, 2.56%, and 8.11% improvement in detecting Ponzi schemes, and 0.05%, 1.36%, and 4.45% in detecting phishing scams (Macro-F1 metric), which demonstrates the effectiveness and superiority of our  in fraud detection.

∙ The detection performance of different types of baselines varies significantly across various fraud categories. LightGBM achieves second-best performance in phishing detection but performed poorly in Ponzi scheme detection, indicating that the general manual features defined by machine learning methods cannot adequately describe diverse fraudulent behaviors. To achieve better detection performance, machine learning methods necessitate meticulous design of manual features tailored to different fraud categories, which lacks good generalization and heavily depends on expert knowledge. 
The two-stage homogeneous graph analysis methods represented by Trans2Vec suffer from similar limitations, in addition to the fact that it focuses more on the structural information of the transaction graph.
Other end-to-end homogeneous graph analysis methods show moderate detection performance for different fraudulent behaviors, somewhat surpassing heterogeneous graph analysis methods. 
A reasonable explanation is that both are affected by the interaction graph construction and data sampling methods. 

Finally, our  achieve the best detection performance for different frauds, indicating its generalizability in fraud detection.








































































































 §.§ Analysis of Multi-view Feature Learning Module


We first deploy ablation and parameter analysis to comprehensively investigate the effectiveness of the multi-view interaction feature learning module.


  §.§.§ Impact of View Setting


As shown in Table <ref>, when our  is adapted to a single-view setting,  (single view), its performance in detecting Ponzi schemes and phishing scams decreases by 1.71% and 0.91% (Macro-F1 metric), respectively, indicating that the utilization of multi-view settings can enhance the characterization of account features to some extent. Furthermore, under the multi-view setting, we vary the number of views within {2, 3, 4, 5, 10, 20}, as depicted in Fig. <ref>, from which we can observe that:
1) As the number of views varies, the Macro-F1 metric fluctuates more significantly compared to Micro-F1, suggesting that the former is more effective in evaluating framework performance under imbalanced data scenarios;
2) The framework's performance (measured by Macro-F1 metric) shows a general trend of initially increasing and then decreasing with the number of views. This suggests that increasing the number of views appropriately is beneficial for the detection of minority class fraud samples, but too many views may lead to feature redundancy, which in turn interferes with analysis of complex Ethereum interaction behaviors.












  §.§.§ Impact of Weight Separation Setting


To validate the effectiveness of the account type-specific transformation in our , we compare the performance of the framework under a weight-sharing setting,  (weight sharing), as shown in Table <ref>. The results show that weight separation learning for different types of account features yields positive gains, suggesting an enhanced distinguishability among the features of distinct accounts. 



  §.§.§ Impact of Self-attention Setting


To validate the superiority of the self-attention in inter-view feature aggregation, we compare three aggregation operations: summation, concatenation, and naive attention. As shown in Table <ref>, the self-attention aggregation operation facilitates our  in achieving improvement ranging from 2.02% to 3.89% and 0.87% to 2.78% on the two datasets (Macro-F1 metric), respectively. This result demonstrates that the self-attention mechanism can more effectively extract and fuse key features from different views, thereby enhancing fraud detection.








 §.§ Analysis of Coarse-grained Feature Contract Module


Furthermore, we conduct ablation and parameter experiments to analyze the effectiveness of the coarse-grained interaction feature contrast module.



  §.§.§ Effect of Contrastive Self-supervision



As shown in Table <ref>, the removal of the contrast module results in a performance degradation of 11.65% and 1.81% on the two datasets, respectively. This observation highlights the effective improvement of our framework's performance in fraud detection through feature contrast. Additionally, we compare the distance margin distribution between different types of meta-interaction features before and after introducing contrastive self-supervision, as shown in Fig. <ref>. It can be observed that after introducing contrastive self-supervision, the peak of the distance margin distribution shifts significantly to the right, indicating that the distance between anchors r^2_∗ and positive samples r^3_∗ is reduced, while the distance between anchors and negative samples r^1_∗ is increased. This phenomenon further illustrates that introducing contrastive self-supervision facilitates  in learning the similarities among interactions of the same type, as well as distinctions between interactions of different types, improving its ability to distinguish different interaction behaviors and ultimately enhancing fraud detection.




  §.§.§ Impact of Trade-off Coefficient


Furthermore, we investigate how the framework's performance is affected by the intensity of the auxiliary module, which is controlled by the trade-off coefficient λ. We vary λ within {0.001, 0.01, 0.1, 1, 10, 100} and observe the fluctuations in the framework's performance, as shown in Fig. <ref>. Firstly, we can see that in detecting Ponzi schemes,  achieves better detection results when λ is larger. Combined with the data statistics from Table <ref> and the distance margin distribution in Fig. <ref>, a reasonable explanation is that the Ponzi scheme dataset is more severely affected by data distribution imbalance and label scarcity issues, and higher intensity of contrast self-supervised constraint are needed to help  better distinguish different interaction behavior patterns. Therefore, appropriately increasing the λ setting helps improve the framework's detection performance.
For the phishing scam dataset, its data distribution is relatively balanced, and the label size is relatively abundant compared to the Ponzi scheme dataset, which facilitates effective learning of account behavior representations within . Consequently, only moderate contrastive self-supervision constraint is required to enhance the discriminative capability of our framework.























  




























































































































  





























































































§ CONCLUSION
 
In this work, we introduce , a novel fraud detection framework for Ethereum that integrates fine-grained interaction feature generation, multi-view feature learning, and coarse-grained interaction feature contrast. Our experiments demonstrate that  significantly outperforms existing methods in detecting Ponzi schemes and phishing scams.  offers a substantial advancement in Ethereum fraud detection and provides a foundation for more secure and trustworthy blockchain ecosystems. Future work will refine and extend  to address emerging fraud patterns and more blockchain platforms.



























IEEEtran 




[
    < g r a p h i c s >
]Chengxiang Jin
	received the BS degree in electrical engineering and automation from Zhejiang University of Science and Technology, Hangzhou, China, in 2021. He is currently pursuing the MS degree in control theory and engineering at Zhejiang University of Technology, Hangzhou, China. His current research interests include graph data mining and blockchain data analytics, especially for heterogeneous graph mining in Ethereum.



[
    < g r a p h i c s >
]Jiajun Zhou
	received the Ph.D degree in control theory and engineering from Zhejiang University of Technology, Hangzhou, China, in 2023. He is currently a Postdoctoral Research Fellow with the Institute of Cyberspace Security, Zhejiang University of Technology. His current research interests include graph data mining, cyberspace security and data management.




[
    < g r a p h i c s >
]Chenxuan Xie
	received the BS degree in automation from Nanchang University, Jiangxi, China, in 2021. He is currently pursuing the MS degree in control engineering at Zhejiang University of Technology, Hangzhou, China. His current research interests include graph data mining and graph neural network, especially for graph heterophily problem and graph transformer.




[
    < g r a p h i c s >
]Shanqing Yu
	received the M.S. degree from the School of Computer Engineering and Science, Shanghai University, China, in 2008 and received the M.S. degree from the Graduate School of Information, Production and Systems, Waseda University, Japan, in 2008, and the Ph.D. degree, in 2011, respectively. She is currently a Lecturer at the Institute of Cyberspace Security and the College of Information Engineering, Zhejiang University of Technology, Hangzhou, China. Her research interests cover intelligent computation and data mining.




[
    < g r a p h i c s >
]Qi Xuan(M'18) received the BS and PhD degrees in control theory and engineering from Zhejiang University, Hangzhou, China, in 2003 and 2008, respectively. He was a Post-Doctoral Researcher with the Department of Information Science and Electronic Engineering, Zhejiang University, from 2008 to 2010, respectively, and a Research Assistant with the Department of Electronic Engineering, City University of Hong Kong, Hong Kong, in 2010 and 2017. From 2012 to 2014, he was a Post-Doctoral Fellow with the Department of Computer Science, University of California at Davis, CA, USA. He is a senior member of the IEEE and is currently a Professor with the Institute of Cyberspace Security, College of Information Engineering, Zhejiang University of Technology, Hangzhou, China. His current research interests include network science, graph data mining, cyberspace security, machine learning, and computer vision.



[
    < g r a p h i c s >
]Xiaoniu Yang
  is currently a Chief Scientist with the Science and Technology on Communication
  Information Security Control Laboratory, Jiaxing, China. He is also an Academician of the Chinese Academy of Engineering and a fellow of the Chinese
  Institute of Electronics. He has published the first software radio book Software Radio Principles and Applications [China: Publishing House of Electronics Industry, X. Yang, C. Lou, and J. Xu, 2001 (in Chinese)]. His current research interests are in software-defined satellites, big data for radio signals, and deep learning-based signal processing.





