Paper Title: On the Limitations and Prospects of Machine Unlearning for Generative AI.



[
On the Limitations and Prospects of Machine Unlearning for Generative AI














equal*


Shiji Zhouthu
Lianzhe Wangthu
Jiangnan Yeic
Yongliang Wuseu
Heng Changthu





thuTsinghua University
seuSoutheast University
icImperial College London

Heng Changchangh17@tsinghua.org.cn




Machine UnLearning, Generative AI

0.3in
]










 

§ ABSTRACT 
Generative AI (GenAI), which aims to synthesize realistic and diverse data samples from latent variables or other data modalities, has achieved remarkable results in various domains, such as natural language, images, audio, and graphs. However, they also pose challenges and risks to data privacy, security, and ethics. Machine unlearning is the process of removing or weakening the influence of specific data samples or features from a trained model, without affecting its performance on other data or tasks. While machine unlearning has shown significant efficacy in traditional machine learning tasks, it is still unclear if it could help GenAI become safer and aligned with human desire. To this end, this position paper provides an in-depth discussion of the machine unlearning approaches for GenAI. Firstly, we formulate the problem of machine unlearning tasks on GenAI and introduce the background. Subsequently, we systematically examine the limitations of machine unlearning on GenAI models by focusing on the two representative branches: LLMs and image generative (diffusion) models. Finally, we provide our prospects mainly from three aspects: benchmark, evaluation metrics, and utility-unlearning trade-off, and conscientiously advocate for the future development of this field.



§ INTRODUCTION










    

















“Remembrance is a form of meeting. Forgetfulness is a form of freedom.”Kahlil Gibran (1926)



In an era marked by the burgeoning influence of Generative AI  (GenAI) <cit.>, we are rapidly progressing toward a digital future dominated by AI-generated content. This technological advancement has become a cornerstone in various domains, including natural language processing, image synthesis, audio generation, and graph-based applications. While GenAI heralds an era of innovation and efficiency, it simultaneously raises pressing concerns about data privacy, security, and ethical implications <cit.>.

The training datasets employed in GenAI often contain sensitive information encompassing private, copyrighted, or potentially harmful content <cit.>. This situation raises significant risks of sensitive data leakage <cit.>, directly conflicting with the growing legislative emphasis on the “right to be forgotten” <cit.>. Instances such as the proliferation of copyright infringement cases post the release of models like Stable Diffusion <cit.>, and The New York Times's lawsuit against OpenAI for content leakage[https://nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec2023.pdf], underscore the urgency of addressing these issues.

In response to these challenges, Machine unlearning <cit.> has emerged as a potentially promising solution. Machine unlearning aims to compel models to forget sensitive information, thereby fundamentally eliminating the risk of content leakage. This approach, which seeks to erase sensitive memories directly, stands in contrast to filtering-based solutions that are often susceptible to bypassing or direct attacks. Current research on Machine unlearning spans various generative models, including Large Language Models (LLMs) <cit.>, image generative models <cit.>, and multi-modal generative models <cit.>. These studies have demonstrated machine unlearning's potential in removing elements like copyrighted styles <cit.>, fictional characters <cit.>, and private data <cit.>.

However, the exploration of MU in the context of GenAI is still nascent, with several limitations hindering its full potential. As illustrated in Figure <ref>, we summarise the urgent limitations of the current machine unlearning methods on GAI into three perspectives. Firstly, as an emerging technique, machine unlearning for GenAI is still distant from achieving a level of efficacy requisite for practical applications. Secondly, the evaluation metrics currently employed in MU research are insufficiently robust and fail to capture the multifaceted impact of unlearning on generative models. Thirdly, the side effects of unlearning, including its impact on model performance, generalization, and safety, are significant concerns that must be addressed. As a basis of this position paper, we aim to raise public awareness of these limitations and provide insights that guide future research to address these problems.



In this position paper, we systematically discuss the limitations from the angles of efficacy, side effects, and measurement. Our discussion ranges from LLMs to image generative models. Based on these current inadequacies of unlearning approaches, we advocate future research to focus on three fundamental paths: the benchmarking of unlearning methods, the development of robust and complete evaluating metrics, and the investigation of the balance between utility and unlearning. We believe exploring these three paths should pave the way for further enhancing machine unlearning on GenAI. Overall, our contributions can be summarized as follows:


  * This is, to the best of our knowledge, the first attempt to comprehensively review the limitations of current research on machine unlearning in GenAI. 

  * We offer a systematic examination of the limitations in current unlearning approaches, covering a broad spectrum of models from LLMs to image generative models. This examination provides researchers with a clearer insight into the urgent problems in this area.

  * Tackling these limitations, we propose three prospects based on the examinations, aiming to chart a course for future research in machine unlearning for GenAI.


Through this paper, we seek to contribute to the growing body of knowledge in GenAI, specifically in the area of machine unlearning, and catalyze further research that addresses the critical challenges identified.



§ BACKGROUND



 §.§ Unlearning Formulation
 
Let 𝒟_tr={(x_i)}_i=1^N be the training data where x_i∈𝒳 is the training input. Suppose we have the original model by optimizing towards the training data:

    θ_0 = min_θ∈𝒦𝔼_x∼𝒟_trℒ(θ,x).

Let 𝒟_f⊆𝒟_tr be a subset of training data that is harmful to the model and needs to be forgotten, and 𝒟_r = 𝒟_tr∖𝒟_f be remaining training data of which information we want to retain. The goal of machine unlearning is to successfully unlearn 𝒟_f, and the exact unlearning is to obtain the gold standard model retrained from scratch with only 𝒟_r:

    θ^* = min_θ∈𝒦𝔼_x∼𝒟_rℒ(θ,x).

Exact unlearning can be obtained by retraining, which causes tremendous cost that is not affordable to frequently updating models. In practice, approximate unlearning aims to finetune the original model to obtain the unlearned model θ^u whose output distribution P_θ^u(·) approximates the distribution of gold standard model P_θ^*(·). Unless otherwise indicated, we only discuss the approximate unlearning in the next context in this paper.





 §.§ Unlearning for LLM

Machine unlearning for LLMs is a crucial technique to align LLMs with human preferences and values and to ensure their ethical and responsible use. The existing methods for machine unlearning for LLMs can be broadly classified into:



  
Parameter Optimization Methods. These methods update the model parameters by minimizing a loss function that penalizes the undesirable outputs or behaviors of the model.  <cit.> proposed a gradient-based unlearning method that minimizes the cross-entropy loss between the model outputs and a predefined target distribution for the data samples that need to be unlearned. They applied their method to three scenarios of unlearning for LLMs: removing harmful responses, erasing copyright-protected content, and eliminating hallucinations.



  
Parameter Merging Methods. These methods reduce the model size and complexity by merging or pruning the model parameters that are most affected by the data samples that need to be unlearned. <cit.> proposed the concept of a task vector, which, through arithmetic operations like negation or addition between task vectors, can selectively modify the model’s output with minimal impact on other model behaviors.



  
In-context Learning Methods. These methods modify the model inputs or outputs by adding or removing certain tokens or features that indicate the data samples or modalities that need to be unlearned.  To unlearn a particular instance in the forget set, <cit.> provided the instance alongside a flipped label and additional correctly labeled instances which are prepended as inputs to the LLM at inference time. These contexts are shown to be able to effectively remove specific information in given instances while maintaining comparable performance with other unlearning methods that need to access the LLM parameters.
















 §.§ Unlearning for Image Generative Model

Image generative models have various applications, such as image editing, style transfer, super-resolution, and data augmentation <cit.>. However, image generative models also face challenges and risks, such as violating data privacy, infringing data ownership, and generating inappropriate or misleading images.
Based on the degree of influence removal achieved, the existing methods for machine unlearning for image generative models can be broadly classified into two categories <cit.>:



  
Exact Unlearning Methods. These methods focus on removing the influence of targeted data points from the model through retraining at the algorithmic level completely. It usually involves censoring images from the training dataset such as removing all people's images <cit.> or excluding undesirable classes of data and then performing model retraining <cit.>. The retraining process of large models is often costly, which makes this dataset removal-based approach less practical.



  
Approximate Unlearning Methods. These methods aim to minimize the influence of target data points in an efficient manner through limited parameter-level updates to the model. This approach is post-hoc and efficient to test and deploy. Among them, <cit.> introduced a new concept of weight saliency and used a gradient-based approach to estimate the influential weights and then conduct unlearning accordingly.<cit.> defined the unlearning process from the knowledge perspective and proposed an entanglement-reduced mask (ERM) structure to reduce the knowledge entanglement during training. <cit.> mainly focused on erasing the high-level visual concept from the text-to-image models. <cit.> migrated Elastic Weight Consolidation (EWC) and Generative Replay (GR) from continual learning to perform unlearning effectively. <cit.> formulated the unlearning problem as a bi-level optimization problem and proposed a first-order method to solve it accordingly.



















§ LIMITATIONS
 


 §.§ The Efficacy of Unlearning

We divide the limitations on the efficacy of the current methods into four subsections. These include the discussions on the overarching efficacy limitations, followed by specific efficacy of the LLMs and the image generative models.


	 





  §.§.§ General Weaknesses



  
Dependence on the Original Training Data.
Many of the existing unlearning methods assume that the unlearning targets are a subset of the training set, and therefore require access to the original training data <cit.>. However, it can be difficult to obtain the original training data at times. Training data can be confidential when they are concerned with privacy or contractual issues. For models trained through distributed learning, the model and data are decentralized and can be aggregated from complex sources. Moreover, the training data can often be lost or corrupted due to the limit of storage space, cyber-attacks, or hardware failures. Unlearning methods that are dependent on the training data can not fit into these scenarios, which hinders the generalization of unlearning methods to more real-world applications.



  
Scalability Issue. 
The parameter optimization technique is still the major fundamental idea of the existing unlearning methods for generative models, which involves the iterative updating of parameters <cit.>. This updating process can be computationally expensive and time-consuming, especially when dealing with large-scale models or datasets. Besides, even though the unlearning of target instances can be conducted in a batch-wise manner to improve the efficiency <cit.>, experiments show that a larger batch size brings more degradation to the performance of the model <cit.>. An alternative way is to feed the unlearning instances sequentially, which is beneficial for maintaining the capability of the model <cit.>. The sequential approach compromises system efficiency and poses significant scalability challenges. Particularly in practical applications, there is a frequent necessity to consecutively apply unlearning on the large generative models within constrained timeframes to maintain the relevance and accuracy of models in dynamic environments. Current SOTA methods for unlearning are not effective in addressing scalability issues, which are critical in production.



  
Information Leakage of What was Forgotten. 
Machine unlearning is often used to remove user data for privacy reasons. In this case, users who require the deletion of data may not expect their identity to be identifiable through the requests. In other words, if a model that has gone through the unlearning process still contains clues revealing the identity of the user who is related to the deleted contents, the unlearning should not be considered complete or effective. Nonetheless, this leakage issue has not been comprehensively tested and the risk can be hidden for most of the unlearning methods designed for generative models. Membership Inference Attack (MIA) is a kind of attack specifically designed to test whether the deleted data can be inferred from the output of the unlearned models. Typically, a binary classifier is trained to distinguish the samples in the forgotten set from those in the retained set. A model without information leakage of the deleted data should be able to puzzle the classifier to output the probability close to 0.5 for all samples. Although MIA has been applied to test the unlearning methods in <cit.> showing their robustness against the leakage of the deletion information, this limitation of leakage can still exists in other unlearning methods of generative models.



  §.§.§ For LLMs



  
Task Dependence. 
LLMs learn general representations of both syntactic and semantic knowledge during their pretraining on the large corpus. This enables them to serve as a more general-purpose tool to solve different tasks. The patterns in the representations are intertwined and can be vulnerable when combined with downstream tasks. Catastrophic forgetting is a typical example of such vulnerability, which often occurs during transfer learning. The model can lose its generalization ability and overfit to the target domain in a catastrophic forgetting <cit.>. Thus, when testing the efficacy of an unlearning method for the LLMs, it is important to conduct comprehensive fidelity experiments on datasets from various domains. Nevertheless, most of the existing work tests the retaining and forgetting performance of the unlearning methods on specific datasets <cit.>. The impact of the unlearning process on the model's generalization ability to other tasks is rarely verified. Although the model preserves its performance on the current task, it remains uncertain whether the nuanced modifications in parameters during unlearning force the model to compromise its capability in other tasks. We argue that an effective unlearning method should minimize the performance degradation of the target model on diverse tasks. 



  
Forget or Lie? 
Nowadays, we hold a higher expectation of generative models than before. This gap of expectation is more prominent in terms of LLMs. In the previous era, we mainly focused on improving the fluency and stability of the generation. Therefore, as the unlearning results from <cit.> shown in Figure <ref>, a worse performance or a fabricated description for the sensitive instances could be viewed as a successful unlearning result. However, simply generating incorrect outputs can no longer be enough when we are expecting factual and reliable generation. LLMs are being transformed to function as knowledge bases <cit.> consisting of structural representations of facts and relations. Besides, substantial efforts have been devoted to reducing hallucinations <cit.>. Fake outputs have become increasingly unacceptable after we force the LLMs to forget target knowledge. Contrary to the conventional methods that might lead to a distorted output distribution, an effective unlearning method should teach the model to generate appropriate explanations for the absence of unlearned information. This necessitates an additional objective in the training of unlearning schemes, which should be specially designed to inhibit the models from producing deceptive or hallucinatory responses. This objective aims to maintain the integrity and reliability of the model's output after unlearning.







  §.§.§ For Image Generative Models



  
Visual Perception Inconsistencies.
Human visual systems exhibit high sensitivity to inconsistencies in images. Conversely, image generative models after unlearning tend to generate images with nuanced discrepancies, which are noticeably contradictory to human perceptual norms. For models with deletion of features <cit.>, the discrepancies can be subtle changes in the details of the image including abnormal colors, shapes, or textures of objects and backgrounds. For models trained to unlearn a large block of an image <cit.>, imbalanced foreground and background, unrealistic patterns, or visual artifacts can occur in the images. This kind of abnormal factor is a critical obstacle to the landing of the image generation unlearning techniques and can also be potentially risky for the privacy of users. Thus, it is significant for unlearning methods to minimize the perception inconsistencies in the generated images to become more reliable and effective. 



  
Creation of Visual Bias. 
Some of the unlearning methods for image generation aim to remove certain features or concepts <cit.> in the images. This process requires the unlearned model to recover the part that was occupied by the patterns related to the concept. The limitation is that the recovery can introduce undesired biases that may not have existed in the original image. 

These biases could manifest in the form of gender-specific disparities or patterns indicative of racial biases. Assuming the primary goal of employing concept-removing unlearning is to eliminate biased features, the introduction of other types of biases could bring unpredictable deficiencies to the data, which might be even more challenging to detect and address.



  
Ambiguity in Prompts. 
Image generative models, especially text-to-image models, generate images that share closely matched representations with the corresponding texts in the same latent space. In the unlearning of text-to-image models, textual concepts usually serve as the prompt to guide the removal of visual features. Under this setting, one of the key challenges is to retain the intrinsic similarities between text and image representations. However, some textual prompts can be ambiguous and some concepts may possess vague boundaries with others. The ambiguity may lead to a mismatch between the unlearning prompt and the image features, thereby introducing incomplete unlearning results. As evidence, we find both <cit.> and <cit.> report difficulties in forgetting concepts that are abstract, ambiguous, and intertwined with others. Consequently, the ambiguity in prompts remains a significant challenge for the unlearning of text-to-image models. This issue necessitates further investigation to enhance the efficacy and accuracy of these models in diverse applications.




 §.§ The Side Effect of Unlearning





  
Impact on Utility. 
During the process of machine unlearning, particularly in text-to-image generative models, it becomes evident that forgetting specific concepts can have some negative consequences on the performance of associated concepts <cit.>. This phenomenon is especially pronounced when considering the intricate relationship between artistic styles. For instance, unlearning the style of Van Gogh may inadvertently impact the style of Claude Monet. Previous studies largely overlooked the inclusion of mechanisms to mitigate these negative outcomes and propose proper methods to control these effects <cit.>.




  
Impact on Generalization. Machine unlearning methods usually inversely process the forget set samples, which potentially influences the test performance. On the one hand, a majority method tries to decay the performance of the unlearned model, by flipping the label <cit.> or updating with inverse gradient <cit.>. However, the forget set performance can be viewed as the test performance of the unlearned model. On the other hand, some methods <cit.> aims to align the output distribution of the forget set and an unseen set of the unlearned model. However, the inherent difference between the forget set and the unseen set may lead to mismatching between the two targets. Therefore, forcing the model to inversely learn the forget set samples may impair the generalization of the test set. 






 §.§ The Difficulty of Measuring

Addressing the complexities of evaluating the efficacy of unlearning across different generative scenarios, such as Language Models and image generative models, poses a multifaceted challenge. Each scenario presents unique obstacles, necessitating a tailored approach. After examining individual scenarios, we will explore the overarching challenges that pervade the field of generative model unlearning.





  
Evaluation for Large-scale Models.
Evaluating unlearning in LMs, particularly large-scale models, encounters the challenge of using general metrics like Membership Inference Attacks (MIA) or other classifier-based methods. <cit.> emphasize this difficulty, pointing out the often inaccessible nature of the full training corpus and the complexities in implementing MIA-like methods within LLMs. However, alternative methods such as using fixed models to evaluate unlearning effects also face limitations, for instance, they may struggle to gauge the similarity between unlearned and original model outputs for utility evaluation, indicating a need for further methodological advancements.



  
Evaluation for Diffusion Models.
Current evaluation methods often employ classifier-based approaches and image quality metrics like Frechet Inception Distance (FID) <cit.>. However, these methods also present limitations. For instance, the use of classifiers may not capture the subtle influences of unlearned data comprehensively. While FID is a measure of image quality and utility, it cannot fully evaluate the differences between images pre- and post-unlearning, nor does it guarantee that generated images adhere to intended conditions or assess changes in image content corresponding to unlearned aspects.








  
Lack of Standardized Metrics. There is a absence of universally accepted metrics in each generative scenario. Even within image generative models, where some consensus on methods like classifier-based evaluation for unlearning exists, these metrics are implemented and interpreted differently across studies <cit.>, underscoring the need for standardized approaches.



  
Measuring Residual Information. Lacking universally accepted metrics to capture the residual information for unlearning, and the controversy surrounding the popular use of metrics like MIA and classifier-based methods persists <cit.>. For example, their application in generative models is debated due to inherent limitations. Furthermore, the direct application of intuitively appealing standards like MIA in generative models proves challenging <cit.>.





  
Limited Access to Original Data and Shadow Models. Having adequate access to original data and shadow models can significantly aid in evaluating unlearning <cit.>. However, especially in large-scale generative models, the volume of data often makes this impractical, hindering effective measurement.







  
Condition-Output Alignment. In conditional generative models, maintaining alignment between output and conditions post-unlearning is crucial. For image generative models, tools like CLIP <cit.> offer some solution, but for language models, evaluating whether responses align with prompts remains a non-trivial task.
































§ FUTURE PROSPECTS















 §.§ Towards More Effective Unlearning for GenAI

We provide three directions where future research can focus in terms of improving the efficacy of unlearning for GenAI. 




  
Transferable and Scalable Unlearning.
In the forthcoming period, with the extremely increasing number of data being pooled into the training of large generative models, the demand for the right to be forgotten will inevitably intensify. This can induce the massive use of the unlearning methods on a significant amount of data. In this case, we emphasize that future unlearning methods should be able to effectively adapt to large-scale applications. The unlearning should be agnostic to the statistics of the original data to enable transferability between different tasks. Moreover, instead of retraining the whole model, more attention needs to be paid to parameter-efficient methods that only adjust a small portion of weights. Only lightweight unlearning methods can become prevalent and permeate into any downstream areas where sensitive data could potentially be located. When designing an unlearning algorithm, researchers may need to care more about a fast and accurate tackling of the privacy problem instead of sacrificing efficiency for a minor performance improvement.



  
Unlearning for General Generative Models.
As we are pacing into the era of Artificial General Intelligence(AGI), we have to consider the demand from general generative models when designing unlearning methods. Since the emergence of generalisability usually comes from extreme scaling up at a very high cost, the loss of such ability will not be affordable. We need unlearning methods that maintain the general ability of the target model on all tasks. This certainly should be accompanied by a comprehensive evaluation benchmark, which will be discussed in later sections. Additionally, the unlearning method should also be curated to prevent hallucinations. As aforementioned in <ref>, existing unlearning methods can lead the generative models to output fabricated facts which may require more effort in cleaning. To address this problem, different from knowledge editing which performs precise alterations of the knowledge, in unlearning we may avoid the injection of new knowledge and seek alternative ways to represent the forgetting of knowledge. 



  
Safe Unlearning
The unlearned model should not become more vulnerable to the attack of privacy or bias. Even though, as we mentioned in <ref>, most of the existing methods ignore the necessity to minimize the revealing of privacy or the creation of bias under hostile attacks. We must consider whether the unlearning algorithm will amplify the bias or privacy issues in the original training data and whether the models after unlearning will exhibit excessive reactions to the modified data points. We may consider integrating differential privacy techniques during the unlearning phase, which can provide a mathematical guarantee of privacy protection. Additionally, fairness-aware algorithms could be adapted to monitor and adjust the model's outputs, ensuring that unlearning does not unfairly impact certain groups. By prioritizing the development of such comprehensive approaches, the field can move towards unlearning methods that not only remove data effectively but also uphold the ethical standards required for responsible AI development.






 §.§ Utility-Unlearning Trade-off

As delineated in the preceding section, extant unlearning algorithms substantially impair the performance of the original model. Consequently, in the design of unlearning algorithms, it is imperative to consider the trade-off between model performance and the efficacy of unlearning. In this section, we envisage several potential solutions that may address this Utility-Unlearning Trade-off in the future.


  
Regularization.
Owing to the small size of the forget set, the disparity in parameters between the retrained model and the original model is typically minimal. However, in current unlearning algorithms, there is often a substantial deviation of the unlearned model from the original model due to an overemphasis on unlearning efficacy. This deviation results in the loss of a considerable amount of useful information, thereby substantially diminishing the utility of the model. Consequently, a very direct approach would be to employ various regularization methods to constrain the changes to the parameters during the unlearning process. Techniques from parameter-efficient fine-tuning methods, such as those employed in LoRA <cit.> or Adapter <cit.> methods, could be adapted for this purpose. 

This approach achieves a more optimal Utility-Unlearning Trade-off and facilitates a faster unlearning efficiency.


  
Multi-Objective Optimization.
Merely simplistically applying regularization constraints could yield unforeseen outcomes, and due to the presence of conflicting training objectives, this approach may result in suboptimal unlearning efficiency. An alternative method worth exploring involves the employment of gradient surgery techniques, commonly employed to address conflicting gradients present in the multi-task learning framework <cit.>. By pruning gradients that conflict with the direction of knowledge preservation, it becomes feasible to retain the performance manifestations of the samples in the retained set intact.





 §.§ Evaluating Metrics


In the pursuit of advancing the field of machine unlearning in generative models, it is imperative to address the notable limitations in current evaluative practices. The intricate nature of generative models, ranging from language to image generation, demands a nuanced approach to metric development. This endeavor is not merely a technical challenge but a fundamental requirement to ensure that unlearning processes align with ethical standards, maintain utility, and adapt to diverse applications. The following directions are proposed not only as responses to identified gaps but as strategic advancements that acknowledge the evolving landscape of GenAI.



  
Holistic and Standardized Metrics.The development of holistic and standardized metrics is crucial for creating a uniform framework for evaluating unlearning across different generative models. This approach will facilitate comparative studies and benchmarking, enabling a clearer understanding of the effectiveness of various unlearning methods. By integrating measures of residual information, utility retention, and model integrity, these metrics can provide a comprehensive assessment that is currently lacking in fragmented and scenario-specific evaluations.



  
Advanced Residual Information Assessment.Advanced methods for residual information assessment are essential to address the limitations of current metrics, which often fail to capture the nuanced effects of unlearning. This necessitates the exploration of novel approaches, such as utilizing AI interpretability techniques, to trace and quantify the lingering influences of unlearned data. Such methods could provide deeper insights into the effectiveness of unlearning processes, bridging the gap between theoretical unlearning and its practical implications.



  
Integrated Efficacy-Utility Metric.A nuanced approach involves formulating an integrated metric that encapsulates both unlearning efficacy and utility preservation. This could be operationalized as a composite score, merging domain-specific measures of unlearning with utility indicators (e.g., FID for image models, ROUGE for text models). Adjusting the weightage in the composite score would allow for flexibility depending on domain-specific requirements.



  
Emphasizing Condition-Output Alignment.In conditional generative models, maintaining the fidelity of outputs to their conditions post-unlearning is critical. Developing metrics that rigorously evaluate this alignment is imperative, especially considering the subjective nature of outputs in these models. 



  
Incorporating Global Influence Evaluation.Evaluating the global influence of unlearning is crucial for understanding its broader impact on a model's utility across various tasks and domains. This perspective is vital for ensuring that unlearning specific data does not inadvertently compromise the model's overall performance and applicability. 



  
Addressing Subjectivity in Output Evaluation.
The subjective nature of outputs from generative models necessitates an evaluation framework that incorporates both quantitative metrics and qualitative assessments. This framework is especially important in scenarios where content, style, or ethical considerations play a significant role in the outputs.







  
Cost-efficiency and Scalability Metrics. Finally, the practical aspects of unlearning necessitate metrics that evaluate cost-efficiency and scalability. This is particularly pertinent for large-scale models, where resource constraints play a crucial role. Metrics that assess computational resources, time, and sample efficiency are vital for understanding the feasibility and practicality of unlearning methods, ensuring they are accessible and implementable in diverse contexts.






 §.§ Unlearning Benchmark

Benchmarking is indeed an important aspect of evaluating and comparing different machine unlearning methods for GenAI. However, there are not many existing benchmarks that specifically address this problem. Therefore, there is a need to develop more comprehensive, reproducible, and interpretable benchmarks for machine unlearning in GenAI.









  
Benchmarking LLMs.
It was only recently that the urgent of benchmarking machine unlearning in LLMs aroused the attention of researchers. As the pioneering research, TOFU (Task of Fictitious Unlearning) benchmark <cit.> involves a dataset of 200 synthetic author profiles, each with 20 question-answer pairs, and a subset known as the ‘forget set’ for unlearning.  
TOFU allows for a controlled evaluation of unlearning with a suite of metrics, offering a dataset specifically designed for this purpose with various task severity.

While TOFU is a significant contribution to the field of machine unlearning in GenAI by introducing the first comprehensive benchmark for unlearning in the context of LLMs, it also has some limitations. 

These limitations inspire us for future directions on further benchmarking LLMs from different perspectives: (1) The synthetic nature of the data and the scenarios may not capture the real-world challenges and risks of machine unlearning in LLMs, such as the diversity and complexity of the data sources, the ambiguity and subjectivity in data contents, and the ethical and legal implications of the data ownership and consent. Therefore, real-world datasets especially where the exact retrain set is inaccessible would be a valuable add-on for effective evaluation. 
(2) The evaluation metrics and datasets may not be sufficient or representative of the forget quality and model utility, as they only cover a limited range of tasks and domains with a small quantity. Therefore, more complex tasks that account for the trade-offs and interactions between different metrics and datasets, such as directly forgetting a specific person rather than a set of people, are important to demonstrate the forget quality and model utility. 
(3) The baseline methods may not be adequate or competitive for machine unlearning in LLMs, as they only include four methods that are based on parameter optimization. In the future, more LLM-native techniques such as parameter merging or in-context learning should be considered to make the baselines more comprehensive.



  
Benchmarking Image Generative Models 
While there have been some efforts to address the issue of unlearning in image generative models, a clearly defined benchmark for evaluating this process is yet to emerge <cit.>.

To develop a comprehensive benchmark for the unlearning capabilities of image generative models, it is essential to consider multiple dimensions: 
(1) The diversity of unlearning goals. These should encompass tasks such as eradicating distinct image aesthetics, excising particular objects from scenes, and filtering out content that is not suitable for all users. 
(2) The situations involving the simultaneous unlearning of multiple concepts. Past endeavors in dataset construction primarily focused on unlearning individual concepts. Yet, it holds paramount importance and possesses practical significance to carry out examinations that evaluate the eradication of multiple concepts concurrently. 
(3) The unlearning objective of synonym concepts. For example, the model's competency in unlearning the artistic style of Van Gogh should be tested. This should include the model's ability to dissociate from works like “Starry Night," which, despite not explicitly naming Van Gogh, could still be indicative of his distinctive style.



§ CONCLUSIONS

In this position paper, we have systematically examined the challenges and limitations in the field of machine unlearning within GenAI. Our analysis reveals critical areas requiring attention: efficacy limitations in LLMs and image generative models compounded by issues of scalability, potential data leakage, the side effects on model utility and safety, and the difficulty of measurement design. We posit more effective unlearning for GenAI, the establishment of robust and nuanced evaluation metrics, and a balanced approach to the utility-unlearning trade-off. These steps are crucial for the development of sophisticated benchmarks, and realizing GenAI's full potential while adhering to ethical standards.






§ IMPACT STATEMENTS

Our contribution lays a foundation for future research, emphasizing the need for innovative, responsible strategies in machine unlearning. We assert that addressing these challenges is imperative for the ethical advancement of GenAI, ensuring its alignment with societal values and legal norms. This paper serves as a call to action for the research community to prioritize responsible and effective unlearning methods in the rapidly evolving landscape of GenAI.


icml2024




























 Paper Title: Hilbert curves for efficient exploratory landscape analysis neighbourhood sampling.











Hilbert curves for efficient exploratory landscape analysis neighbourhood sampling
     Johannes J. Pienaar

	Department of Computer Science

	University of Pretoria 

        Pretoria, South Africa 

	 

	
	https://orcid.org/0000-0003-3546-1467
    < g r a p h i c s >
Anna S. Bosman 

	Department of Computer Science

	University of Pretoria 

        Pretoria, South Africa 

	 

 https://orcid.org/0000-0002-6070-2632
    < g r a p h i c s >
Katherine M. Malan 

	Department of Decision Sciences

	University of South Africa 

        Pretoria, South Africa 

	 



    August 5, 2024
======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 

Landscape analysis aims to characterise optimisation problems based on their objective (or fitness) function landscape properties. 
The problem search space is typically sampled, and various landscape features are estimated based on the samples.
One particularly salient set of features is information content, which requires the samples to be sequences of neighbouring solutions, such that the local relationships between consecutive sample points are preserved. Generating such spatially correlated samples that also provide good search space coverage is challenging. It is therefore common to first obtain an unordered sample with good search space coverage, and then apply an ordering algorithm such as the nearest neighbour to minimise the distance between consecutive points in the sample. However, the nearest neighbour algorithm becomes computationally prohibitive in higher dimensions, thus there is a need for more efficient alternatives.
In this study, Hilbert space-filling curves are proposed as a method to efficiently obtain high-quality ordered samples. Hilbert curves are a special case of fractal curves, and guarantee uniform coverage of a bounded search space while providing a spatially correlated sample. We study the effectiveness of Hilbert curves as samplers, and discover that they are capable of extracting salient features at a fraction of the computational cost compared to Latin hypercube sampling with post-factum ordering. Further, we investigate the use of Hilbert curves as an ordering strategy, and find that they order the sample significantly faster than the nearest neighbour ordering, without sacrificing the saliency of the extracted features.














§ INTRODUCTION

Search landscape analysis has established itself as a useful approach for understanding complex optimisation problems and analysing evolutionary algorithm behaviour <cit.>. Many landscape analysis techniques have been developed over the years, with the most widely used techniques including fitness distance correlation <cit.>, local optima networks <cit.>, and exploratory landscape analysis (ELA) <cit.>. When landscape analysis produces numeric outputs, the resulting feature vectors can be used as abstract representations of problem instances, where instances with similar feature vectors are assumed to fall into similar problem classes. If these feature vectors effectively capture the important characteristics of problems, they can be used as the feature component for automated algorithm design (AAD) – specifically, automated algorithm configuration and selection. A number of recent studies have achieved different aspects of AAD using landscape analysis in specific contexts <cit.>. 

Landscape analysis approaches differ in terms of what they measure or predict (e.g. ruggedness, modality, presence of funnels, and so on), and also on what they produce (e.g. numerical results or a visualisation of a phenomenon) <cit.>. They can also be distinguished based on the scale of the analysis <cit.> – a global approach attempts to characterise the features of the search space as a whole, while a local approach will consider the features of the landscape in the neighbourhood of solutions. Fitness distance correlation <cit.> and local optima networks <cit.> are both examples of global approaches, whereas the average length of an adaptive walk <cit.> is an example of a local landscape feature. 

Many landscape analysis techniques that measure local features are based on samples that are spatially correlated, i.e. sequences of neighbouring solutions, as opposed to a sample of independent solutions from the whole search space.  Techniques that require such sequences of sampled solutions include correlation length for measuring ruggedness <cit.>, entropic profiles of ruggedness and smoothness with respect to neutrality <cit.> – adapted as a single measure of ruggedness for continuous spaces <cit.>, approximations of gradient <cit.>, information content features <cit.>, and measures of neutrality <cit.>. 

In the context of numerical optimisation, a wide range of landscape analysis approaches have been implemented in the R package  <cit.>, which has also been re-implemented in Python ([<https://pypi.org/project/pflacco/>]). Over 300 landscape metrics are included in , organised into 17 sets, which include the original ELA metrics <cit.> covering six of the feature sets, and the set of information content features <cit.> consisting of five metrics. In a study of the subset of the “cheap" feature sets in , Renau et al. <cit.> found that the two most salient (in terms of distinguishing between problems) and robust feature metrics were from the information content and ELA meta-model feature sets. Unlike the other feature sets in , the information content metrics require spatially correlated samples.
 The saliency of the information content feature metrics and the additional requirement of neighbourhood ordering is the motivation for this study.  

To date, the two main strategies proposed for generating spatially correlated samples in continuous search spaces are random walks <cit.> and post-factum ordering of global samples <cit.>. Desirable properties of a sampling strategy for estimating local features are that the solutions provide good coverage of the search space <cit.>, successive points are positioned close to each other (compared to other solutions in the sample) to capture landscape changes in the neighbourhood, and that the process has low computational expense. The most commonly used approaches to neighbourhood sampling for numerical optimisation are uniform random sampling or Latin hypercube sampling (LHS) <cit.>, followed by either random or nearest-neighbour ordering. These are the approaches implemented in  and .

In this paper, we investigate the use of Hilbert curves <cit.> as a new sampling strategy. Hilbert curves are fractal space-filling curves with two desirable properties: (1) they guarantee uniform search space coverage, and (2) neighbouring points on the curve are located close to one another. We show that Hilbert curves are comparable to ordered LHS in terms of the saliency of the extracted features, but are significantly cheaper to compute. Additionally, we show that a Hilbert curve can be used to spatially order an LHS sample, resulting in significant computational gains compared to the commonly used nearest neighbour ordering method.   







§ HILBERT CURVES

A space-filling curve is a surjective continuous function from the unit interval [0,1] to a unit hypercube [0,1]^d. Surjectivity implies that every point in the hypercube maps to at least one point in the interval, and continuity ensures that no areas in [0,1]^d are missed <cit.>. Space-filling curves are a special case of fractal curves, and are guaranteed to fill a continuous space in the limit. 

Our interest in space-filling curves derives from two useful properties that they offer, namely (1) uniform coverage of a bounded d-dimensional space, and (2) the ability to provide a unique mapping between points in the d-dimensional space and points on the  1-dimensional curve <cit.>. Specifically, this study considers the Hilbert space-filling curve, first proposed by D. Hilbert in 1891 <cit.>. The Hilbert curve is defined recursively, and can be constructed through a limit process of iteration. Each successive iteration creates an approximation of the true Hilbert curve that passes through more points in the d-dimensional unit hypercube. For practical purposes, the number of iterations is chosen to be finite, and is further referred to as the order of the Hilbert curve. 

Consider the construction of a Hilbert curve in 2D. For the first iteration, a line on the closed interval [0,1] and a square [0,1]^2 are taken. Four equidistant points are selected on the line, where the starting point is at 0 and the end point is at 1. The square is subdivided into four equal parts. Intervals of the line connecting each pair of points are then mapped onto the square such that the intervals adjacent on the line share a common edge on the square. This results in a simple U-shape, illustrated in Figure <ref>(hc:p1). At each subsequent iteration, the curve from the previous iteration is divided into four equal parts. Each part is then shrunk by a factor of 1/2, rotated, and repositioned such that the four curves connect at their endpoints in a U-shaped or reverse U-shaped pattern. Figure <ref> shows 2D Hilbert curves of order 1 to 3. 





The Hilbert curve is bijective, i.e. the mapping between points on the curve and a d-dimensional space is reversible. Bijectivity of space-filling curves has been useful in organising multi-dimensional data storage and retrieval systems, since it allows a linear index to the data to be constructed and searched <cit.>. Ideally, closely related data in the d-dimensional space would also be close together on the 1-dimensional curve. Faloutsos and Roseman <cit.> investigated the clustering properties of Hilbert curves compared to other space-filling curves, and found Hilbert curves to be superior when measuring the average distance (along the curve) to the nearest neighbours of a given point. This finding suggests that Hilbert curves may serve as a viable alternative to nearest neighbour sorting of a sample. 

For the Hilbert curve implementation in this study, we use the  Python package (<https://pypi.org/project/hilbertcurve/>), based on the algorithm presented by Skilling <cit.>. All code used for the experiments conducted in this study can be found at <https://github.com/jpienaar-tuks/MIT807>.



§ HILBERT CURVES AS SAMPLERS


Given a Hilbert curve of a particular order, the vertices can be used as a basis for a sample in the associated multidimensional space. Table <ref> shows how the number of vertices grows with the order of the curve and the dimension of the search space. When sampling for ELA, it is common practice to use sample sizes of 10^2× d to 10^3× d <cit.>. 

Table <ref> highlights in bold the order and dimension combinations where the number of vertices exceeds a common sample budget of 10^3 × d. To remain within a sampling budget, we sub-sample randomly from the set of vertices on a Hilbert curve with a minimum order of 3. 



 §.§ Stochastic sampling using Hilbert curves



Since the generation of a Hilbert curve is deterministic, the following two strategies were evaluated to introduce stochasticity:

    
  * Selecting random points along the edges of the Hilbert curve, illustrated in Figure <ref>(fig:hilbert_along_edges): Given any two sequential vertices, P_i and P_i+1, a new point is selected P_j = rP_i+(1-r)P_i+1, where r ∼ U(0,1). 
    
  * Selecting points near vertices of the curve, illustrated in Figure <ref>(fig:hilbert_near_vertexes): For each vertex P_i, generate a new point drawn from a normal distribution centred on P_i with a standard deviation σ (constrained by the bounds of the search space). The step size of the Hilbert curve before scaling is 1, so σ was empirically chosen to be 0.3 to prevent excessive potential overlap with points generated by neighbouring vertices.




Neighbouring vertices on a Hilbert curve are equidistant, so the sequence of vertices has a constant step size. The two randomisation techniques introduce variation into the step size. 
Figure <ref> shows that 

the randomisation near vertices strategy produces step sizes with a more normal distribution. For all further experiments we have used the randomisation around vertices strategy.



 §.§ Search space coverage


We now investigate the extent to which Hilbert curve sampling covers the search space compared to competing strategies such as Latin hypercube sampling (LHS). To investigate the search space coverage, the following three strategies were used to draw samples from search space [-5,5]^d of sizes n ∈{100d, 316d, 1000d} for dimensions d ∈{5, 10, 20, 30}:

    
  * Hilbert curve: randomly select the required number of points from the curve (without replacement).
    
  * LHS: generate the required sample using a Latin hypercube design <cit.>.
    
  * Random walk: use a simple random walk <cit.> with maximum step size of 1 to generate the sample.

For each sample size and dimension combination, a random uniform sample was drawn as a reference set. Thirty independent runs of each sampling strategy were implemented, and for each run, the Hausdorff distance <cit.> to the reference set was calculated. The Hausdorff distance as a measure of search space coverage was first proposed and investigated by Lang and Engelbrecht <cit.>, and their methodology is followed in this study. Statistical significance tests were performed for each sampling strategy as proposed by Derrac et al. <cit.> and described by Lang and Engelbrecht <cit.>.





Table <ref> shows that the Hilbert curve and LHS have similar Hausdorff distances, implying a similar coverage of the search space, whereas the random walk provides the worst coverage. Lower values were achieved in most cases by the Hilbert curve sampler, confirmed by the statistical significance tests presented in Table <ref>. The null hypothesis is that there is no significant difference between the Hausdorff distances for each of the sampling strategies. Results show that the null hypothesis can be rejected, given the p-values and a significance level of 0.05. We therefore conclude that the Hilbert curve sampler achieves a more uniform search space coverage than LHS.

    
    




 §.§ Computational cost

We now compare the cost of the Hilbert curve as a sampler to alternative sampling strategies. 
Generating the Hilbert curve is not computationally cheap, but we expect that this upfront investment will be justified if the sample is subsequently used to calculate landscape metrics that require ordered samples. 


To evaluate the computational cost, a performance counter[Python's ] was used to keep track of the time to a) generate the sample and b) calculate information content metrics for both the Hilbert curve and LHS. Using the  library, the information content () metrics were calculated for each of the 24 BBOB functions defined as part of the COCO platform <cit.>. This was done for each dimension and sample size listed in Table <ref>. Note that for LHS to be used for information content metrics, the sample needs to be ordered. We evaluated both the random ordering and the nearest neighbour ordering strategies. Also note that the information content function provided by the  library was modified to accept  as an ordering argument to use the pre-ordered Hilbert curve samples.

Figure <ref>(fig:timing:generation) 
shows that the Hilbert curve sampler is slower than LHS to generate a sample of the same size. While the computational cost of the Hilbert curve sampler seems to grow at least quadratically, it remains acceptable even for large sample sizes (just over 1.2 seconds to generate a sample of 30 000 points). 

Figure <ref>(fig:timing:total) 
shows that for the information content metrics, Hilbert curve sampling strategy is substantially faster (especially at larger sample sizes) than LHS with nearest neighbour ordering.























    























 §.§ Predictive performance of Hilbert curve samples



We have shown that Hilbert curve sampling provides an efficient method for generating spatially correlated sequences of solutions from continuous search spaces, and that these sequences provide good coverage of the search space. We now investigate whether the landscape features extracted from these samples provide good predictive performance for algorithm selection. 

Following the approach by Muñoz et al. <cit.>, we use the task of predicting the class of each BBOB function as a proxy for algorithm selection: if the landscape features of function instances can be used to discriminate between problem classes, then they should also be effective predictor variables for algorithm selection. Five target classes corresponding to the BBOB function groupings as outlined in <cit.> were used, namely (1) separable functions: {f_1… f_5}, (2) functions with low or moderate conditioning: {f_6… f_9}, (3) unimodal functions and functions with high conditioning: {f_10… f_14}, (4) multimodal functions with adequate global structure: {f_15… f_19}, (5) functions with low or moderate conditioning: {f_20… f_24}.

The experimental setup is as follows: 24 BBOB functions in dimensions d ∈{2,5,10,20} are used, with an evaluation budget of 1 000 × d. Each BBOB function has 15 predefined instances, and three random instances per function are held out as testing data[This leave-one-instance out approach is used instead of the leave-one-problem out approach due to the small number of classes.]. For each combination of instance and dimension, 1 000d samples are drawn from [-5,5]^d, using three sampling strategies: Hilbert curve (HC), Latin hypercube (LH) and random walk (RW). Four feature sets from the  package were selected that did not require further function evaluations, namely: dispersion (), ELA y-distribution (), ELA meta model (), and information content ().=-1

Table <ref> gives the testing accuracy of a decision tree, k-nearest neighbour, and random forest classifier 

for the task of predicting the function class from the landscape metrics. The  version 1.1.3 default settings were used in all instances. Results show that a random forest model was the most effective at predicting the function class across all sampling strategies. Overall, the RW strategy emerges as the least competitive performer, while the Hilbert curve and Latin hypercube sampling strategies were comparable for all three classifiers.  

    

Table <ref> gives the performance for each of the three sampling strategies based on the four feature sets separately. An observable trend is that both the  and  features are very salient features and allow the classifiers to accurately discriminate between the function classes, confirming results of Renau et al. <cit.>. 

    
 











    




§ HILBERT CURVES AS AN ORDERING TOOL

In this section, we investigate the use of Hilbert curves as an ordering aid for samples generated from other sampling methodologies. 
A Hilbert curve of order p maps each point on a 1-dimensional curve [0,…,2^dp] to a point in [0,…,2^p-1]^d and vice versa, as illustrated earlier in Figure <ref>. The mapping is bijective, and allows for an efficient mapping from a point in the d-dimensional space to a point on the 1-dimensional Hilbert curve. 

This d-D to 1-D mapping can be exploited, given that a Latin hypercube sample of n points divides each axis into n intervals. By selecting the appropriate curve order p, i.e. 
    p=⌈log_2(n+1)⌉,
we can ensure that the resulting Hilbert curve is of sufficient length to accurately map all LHS points to points on the Hilbert curve. The ordering of the points on the Hilbert curve can then be used to provide a spatially correlated ordering of the Latin hypercube sample.

Given that the current best practice of using a greedy nearest neighbour strategy to order LHS requires calculating at least half of the pairwise distance matrix between all points in the sample, the Hilbert curve ordering strategy has the potential to be significantly faster. Furthermore, the nearest neighbour strategy tends to start with relatively short step sizes, which increase as the number of unvisited points decrease. Hilbert curve ordering is likely to provide more consistent step sizes.



 §.§ Step size consistency

Figure <ref> shows a visual comparison of the Hilbert curve and nearest neighbour ordering strategies on a 2D Sphere function sample. The underlying sample X for both strategies is identical, and was obtained using LHS. In the figures the orderings produced by the Hilbert curve (HC) and the nearest neighbour (NN) approach are depicted as a red line. Colour scale is used to indicate the fitness values of the sampled points. It is evident from Figure <ref> that the maximum step size is smaller for HC than for NN. The step size distributions shown in Figures <ref>(fig:step_size_viz:hc) and <ref>(fig:step_size_viz:nn) confirm that the maximum step size of HC is lower, and show a greater skew in the step size distribution for NN.







 §.§ Computational cost

To evaluate the computational cost of the Hilbert curve ordering, we compare it to the nearest neighbour and random (RND) ordering strategies applied to a sample generated using LHS. The Latin hypercube samples of sizes n ∈{100d, 316d, 1000d} were drawn from search space [-5,5]^d for dimensions d ∈{5, 10, 20, 30}. For each configuration, 24 independent samples were drawn, and timed with a performance counter (Python's ). 

Figure <ref> shows the performance comparison between the various ordering strategies. Hilbert curve sampling as discussed in Section <ref> is included for completeness, as it does not require additional ordering. From Figure <ref>, NN ordering performs the worst, especially for large sample sizes. Furthermore, RND sorting and Hilbert curve sampling are equally fast (red and blue curves overlap). Hilbert curve ordering produces an intermediate result, slightly slower than random ordering, but significantly faster than the NN ordering. Since the underlying sample is generated by LHS, Hilbert curve ordering retains all the benefits of LHS for ELA features insensitive to order (such as , or ).






 §.§ Evaluation of features generated using Hilbert curve ordering


We now compare the information content features generated from a Latin hypercube sample using the HC, NN, and RND ordering. Since the feature values are evaluated rather than the computational cost, this experiment was restricted to dimensions d∈{2,5} and sample sizes n∈{100d, 1000d}. A single sample set X was generated and evaluated on each of the BBOB functions to generate corresponding y = f(X) results. Information content metrics were then calculated on these (X, y) pairs using each of the ordering strategies. This procedure was repeated 30 times.

Figures <ref> and <ref> present initial partial information (M_0) and maximum information content (H_max) landscape features based on the entropy of the sample. We consider BBOB problems in 5D (problem indices listed on the x-axis), and calculate feature values based on NN, RND, and HC ordering for n=1000d.

It is evident from Figure <ref> that the HC ordering produces values that lie between RND and NN, but follow a pattern similar to that of NN. It is clear that M_0^RND converges to the same value regardless of the underlying function (confirming the results of  <cit.>). M_0^NN and M_0^HC, on the other hand, produce values that allow for discrimination between functions. Since larger values for M_0 indicate a more rugged landscape (more changes in concavity over the length of the walk), the HC ordering would on average indicate a more rugged landscape than the NN ordering.

According to Figure <ref>, H_max^RND varies over a wider range than M_0^RND. However, the H_max^RND values still tend to cluster around the upper edge of the graph, closer to the maximum value of 1. In comparison, H_max^NN and H_max^HC vary over a larger part of the [0,1] range. Larger values for H_max indicate a larger variety of objects in the landscape (pits, peaks, plateaus). Thus, HC ordering again indicates a more rugged landscape than the NN ordering.















To evaluate the relative saliency of the features produced by the various ordering strategies, permutation-based feature importance technique <cit.> is employed. We train a random forest (RF) classifier using information content features as input ({ϵ_s, 
ϵ_max, ϵ_0.5, H_max and  M_0}, as defined in <cit.>), and the function class as output (as defined in <cit.> and used in Section <ref>). Feature saliency is gauged by randomly permuting each input feature in turn across the dataset, and observing the corresponding reduction in RF accuracy. For this purpose, the features are grouped by the ordering strategy, and a RF is trained on 2/3 of the data, with the remaining 1/3 held out for testing.  

After training, the base accuracy of the RF classifier is evaluated on the held out data. The accuracy of the RF classifier is then re-evaluated as each feature in the held-out dataset is randomly permuted in turn, and the difference to the base accuracy is recorded. This procedure is repeated 10 times for each sampling strategy. The results are shown in Table <ref>. 

Table <ref> shows that  ϵ_s was the most salient feature across all sample orderings, with the RF trained on the HC ordered features relying on ϵ_s more heavily than the RF trained on the NN or RND ordered features. RND sample ordering made the M_0 feature near useless for the classifier, in line with the observations made in Figure <ref>. Conversely, RF trained on the NN ordering relied on M_0 more than RF trained on the HC ordered features. Overall, HC ordering yielded marginally better RF performance compared to the NN and RND ordering approaches.






    






§ CONCLUSION

This paper proposed the use of Hilbert space-filling curves in the context of optimisation problem landscape analysis for the purpose of (1) sampling the search space in a spatially correlated manner that also guarantees uniform coverage, and (2) spatially ordering samples generated using other  sampling algorithms such as the Latin hypercube. Experiments were conducted to evaluate the relative computational efficiency of the Hilbert curves, as well as the saliency of the landscape features extracted using Hilbert curve sampling and Hilbert curve ordering. In the context of sampling, Hilbert curves were significantly faster than Latin hypercube sampling for the purpose of generating order-sensitive features such as the information content metrics. Features extracted by the Hilbert curves were informative, and allowed for successful discrimination between problem classes. As an ordering tool, Hilbert curves performed significantly faster than the commonly used nearest neighbour ordering, and also yielded salient landscape features. Thus, Hilbert curves present a viable computationally efficient alternative to both Latin hypercube sampling, and nearest neighbour ordering of a sample for the purpose of landscape analysis.
 


    



Displayed equations are centered and set on a separate
line.

    x + y = z

Please try to avoid rasterized images for line-art diagrams and
schemas. Whenever possible, use vector graphics instead (see
Fig. <ref>).



<cit.>








splncs04




 Paper Title: Empirical Bayes Linked Matrix Decomposition.



Empirical Bayes Linked Matrix Decomposition]Empirical Bayes Linked Matrix Decomposition










[1]Eric F. Lockelock@umn.edu

*[1]Division of Biostatistics and Health Data Science, School of Public Health, University of Minnesota, Minneapolis, 55455, MN, USA







Data for several applications in diverse fields can be represented as multiple matrices that are linked across rows or columns.  This is particularly common in molecular biomedical research, in which multiple molecular “omics" technologies may capture different feature sets (e.g., corresponding to rows in a matrix) and/or different sample populations (corresponding to columns).  This has motivated a large body of work on integrative matrix factorization approaches that identify and decompose low-dimensional signal that is shared across multiple matrices or specific to a given matrix.  We propose an empirical variational Bayesian approach to this problem that has several advantages over existing techniques, including the flexibility to accommodate shared signal over any number of row or column sets (i.e., bidimensional integration), an intuitive model-based objective function that yields appropriate shrinkage for the inferred signals, and a relatively efficient estimation algorithm with no tuning parameters.  A general result establishes conditions for the uniqueness of the underlying decomposition for a broad family of methods that includes the proposed approach.  For scenarios with missing data, we describe an associated iterative imputation approach that is novel for the single-matrix context and a powerful approach for “blockwise" imputation (in which an entire row or column is missing) in various linked matrix contexts.  Extensive simulations show that the method performs very well under different scenarios with respect to recovering underlying low-rank signal, accurately decomposing shared and specific signals, and accurately imputing missing data. The approach is applied to gene expression and miRNA data from breast cancer tissue and normal breast tissue, for which it gives an informative decomposition of variation and outperforms alternative strategies for missing data imputation. 



















[
    *
    August 5, 2024
==================





§ INTRODUCTION



Low-rank matrix factorization techniques are widely used for many machine learning tasks such as data compression and dimension reduction, denoising to approximate underlying signal, and matrix completion.  Often, instead of a single matrix, the data for a given application takes the form of multiple matrices that are linked by rows and/or columns.  For example, in Section <ref> we consider data from The Cancer Genome Atlas, in which  gene expression (mRNA) and microRNA (miRNA) data are available from different platforms for breast cancer tumor samples and normal breast tissue samples from unrelated individuals.  Thus, the data can be represented as four matrices: (1) mRNA for cancer tissues, (2) mRNA for normal tissues, (3) miRNA for cancer tissues, and (4) miRNA for normal tissues; these matrices share both rows (here, molecular features) columns (here, samples), giving a bidimensionally linked structure. Furthermore, these data have column-wise missingness, in which molecular data for one of the platforms (mRNA or miRNA) is entirely missing for some samples in each of the cancer and control cohorts.  We are interested in imputing the missing data, as well as identifying patterns of variation that are shared or specific to the different data sources and sample cohorts; for these tasks, a low-rank factorization approach that integrates all of the matrices together while accounting for their differences is attractive.  

There is a growing and dynamic literature on multi-matrix low-rank factorization approaches.  The majority of existing methods assume that just one dimension, rows or columns, is shared across matrices, i.e., unidimensional integration. For example, the Joint and Individual Variation Explained (JIVE) approach <cit.> decomposes low-rank structure that is shared across matrices from variation that is specific to each matrix.  Several related approaches yield a similar decomposition using different strategies or assumptions <cit.>, while other approaches such as structural learning and integrative decomposition (SLIDE) <cit.> allow for structure that is partially shared across any subset of matrices.  Such an approach is efficient and useful for interpretation, as for many applications identifying shared, partially shared, and specific underlying structure is of interest.  

The bidimensional linked matrix factorization (BIDIFAC) approach <cit.> was developed to accommodate bidimensionally linked integration scenarios, by identifying structure that is shared or specific across both row sets and column sets.  The objective function for BIDIFAC involves a structured nuclear norm penalty, which approximates the data as a decomposition of low-rank modules that are shared across row or column subsets.  This approach is attractive in the bidimensionally linked context because it has favorable convergence properties and generally leads to a uniquely-defined decomposition <cit.>; methods for unidimensional integration without shrinkage penalties typically require orthogonality <cit.> or other constraints to assure uniqueness, which do not extend to the bidimensionally linked context.  Furthermore, the application of the structured nuclear norm penalty for unidimensional integration (termed  UNIFAC) has been shown to outperform alternative approaches with respect to accurately decomposing shared and specific structures <cit.> and imputing missing data <cit.> under certain scenarios. However, a drawback is that it tends to over-penalize the estimated structure, and is thus biased toward an over-conservative solution with respect to the underlying low-rank structure <cit.>.

For a single matrix the tendency of the nuclear norm to over-shrink low-rank signal has been well-documented, and in contrast low-rank approximations without any further shrinkage tend to overfit <cit.>.  Empirical Bayes approaches to low-rank factorization are an attractive compromise, in which the optimal levels of shrinkage are determined empirically within a model-based framework <cit.>. Under a variational Bayes approach, the true posterior distribution for Bayesian model is approximated by minimizing the free energy (reducing to Kullback-Leibler divergence) under simpler distributional assumptions <cit.>.  In particular, an empirical variational Bayes approach to low-rank matrix factorization under an assumption of normally distributed factor matrices has desirable theoretical properties, and the resulting approximation for a scenario without missing data can be efficiently computed via a closed-form shrinkage formula on the observed singular values of the matrix <cit.>.   

In this article, we describe an empirical variational Bayesian approach to identify and decompose shared, partially shared, or specific low-rank structure in the context of unidimensionally or bidimensionally linked matrices.  Moreover, we describe an accompanying iterative imputation approach analogous to an expectation-maximization (EM) algorithm that can handle entry-wise missing or row/column-wise missing values.  The proposed approach is free of any tuning parameters, and is shown to share desirable properties of the BIDIFAC approach while addressing the issue of over-shrinkage.     




§ NOTATION AND SETTING



Throughout this article bold uppercase characters (`') correspond to matrices and bold lowercase characters (`𝐱') correspond to vectors. Square brackets index the entries within a vector or matrix, e.g., `[m,n]'.  The singular value decomposition (SVD) of a matrix : M × N is given by ___^T, where the diagonal entries of _, _[r,r],  are the singular values of  ordered from largest to smallest for r=1,, min(M,N).  The Frobenius norm is given by ||·||_F, so that ||X||_F^2 is the sum of the squared entries in ; the nuclear norm is given by ||·||_*, which is the sum of the singular values in a matrix: 
    ||||_*=∑_r=1^min(M,N)_[r,r].


  We consider the setting of a collection of matrices  _ij: M_i × N_j for i=1,,I and j=1,,J. The column sets N_1,, N_J and row sets M_1,,M_L are consistent across the matrices.   The subscript `' defines concatenation across row or column sets, and the full data  can be represented as a single M × N  matrix where M = M_1++M_I and N = N_1++N_J:   

    =  [ [ _11 _12     _1J;   ⋮   ⋮   ⋮   ⋮; _I1 _I2     _IJ ] ]    where  _ij  are   M_i × N_j.

As for our motivating application in Section <ref>, the I column sets may correspond to different sample cohorts and the J row sets may correspond to features from different high-dimensional sources.   



§ SINGLE-MATRIX RESULTS


Here we review some established results for a single matrix that are critical to what follows. Propositions <ref> and <ref> describe solutions to the least squares matrix approximation problem under a rank-constrained or nuclear norm penalized criterion, respectively.

  For : M × N, the minimizer of the least squares objective ||X-||_F^2 under the constraint rank()=R ≤min (M,N) is given by =__S _^T where _ is diagonal with _[r,r]=_[r,r] for r=1,,R and _[r,r]=0 for r > R.  




  For : M × N, the minimizer of the nuclear norm penalized objective 1/2||X-||_F^2+λ ||||_* is given by =__S _^T where _S is diagonal with _[r,r]=max(_[r,r]-λ,0) for all r. 


Proposition <ref> is well-known, and a proof of Proposition <ref> can be found in <cit.>.  Because of how they operate on the singular values , Proposition <ref> represents a hard-thresholding and Proposition <ref> a soft-thresholding approach to low-rank approximation.  Proposition <ref> below establishes that the soft-thresholding operator also solves an objective with L_2-penalties on the factor components of , while Corollary <ref> further establishes that it is the posterior mode of a Bayesian model with normal priors on the factor components.  


   For : M × N,

    : M× H,: N× Hmin ||-^T||_F^2+λ (||||_F^2+||||_F^2) = : rank()≤ Hmin ||-||_F^2 +2λ ||||_*

   Moreover, =^T, where  solves the right-hand side of (<ref>) and {,  } solves the left-hand side of (<ref>).


   For : M × N,
if : M × H and : N × H have independent Normal(0, σ^2/λ) entries, and =^T+ where  has independent Normal(0,σ^2) entries, then the mode of the posterior distribution p(,|) satisfies =^T, where  solves the right-hand side of (<ref>).

A proof of Proposition <ref> can be found in <cit.>, and Corollary <ref> follows by noting that the log of the posterior density for the specified model is proportional to the left-hand side of (<ref>).  While the results hold for general H, in what follows we set H as the largest possible rank H= min (M,N); the actual estimated rank will tend to be smaller due to singular value thresholding (as in Proposition <ref>).     The distribution of singular values for the error matrix  has been well-studied, including the following results on the first singular value under general conditions from <cit.>:
    If : M × N has independent entries with mean 0, variance σ^2 and finite fourth moment, then [1,1] ≈σ (√(()M)+√(N)) as M,N →∞; further, if the entries are Gaussian, 𝔼([1,1]) ≤σ (√(()M)+√(N)) for any M,N.
    

These results can motivate choice of the nuclear norm penalty λ.  A conservative approach, if the error variance σ^2 is known, is to select λ = σ (√(M)+√(N)); per Propositions <ref> and <ref>, this will tend to shrink to 0 any singular values in  that result from error with no true low-rank signal.  Another approach is to adopt an empirical Bayes estimate for λ based on the model in Corollary <ref>. However, a general limitation of the nuclear norm approach to low-rank matrix reconstruction is the use of a universal threshold on all singular values; in practice larger singular values should be penalized less, as they capture a relatively lower ratio of error and higher signal  <cit.>.  An analogous interpretation is that the entries of  and  should not have identical variance across columns, and this can be relaxed by a model in which they have column-specific variances: 
[·,r] ∼Normal(,τ^2_,r) and [·,r] ∼Normal(,τ^2_,r) for r=1,…,H.[For most scenarios τ^2_,r and τ^2_,r are not independently identifiable, as the level of signal is determined by their product.  ] 

For given ={τ_,rτ_,r}_r=1^H, the approximation resulting from the posterior mode for p(,|, ) is given by thresholding the singular values of , analogous to Corollary <ref>.  To determine , directly maximizing the joint likelihood p(, , |) over , and  degenerates to → in the global solution, though it can have a reasonable local mode that is a singular value thresholding operator <cit.>.  Further, a marginal empirical Bayes estimator that optimizes p(|) (effectively integrating over  and ) is difficult to obtain computationally or analytically. As an alternative, one can use an empirical variational Bayes (EVB) approach.  In general, for a variational Bayes approach <cit.>, the true posterior of a Bayesian model with parameters Θ and data , p(Θ|) is approximated by another distribution q(Θ) under simplifying assumptions to minimize the free energy

    F(q) = E_q log q(Θ)/p(Θ,) = E_q log q(Θ)/p(Θ|)- log  p(),

where E_q(·) is the expected value with respect to the distribution defined by q(·). 
In the context of low-rank approximation, a useful simplifying assumption for q(·) is that the left and right factor matrices are independent, q(,)=q_u() q_v().  Thus, for hyperparameters  and σ the free energy to minimize is

    F(q |σ, ) = E_q log q_u()q_v()/p(|, , σ) p(|_) p(|_),

and a point estimate for low-rank structure  can be obtained via its expected value under q(·): 

    =E_q (^T).

Under an empirical variational Bayes approach,  and σ are also estimated by minimizing the free energy in (<ref>).  For fully-observed , <cit.> showed that the global solution to the empirical variation Bayes objective can be obtained in closed form as a singular value thresholding operator; their result for fixed σ is given in Theorem <ref>, and for estimation of σ in Theorem <ref>.
 
  The approximation  (<ref>) obtained by minimizing (<ref>) over q and  is ==__S _^T where _S is diagonal with 
 
    _[r,r]=0  if  D_[r,r]<σ√(M+N+√(MN) (κ + 1/κ))
 and 
 
    _[r,r]= (_[r,r]^2-(M+N) σ^2+√((_[r,r]^2-(M+N)σ^2)^2-4MNσ^4))/(2_[r,r])  otherwise.

The value κ is given as the zero-cross point of the function
 
    f(x)=x √(N/M)   log ( x√(M/N)+1 ) +x√(M/N)  log ( x√(N/M)+1 ) -1.

 
 
  
  Assume without loss of generality that M>N and α=N/M. The estimate σ̂ that minimizes (<ref>) is the global minimizer of  
 
    Ψ(σ)=∑_r=1^N ψ_1 (_X[r,r]^2/M σ^2),

 where 
    ψ_2(x) = x-log x+1_{x>c}ψ_2(x),

 
    ψ_2(x) = log  (√(α)ψ_3(x)+1)+αlog  (ψ_3(x)/√(α)+1)-αψ_3(x),

 and 
 
    ψ_3(x)=1/2 √(α)(x-1-α + √((x-(1+α))^2-4 α)).



Together, Theorems <ref> and <ref> provide a powerful and efficient way to approximate low-rank signal without any tuning parameters.  The only substantial computing requirement is in obtaining the SVD of , as other steps involve univariate functions that are trivial to solve.    

         



§ BIDIMENSIONAL LINKED MATRIX FACTORIZATION


BIDIFAC+ <cit.> was developed to decompose bidimensionally linked data as in (<ref>) into modules of low-rank structure that are shared across column sets and/or row sets.      That is,

    _≈∑_k=1^K ,
  
  where each  is a concatenation of blocks ^(k)_ij as in (<ref>) and 
      
    ^(k)_ij = _M_i × N_j   if [i,k] = 0  or [j,k] = 0
    _i^(k)_j^(k) T   if [i,k] = 1  and  [j,k] = 1 .

  Here,  and  are binary matrices that indicate the presence of module k () across the row and column sets; that is,  is low-rank structure specific to the submatrix  defined by the row sets identified in [,k] and the column sets identified in [,k].  This general framework subsumes several other integrative decompositions of structured variation. For example, in the unidimensionally linked case with shared columns J=1, JIVE <cit.> identifies components that are either shared across all matrices ([i,k]=1 for all i) or specific to a single matrix ([i,k]=1 for just one i), while SLIDE <cit.> allows for components that are shared across any number of matrices.  The BIDIFAC  method <cit.> is a special case in which modules are either globally shared ([i,k]=[j,k]=1 for all i,j), shared across all rows for a given column set ([i,k]=1 for all i and [j,k]=1 for one j), shared across all columns for a given row set ([j,k]=1 for all j and [i,k]=1 for one i), or specific to one matrix ([i,k]=1 and [j,k]=1 for just one i,j). 
  
 The BIDIFAC and BIDIFAC+ approaches solve a structured nuclear norm objective: 


    1/2||_-∑_k=1^K  ||^2_F + ∑_k=1^K λ_k ||||_*,

subject to ^(k)_ij = _M_i × N_j if  [i,k] = 0 or [j,k] = 0.  For fixed  and , objective (<ref>) may be solved with an iterative soft-singular value thresholding algorithm that cycles through the estimation of each   with applications of Proposition <ref>. For smaller I and J,  and  can enumerate all possible submatrices of the row and column sets; the solution may still be sparse, with = if the submatrix corresponding to module k has no particular shared structure.  Alternatively, if the number of row and column sets I and J are large, <cit.> describe an algorithm to adaptively estimate  and  to give the submatrices with the most significant low-rank structure during estimation.

The choice of the module-specific penalty parameters λ_k is critical.  By default, λ_k are selected by an approach motivated by Proposition <ref>; under the assumption that each matrix _ij has error variance 1, λ_k depends on the total number of non-zero rows and columns in its submatrix,

    λ_k = √(∑_i=1^I [i,k] M_i)+ √(∑_j=1^J [j,k] N_j),

for k=1,,K.  This approach will shrink to 0 any singular values in the submatrix that are due to error only.  In practice each data matrix _ij is scaled to have error variance 1 beforehand; if the error variance is unknown, then it can be taken as the total variance in _ij (a conservative choice) or via other approaches such as the median absolute deviation estimator in <cit.>.





§ EMPIRICAL VARIATIONAL BIDIFAC (EV-BIDIFAC)



While the BIDIFAC+ approach has several advantages, one weakness is the required standardization by the noise variance in each data matrix, which is estimated pre-hoc and not within a cohesive framework.  More critically, the use of nuclear norm penalties generally results in over-shrinkage of the signal, and the default theoretically-motivated choice of the λ_k's is particularly conservative.  Thus, we aim to develop a more flexible empirical variational Bayes approach to the decomposition of bidimensionally linked matrices, analogous to that for  a single matrix in Section <ref>.  Our psuedo-Bayesian model is as follows for i=1, I and j=1,,J: 


    _ij = ∑_k=1^K _i^(k)_j^(k) T + _i,j where _i,j[l,m] iid∼Normal(0, σ^2_ij), 
    _i^(k)[·,r]= if [i,k]=0 and  _i^(k)[·,r]∼Normal(, σ_ij^2_u[k,r] )  if [i,k]=1, 
    _j^(k)[·,r]= if [j,k]=0 and  _j^(k)[·,r] ∼Normal(, σ_ij^2_v[k,r]  )  if [j,k]=1,

where  is an identity matrix of appropriate dimension. 
 Using repeated applications of Corollary <ref>, the BIDIFAC+ objective can be shown to give the posterior mode of the model specified in (<ref>) for which ={σ_ij: i=1,,I, j=1,,J} and ={_u, _v} are fixed, and ^2_u[k,r]=^2_v[k,r]=1/λ_k for all r.  We extend this framework by allowing  and  to be estimated within a cohesive empirical variational Bayes framework.  We approximate the true posterior with a distribution q(·), for which the factor matrices are mutually independent: 
 
    q(,)=∏_k=1^K q_u,k(^(k)) q_v,k(^(k)).

The corresponding free energy is given by 

    F(q |, ) = E_q log∏_k=1^K q_u,k(^(k)) q_v,k(^(k))/p(|, , ) p({^(k)}_k=1^K |_) p({^(k)}_k=1^K |_),

with the densities p(·) in the denominator as specified in model (<ref>).  The resulting low-rank fit and decomposition is then given by 

    _= E_q ( ∑_k=1^K _·^(k)_·^(k) T) = ∑_k=1^K _^(k)
 
where for k=1,,K,

    _^(k) = E_q_u,q_v( _·^(k)_·^(k) T).


Algorithm <ref> describes an iterative approach to compute the empirical variational Bayes solution.  This algorithm converges to a blockwise minimum of the free energy (<ref>) over q and  with  fixed at the distinct minimizer of (<ref>) for each _ij separately.  Alternatively, one can update the variances  over the iterative algorithm to minimize free energy under the joint model.  In Algorithm <ref>, the number of modules K and the row- and column-set inclusions for the modules,  and , are assumed to be fixed.  For a smaller number of linked matrices, one can set  and  to enumerate all of the possible submatrices of shared, partially shared, or specific structure: K=(2^I-1)(2^J-1).  The estimated decomposition may still be sparse in that = if module k has no distinct low-rank structure, as demonstrated in Section <ref>.  If enumerating all submatrices of the row- and column-sets is not feasible due to a large number matrices, one can specify K,  and  based on prior knowledge. Alternatively,  we may approximate the fully enumerated decomposition with a large upper bound on the number distinct modules, K, and the row- and column-set inclusions  and  estimated empirically as in <cit.>.                
   
  
  



§ UNIQUENESS


  
  The uniqueness of decompositions into shared, partially shared, and specific low-rank structure requires careful consideration, as the terms in the model are generally not uniquely defined in general without further constraints.  For unidimensional decompositions of low-rank structure, conditions of orthogonality are commonly used to assure the different components are uniquely identified <cit.>; however, such conditions are not straightforward to extend to decompositions for bidimensionally linked data.  Alternatively, <cit.> showed that the terms in the BIDIFAC+ solution are uniquely defined under certain conditions, without orthogonality, as the minimizer of the structured nuclear norm objective in (<ref>).     Here we extend this result to a much broader class of penalties, that includes the EV-BIDIFAC decomposition.   
  
   For fixed row-set and column-set indicators for the modules,  and , let  be the set of possible decompositions that yield a given approximation _:

    = {|_ = ∑_k=1^K }.
  
In most setting the cardinality of || is infinite without further conditions.  Define P_ (·) for a vector  as a weighted sum of the singular values of a matrix:

    P_ () = ∑[r] _[r,r],

and let (·) give a structured penalty on the different modules of a decomposition,:

    () = ∑_k=1^K  P__k (),
 
with non-zero entries for each _k.  Theorem <ref> gives sufficient conditions for the minimizer of this penalty to be uniquely determined.   

  
  
Consider ∈ and let _^(k)^(k)_^(k) T give the SVD of  for k=1,,K. The following three properties uniquely identify .


  *  minimizes (·) over ,   

  * {_i^(k)[,r]: [i,k] = 1  and ^(k)[r,r]>0} are linearly independent for i=1, I,  

  * {_j^(k)[,r]: [j,k] = 1  and ^(k)[r,r]>0} are linearly independent for j=1,,J.

 
 
 The proof of Theorem <ref> is given in Appendix <ref>.  Note that the linear independence conditions are straightforward to verify, and are likely to hold in practice if the ranks of the terms in  are small relative  to the matrix dimensions.  As noted in Proposition <ref>, the terms in any accumulation point of Algorithm <ref> also give a coordinate-wise optimum of (·) for a certain set of penalties {_k}_k=1^K, and thus the EV-BIDIFAC solution is unique in this sense.     

  
  
If ∈ is an accumulation point of Algorithm <ref>, then it is a coordinate-wise optimum of (·) over  for some {_k}_k=1^K. 
 
 




§ MISSING DATA IMPUTATION


Now we consider the context in which the data are not fully observed and missing data must be imputed.  Let ℳ index observations in the full dataset  that are not observed: ℳ={(m,n): [m,n]  is missing}.  We develop an Expectation-Maximization (EM) approach for imputation that is analogous to previous approaches described for hard-thresholding <cit.>, soft-thresholding <cit.>, and BIDIFAC+ <cit.>.  The procedure for fixed estimates of the noise variance for each matrix, σ_ij, is described in Algorithm <ref>.  
     

Note that Step 4. in Algorithm <ref> is a partial maximization step,  as the free energy in (<ref>) is conditionally minimized for each module given the current imputed values, while Step 5. is an expectation step replacing the missing data with their expected value (<ref>) under the current model fit. Further, the algorithm can be shown to minimize the free energy over the missing data {[m,n] : (m,n) ∈ℳ}; this result is given as Theorem <ref>, and is appealing because all unknowns (model hyperpameters, parameters and missing entries) are then estimated via optimizing the same unified free energy objective.  

 
 
Algorithm <ref> iteratively minimizes the free energy (<ref>) over q(·), , and {[m,n] : (m,n) ∈ℳ}.    
 
 
  
 
 A proof of Theorem <ref> is provided in Appendix <ref>. For the choice of , we suggest two different approaches, depending on the type of missingness.  If entries are missing from an entire row or columns of a given data matrix _ij (i.e., row- or column-wise missing), then σ_ij can be estimated by the applying Theorem 2 to the complete submatrix {[m,n] : (m,n) ∉ℳ}.  If the matrix is missing entries without entire rows or columns, then we update σ_ij iteratively during Algorithm <ref> as follows at the end of each cycle: 1. apply Theorem 2 to the matrix _ij with currently imputed values to obtain σ'_ij, 2. update σ_ij=σ'_ij(M_iN_j/M_iN_j-|ℳ_ij|) where |ℳ_ij| is the number of missing entries in _ij.    










§ SIMULATIONS


Here we present a collection of simulations to illustrate and compare the proposed approach with respect to various criteria.  We explore several conditions covering different signal-to-noise ratios, missing data scenarios, and linked matrix patterns with shared and specific structure.  Further simulation results (e.g., exploring different matrix dimensions) are available at

<https://ericfrazerlock.com/ev-bidifac-sims.html>.     



 §.§ Single matrix



We first describe a simple simulation on a single matrix to illustrate the empirical variational Bayes approach to matrix factorization as a flexible compromise between a hard-thresholding and a conservative soft-thresholding approach.  We consider a fully observed matrix : 1000 × 100, with = c ^T + where the entries of , : M × 10 and : N × 10 are each independent standard normal, and c is a manipulated value controlling the signal-to-noise ratio.  We consider 10 values of c distributed on a log-uniform scale between c=0.05 and c=1, and generate 10 replications of the simulation for each value of c. For each generated dataset, we apply the following approaches:


  * OPT: The oracle operator on the singular values when the structure =c ^T is known: _OPT=_X _X^T where  is diagonal with entries that solve the OLS problem to minimize ||-_X _X^T||_F^2. 

  * EVB: the empirical variational Bayes approach applying Theorems <ref> and <ref>.

  * HT: The hard-thresholding approach described in Proposition <ref>, using the true rank R=10.

  * NN: The nuclear norm penalized soft-thresholding approach described in Proposition <ref>, with λ=√(M)+√(N) (this correspond to a threshold motivated by Proposition <ref> with true noise variance σ=1).      

For each setting and for each method we compute the relative squared error (RSE) and oracle-normalized squared error (ONSE)
 
    RSE=||-||_F^2/||||_F^2 and ONSE=||-||_F^2/||-_OPT||_F^2.

Figure <ref> shows the mean values for both RSE and ONSE under the different methods as a function of the signal-to-noise c. The hard-thresholding approach tends to overfit the signal, which can lead to particularly poor recovery when the signal is smaller relative to the noise. In contrast, the nuclear norm penalty is appropriately conservative in that it never overfits and performs better for lower signal; however, it tends to over-shrink for higher signal and can have several fold higher error than hard-thresholding.  The EVB approach is a flexible compromise between the two methods, performing comparably or better than both methods for all scenarios and substantially outperforming them for moderate levels of signal.  Moreover, performance of the EVB method is impressively close to the optimal in most scenarios.      



We consider another scenario with low-rank structure in which the underlying components have heterogeneous levels of signal.  First, we simulate '=' '^T where the entries of  : M × 10 and : 10 × N are independent standard normal. Then, we generate the signal as 
    = _S'_S _S'^T
 where _S is diagonal with entries sampled randomly from a log-uniform distribution between 0.05 √(MN) and √(MN), and =+ with entries of  independent standard normal.  Thus, some singular values of the true low-rank structure  may correspond to high signal, others may be of moderate signal, and others may be of weak or undetectable signal.  We generate 100 replications in this case, and for each we apply the EVB, HT, and NN methods described above, as well as the following additional approaches:

  
  * HT-OPT: the oracle hard-thresholding approach, in which the rank is selected that results in minimal squared error between the estimated signal and the true signal.  
  
  * NN-OPT: the oracle soft-thresholding approach, in which the nuclear norm penalty is selected that results in minimal squared error between the estimated signal and the true signal. 

  * RMT: the low-rank reconstruction approach proposed in <cit.>, based on minimizing squared error loss using asymptotic random matrix theory, with the true error variance σ^2=1. 
            
Boxplots showing distribution of RSE and ONSE for recovering the underlying structure across replications are shown in Figure <ref>.  The EVB and RMT methods have performance close to the oracle, and vastly outperform hard-thresholding and nuclear norm penalized approaches.  The relatively poor performance of HT-OPT suggests that singular values benefit from some shrinkage, while the poor performance of NN-OPT indicates that the level of shrinkage should not be uniform across singular values.  
 

 

We also compare missing data imputation performance of the EVB approach using Algorithm <ref>.  For each of the 100 simulated datasets generated, we consider scenarios in which 20%, 50%, or 80% of the entries are randomly held out as missing and imputed.  In addition to EVB imputation, we apply similar EM-imputation methods for the soft-thresholding (corresponding to softImpute of <cit.>) and hard thresholding <cit.> approaches. The RMT approach is not considered here, as it does not have an associated missing data imputation procedure.  For the NN-OPT and HT-OPT approaches, we select the value of λ or R, respectively, that minimizes squared imputation error for the signal of the missing data 

    RSE_miss=∑_(m,n) ∈ℳ ([m,n]-[m,n])^2/∑_(m,n) ∈ℳ[m,n]^2.

For oracle normalized imputation error ONSE_miss, we divide RSE_miss by the RSE for the OPT method in (<ref>).  The results are shown in Figure <ref>, and demonstrate that the EVB imputation approach is stable for different levels of missingness and substantially outperforms the other approaches.






 §.§ Two linked matrices


Here we generate two matrices, _1: 500 × 100 and _2: 500 × 100, that are linked by columns: _=[_1;_2]: 1000 × 100.  We consider this context primarily to allow comparison of the EV-BIDIFAC approach with several other methods that identify shared and unshared low-rank structure in this setting. 

First, we perform an illustrative simulation akin to that in Figure <ref> for a single matrix. We simulate shared structure _0=c _0 _0^T and individual structures  _1=c _1 _1^T _2=c _2 _2^T, where _0: 1000 × 5, _0: 100 × 5, _1: 500 × 5, _2: 500 × 5, _1: 100 × 5, and _2: 100 × 5 each have independent entries from a standard normal distribution. The observed data are then generated as _=_0+[_1;_2]+ where entries of  are independent standard normal. Thus c again controls the signal to noise ratio and we consider 10 values of c distributed on a log-uniform scale between c=0.05 and c=1, and generate 10 replications  for each value of c.  Here we consider estimates for the following methods:


  * EVB: the EV-BIDIFAC approach described in Algorithm <ref>

  * JIVE: the JIVE method implemented in <cit.> for true shared and unshared ranks R=5, R_1=5, R_2=5.

  * UNIFAC: the BIDIFAC method <cit.> for vertical integration using the true noise variance σ^2=1.	

Here JIVE is analogous to HT and UNIFAC to NN for a single matrix, as they involve iterative applications HT or NN, respectively, to the shared and unshared components.  For each method we compute RSE for the overall structure =_0+[_1;_2] as in (<ref>), and also compute the relative decomposition squared error (RDSE) as 

    RDSE=||_0-_0||_F^2+||_1-_1||_F^2+||_2-_2||_F^2/||_0||_F^2+||_1||_F^2+||_2||_F^2.

The results are shown in Figure <ref>. The results for overall RSE are similar to that for a single matrix, with UNIFAC appropriately conservative for lower signal but substantially underperforming JIVE for higher signal due to over-shrinkage.  However, per the RDSE results, UNIFAC can still identify the shared and unshared components more accurately with higher signal – this phenomenon was noted previously <cit.> and is in part due to orthogonality restrictions imposed by JIVE and related methods that are required for uniqueness of the decomposition without shrinkage.  The EVB approach again acts as a flexible compromise between the two methods, and substantially outperforms both for many s2n scenarios, particularly with respect to RDSE.  

 

Now, we compare RSE and RDSE under a scenario with heterogenous signal levels for the factors in each of the shared and unshared structures.  The rank-5 signal matrices _0, _1, and _2 are each generated as in (<ref>) with singular values again generated uniformly from a log-uniform distribution between 0.05√(MN)*0.05 and √(MN) for shared structure and between 0.05√(M_i N) and √(M_i N) for specific structure, and 100 replications are generated for this scenario.  In addition to the EVB, JIVE, UNIFAC methods we also consider the following approaches:


  * SLIDE: the structural learning and integrative decomposition (SLIDE) method <cit.>, with the true number of shared and unshared components (i.e., the ranks) specified.  

  * HNN.OPT: the hierarchical nuclear norm (HNN) approach <cit.>, with tuning parameters selected on a grid to minimize actual RSE for the true signal.   	

 Figure <ref> gives distribution of RSE and RDSE over the replications for the different methods. The EVB approach universally outperforms other methods in terms of both RSE and RDSE, and the benefit is large (by a factor of 2 or more) for both criteria in several cases.  While the HNN method is motivated by a model with shared and unshared low-rank structure, it does not provide an explicit decomposition of the structure and thus it is not shown for RDSE.    

 




 §.§ Bidimensionally linked matrices



Now, we consider the full bidimensionally linked framework in (<ref>), with I=J=2, N_1=N_2=50 and D_1=D_2=500.  Data were generated as 

    =∑_k=1^K  + ,

where the entries of  are independent standard normal.  There are K=9 possible distinct modules of low-rank structure in this setting: globally shared, shared across rows for either of the of the J=2 column sets, shared across columns for either of the I=2 row sets, or structure specific to any of the 4 matrices _ij.  We generate 100 datasets, where for each replication 5 of the 9 possible modules are randomly selected to have non-zero low-rank structure and the others are set to zero, =0.  All non-zero modules are generated to have rank 2 structure on the given submatrix as in (<ref>).  For each simulated dataset, we apply the following methods:


  * EB-BIDI: the approach in Algorithm <ref>, with  and  enumerating all K=9 possible modules.

  * BIDIFAC: the BIDIFAC algorithm described in <cit.> with  and  enumerating all possible modules and using the true noise variance σ^2=1.

  * EB-SEP: the EVB approach applied to each matrix _ij separately 

  * EB-JOINT: the EVB approach applied to the overall matrix .	

 For each method, we compute the overall RSE for _=∑_k=1^K as in (<ref>), and for EB-BIDI and BIDIFAC we compute the RDSE as 
 
    RDSE=∑_k=1^9 ||-||_F^2/∑_k=1^9 ||||_F^2.

 The results are summarized in Figure <ref>.  Overall RSE is substantially better for EB-BIDI vs. other methods; in comparison to EB-SEP and EB-JOINT, this illustrates the advantages of a parsimonious decomposition of the low-rank signal into the different shared and unshared components with respect recovering the overall signals.  Moreover, EB-BIDI has substantially better RDSE compared to BIDIFAC, demonstrating the advantages of the empirical variational Bayes objective with respect to accurately decomposing the structure into its different components.  Moreover, EB-BIDI correctly estimated modules with no true structure (=) as = for all of the 400 total instances (100 total datasets with 4 possible modules set to  in each case); on the other hand, it estimated = in only 19/500=3.8% of instances when ≠ 0, which occured when  was difficult to detect due to a small signal size.  This demonstrates how the approach can correctly recover the underlying rank-sparsity structure, only estimating non-zero structure for a module if it truly has distinct low-rank structure on its given rows and columns.               

 


We extend this simulation design to consider missing data imputation accuracy for the generated datasets. We consider two different missingness scenarios: an entry-wise missing scenario in which 20% of the entries are randomly set to missing in each matrix, and a blockwise missing scenario in which 10% of the columns and 10% of the rows are randomly selected to have all of their entries missing in each matrix.  In addition to imputation using the previous methods, we also consider the NN.OPT and HT.OPT methods as described in Section <ref>. For entrywise imputation the RSE_miss is computed as in (<ref>). For blockwise imputation, the true underlying structure is taken to be the sum of column-shared structure for missing rows and the sum of row-shared structure for missing columns, as this is the only estimable signal when an entire row or column is missing.  That is, letting _^C=∑_k:[·,k]=1 and _^R=∑_k:[·,k]=1, 

    RSE_miss=∑_(m,n) ∈ℳ^R (^C_[m,n]-_[m,n])^2+∑_(m,n) ∈ℳ^C (^R_[m,n]-_[m,n])^2/∑_(m,n) ∈ℳ^R^C_[m,n]^2+∑_(m,n) ∈ℳ^C^R_[m,n]^2,

where ℳ^R indexes row-missing entries and ℳ^C indexes column missing entries. 
The results are shown in Figure <ref>.  The EB-BIDI method substantially and consistently outperforms the other approaches with respect to missing data imputation, and the advantages are particularly large for blockwise missing imputation.  Note that the EB-SEP approach does not estimate any structure for blockwise missing data, as it is not estimable when considering each matrix separately.     
 




§ APPLICATION TO BRCA DATA



Here we describe an application to gene expression (mRNA) and microRNA (miRNA) data for breast cancer (BRCA) from the Cancer Genome Atlas  (TCGA) <cit.>. The data were processed as described in <cit.>, resulting in D_1=500 mRNAs and D_2=500 miRNAs for N_1=660 cancer tumor samples and N_2=86 non-cancerous breast tissue samples from unrelated individuals.  Thus, the data can be represented as bidimensionally linked matrices as in (<ref>), where _11 is mRNA data for cancer samples, _12 is mRNA data for normal samples, _21 is miRNA data for cancer samples and _22 is miRNA data for normal samples.  The decomposition of data into different modules of shared structured variation is very well-motivated in this context, as low-rank structure on any of the K=9 possible modules is plausible and interpretable.  For example, it is reasonable to expect shared structure across mRNA and miRNAs, as miRNAs can regulate mRNA. It is also reasonable to expect some shared structure across tumor and normal samples, as breast tumors arise from normal tissue and so they likely share some patterns of molecular variation.  However, it is also reasonable to expect specific structure in each case, e.g., patterns of molecular variability that are `cancer-specific' (i.e., not present in normal tissue) are of particular interest.  The BIDIFAC approach decomposes such shared and unique structure across both row sets (mRNA/miRNA) and column sets (tumor/normal), while other methods do not. Moreover, as demonstrated in our simulations, the empirical variational (EV) Bayes approach captures underlying low-rank structure more efficiently and accurately than other low-rank approximation methods. Thus, we expect our EV-BIDIFAC approach to decompose the underlying shared and unshared signals in an informative way, and accurately recover the underlying signal in the full dataset _·· better than alternative methods.    




We applied EV-BIDIFAC wherein  and  enumerate all K=9 possible modules, and significant distinct low-rank structure was identified for each module (≠ for all k).  Figure <ref> shows heatmaps of the original data (including missing data for miRNA and mRNA) and the resulting overall fit _=∑_k=1^K.  While all estimated structures are low-rank, it captures the original data well and imputation of the missing columns appears reasonable. Considering the relative amount of variation present in the different modules reveals the extent of shared variation across mRNA and miRNA, and across tumor and normal cells.  Interestingly, only 1.47% of the low-rank signal is in the module shared globally across all submatrices, while 59.8% of the signal is shared across mRNA and miRNA but specific to tumors. This suggests that the substantial coordinated activity between mRNA and miRNA in breast tumors is cancer-specific and not present in the normal cells they derive from.   

We further compare imputation accuracy for data under entry-wise, column-wise or row-wise missingness under a cross-validation scheme.  The accuracy of entry-wise imputation is of interest because it is a good proxy for the accuracy of the the overall fit of the estimated structure.  Column-wise imputation accuracy is of interest for two reasons: (1) the actual data have column-wise missingness, as not all samples have both mRNA and miRNA data in the full TCGA cohort, and (2) it provides a robust measure of the level of association between the two data sources, addressing how well can we predict gene expression with miRNA data, and vice versa.  Row-wise imputation accuracy is also of interest, as it provides a measure of similarity in molecular structure between cancer and normal samples (e.g., how well can we predict levels of a missing gene in cancer samples, given its associations in normal samples).

We create 20 folds in the data, where for each fold 5% of the columns, 5% of the rows, and 5% of the remaining entries in each matrix are randomly withheld as missing. For each fold, the missing data are imputed using the EV-BIDIFAC, BIDIFAC, EB-SEP or EB-JOINT approaches described in Section <ref>. We further consider nuclear norm penalized imputation (NN) and hard-thresholding (HT) using softImpute <cit.>, jointly for the full matrix (NN-JOINT and HT-JOINT) or separately for each submatrix (NN-SEP and HT-SEP).  In each case the nuclear norm penalty is selected via random matrix theory, and the rank for hard-thresholding is determined via cross-validation by optimizing MSE for an additional 5% of randomly withheld entries.  As an alternative to low-rank methods we consider k-nearest neighbor (KNN) imputation using the package impute <cit.> on the full matrix. Lastly, we consider a structured matrix completion approach <cit.> designed to impute a contiguous missing submatrix under a low-rank approximation using the StructureMC package <cit.>. As StructureMC imputes only one submatrix, it does not apply to randomly missing entries, and we hold out one set of rows or columns at a time to impute from the full matrix, with all other entries observed.    In each case, the relative squared error for the true observed data is computed and averaged across the four datasets yielding 

    MRSE_miss=1/4∑_i=1^2 ∑_j=1^2 ∑_(m,n) ∈ℳ_ij (_ij[m,n]-_ij[m,n])^2/∑_(m,n) ∈ℳ_ij_ij[m,n]^2.

The MRSE_miss under different imputation scenarios are summarized in Table <ref>.  The EV-BIDIFAC approach performs relatively well across all scenarios.  For entry-wise missing data EV-BIDIFAC performs much better than BIDIFAC (likely due to over-shrinkage of the signal for BIDIFAC) and similarly to EB-SEP, indicating that structure can be well-estimated by considering each dataset separately; however, separate estimation does not allow for row- or column imputation and does not estimate the shared variation that is of interest.  The MRSE_miss for row-missing data is relatively modest (0.90) compared to entry-wise missing (0.38), indicating that there is little shared information between tumor and normal cells.  The joint approach EB-JOINT on  performs relatively poorly across all scenarios as it tends to overfit shared variation when there  substantial specific variation across platforms (mRNA and miRNA) and cohorts (tumor and normal); this illustrates the benefits of a comprehensive decomposition of shared and specific variation across the constituent matrices.      
  



     

    



§ CONCLUSION AND DISCUSSION



The proposed EV-BIDIFAC approach can be considered as a flexible compromise between hard- and soft- thresholding techniques for the linked matrix decomposition problem, and has impressive performance with respect to 1. estimating underlying low-rank structure, 2. decomposing underlying structure into its shared, partially-shared and specific components, and 3. missing data imputation under a variety of scenarios.  An appealing aspect of the method is that it derives from a unified and intuitive model with a corresponding objective function that does not require any pre-hoc choices or parameters to tune.  In simulation comparisons the proposed approach tended to outperform other methods even when their parameters are set to the true values (e.g., true rank(s) or true error variance) or ideal values (e.g., to minimize error with knowledge of the true structure).  



While we have focused on obtaining a point estimate for underlying structure that minimizes free energy, this could also be used to empirically estimate hyperparameters and initialize a sampling algorithm for the associated fully Bayesian model.  This would capture uncertainty in the underlying factorization, which can be used for multiple imputation of missing values or to propagate uncertainty if the lower-dimensional components are used in subsequent statistical modeling.  A related Bayesian model is described in <cit.>, which also illustrates the importance of uncertainty propagation. 

Often, data take the form of higher-order tensors instead of matrices, and low-rank tensor factorizations are useful to uncover their underlying structure <cit.>. The models considered here may be extended to the tensor setting, e.g., by allowing  factors for each dimension of a tensor to have a distribution analogous to that for  and  in the matrix setting. However, the algorithms are not straightforward to extend, as there is likely not a closed-form solution for a tensor akin to that in Theorem <ref> for a matrix.  Extending this approach to the setting of a single tensor or linked tensors is an interesting    direction for future work.   

      



§ SUPPLEMENTARY MATERIALS
 

Reproducible R scripts for all results presented in this manuscript are available in a supplementary zipped folder.  Software to perform the EV-BIDIFAC method is available at <https://github.com/lockEF/bidifac>. Additional simulation study results are available at <https://ericfrazerlock.com/ev-bidifac-sims.html>.    



§ ACKNOWLEDGEMENT
   

This work was supported by the NIH National Institute of General Medical Sciences (NIGMS) grant R01-GM130622. 





§ PROOFS




 §.§ Proof of Theorem <ref>



The proof for Theorem <ref> follows a similar structure to that for Theorem 1 in <cit.>. We first establish four lemmas, Lemmas <ref>, <ref>, <ref>, and <ref>, which are useful to prove the main result.
 
  Take two decompositions  ∈ and ∈, and assume that both minimize the structured penalty for given {}_k=1^K: 

    ()=()=min   ().

Then, for any α∈ [0,1],

    ||α+(1-α)||_* = α ||||_*+(1-α) ||||_*

for k=1,,K.
 


Because  is a convex space and  is a convex function, the set of minimizers of   over  is also convex.  Thus, 

    ({α + (1-α) }_k=1^K )=min   ().

The result follows from the convexity of P_, which implies that for any two matrices of equal size  and , 

    P_(α + (1-α) ) ≤α P_() + (1-α)P_().
 
  Applying (<ref>) to each additive term  in  gives 

    ({α + (1-α) }_k=1^K )    ≤α() + (1-α) () 
       =  min   ().

Because {α + (1-α) }_k=1^K ∈, the inequality  in (<ref>) must be an equality,  and it follows that the inequality (<ref>) must be an equality for each penalized term in the decomposition.   




Take two matrices  and . If ||+||_*=||||_*+||||_*, and _+ ^T is the SVD of +, then = ^T where  is diagonal and ||||_*=||||_*, and = ^T where  is diagonal and ||||_*=||||_*. 


This result is proved in Lemma 2 of <cit.>.
 



Take two matrices  and . If P_ (+)=P_ () +P_(), and _+ ^T is the SVD of +, then = ^T where  is diagonal and P_()=P_(), and = ^T where  is diagonal and P_()=P_(). 


The result follows from repeated applications of Lemma <ref> to the rank-1 terms of  and .  Note that P_ (+)=P_ () +P_() implies 

    P_(∑_r _[·,r] _[r,r]_[·,r]^T+_[·,r] _[r,r]_[·,r]^T  ) = ∑_r [r] (_[r,r] + _[r,r]),

which requires that 
    || _[·,r] _[r,r]_[·,r]^T+_[·,r] _[r,r]_[·,r]^T ||_*
    =|| _[·,r] _[r,r]_[·,r]^T||_*+||_[·,r] _[r,r]_[·,r]^T ||_*
 
for each r.
 



Take two decompositions  ∈ and ∈, and assume that both satisfy 

    ()=()=min   ().
 Then, = _^(k)_^(k) T and = _^(k)^(k)_^(k) T where 
_^(k): M × R_k and _^(k): N × R_k have orthonormal columns, and
^(k) and ^(k) are diagonal.


The result follows as a direct corollary of Lemmas <ref> and <ref>,  as Lemma <ref> implies P_(+)=P_(|)+P_() for each k, and then Lemma <ref> 
implies the result. 	


Theorem <ref> is then established as follows:










Take two decomposition  and  that satisfy properties 1., 2., and 3. of Theorem 1; we will show that =. For each k=1,,K, write = _^(k)_^(k) T and = _^(k)^(k)_^(k) T as in Lemma <ref>.   Then, it suffices to show that ^(k)[r,r]=^(k)[r,r] for all k,r.  

First, consider module k=1 with [,1]=[1  0  ⋯  0]^T and [,1]=[1   0  ⋯ 0]^T.    By way of contradiction, assume ^(1)[1,1]>0 and ^(1)[1,1]=0.  The linear independence of {_j^(k)[,r]: ^(k)[r,r]>0} and {_j^(k)[,r]: ^(k)[r,r]>0} implies that

    (_) = {_^(k)[,r]: ^(k)[r,r]>0}={{_^(k)[,r]: ^(k)[r,r]>0}.

Thus, ^(1)[,1]] ∈{{_^(k)[,r]: ^(k)[r,r]>0}, and it follows from the orthogonality of ^(1)[,1] and {^(1)[,r], r>1} that 
    _^(1)[,1] ∈{{_^(k)[,r]: ^(k)[r,r]>0  and   k>1}.

Moreover, because _i^(1)= for any i>1 and {_i^(k)[,r]: ^(k)[r,r]>0} are linearly independent it follows that
   
    _^(1)[,1] ∈{_^(k)[,r]: ^(k)[r,r]>0,   k>1,   and [i,k]=0  for any  i>1}.

Note that (<ref>) implies _1^(1)[,1] ∈(_12 + ⋯ + (_1J),
however, this is contradicted by the linear independence of _1^(1)[,1] and {_i^(k)[,r]: ^(k)[r,r]>0, k>1}.  Thus, we conclude that ^(1)[1,1]>0 implies ^(1)[1,1]>0.  Analogous arguments show that ^(k)[r,r]>0 if and only if ^(k)[r,r]>0 for any pair (r,k).  It follows that  {_i^(k)[,r]: ^(k)[r,r]>0  or ^(k)[r,r]>0} are linearly independent for i=1, I, and {_j^(k)[,r]: ^(k)[r,r]>0  or ^(k)[r,r]>0} are linearly independent for j=1,,J.  Thus,   

    ∑_k=1^K _^(k) (^(k)-^(k)) _^(k) T = ∑_k=1^K  -  = _ - _=

implies that ^(k)[r,r]=^(k)[r,r] for all k,r.  


 




 §.§ Proof of Theorem <ref>


It is clear that the partial `M-steps' in Algorithm <ref> each partially optimize the free energy.  It remains to be shown that the missing entries minimize free energy when they are replaced with their conditional expectation under the model. Note that the free energy (<ref>) can be expressed as 

    E_q log(∏_k=1^K q_u,k(_k) q_v,k (_k) ) - E_q log(p({}_k=1^K |_) p({}_k=1^K |_))-E_q log(p({|, ,  ) ).

Consider the last term in the expression, decomposing the missing and non-missing entries of :

    E_q log(p(|, ,  ) )    = E_q log(p({[m,n]: (m,n) ∉ℳ}|, ,  ) p({[m,n]: (m,n) ∈ℳ}|, ,  ) ) 
       = E_q log(p({[m,n]: (m,n) ∉ℳ}|, ,  ) ) + 
                        E_q log(p({[m,n]: (m,n) ∈ℳ}|, , )).

Missing data {[m,n]: (m,n) ∈ℳ} are random variables that are independent and normally distributed given {, } as in (<ref>).  Thus, the last term in (<ref>) can be expressed as follows:

    E_q log(p({[m,n]: (m,n) ∈ℳ}|, , ))    = c_1 + ∑_(m,n) ∈ℳ1/2 ^2[m,n]  E_q ([m,n]-𝐒_[m,n])^2 
       ≥ c_1 + ∑_(m,n) ∈ℳ1/2 ^2[m,n]  E_q(𝐒_[m,n])-𝐒_[m,n])^2 .

Thus, for any given q, the free energy is minimized at _[m,n]=E_q S_[m,n]=_[m,n] for (m,n) ∈ℳ. 
















.








29

#1ISBN #1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<https://doi.org/#1>et al.#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<><#>1#1#1#1#1#1#1#1#1#1#1#1#1#1PreBibitemsHook


[Attias1999]attias1999

Attias, H.:
Inferring parameters and structure of latent variable models by
  variational Bayes.
In: Proceedings of the Fifteenth Conference on Uncertainty in
  Artificial Intelligence,
pp. 21–30.
Morgan Kaufmann Publishers Inc.,
San Francisco, CA, USA
(1999)




[Cai et al.2016]cai2016structured

Cai, T.,
Cai, T.T.,
Zhang, A.:
Structured matrix completion with applications to genomic data
  integration.
Journal of the American Statistical Association
111(514),
621–633
(2016)




[Feng et al.2018]feng2018angle

Feng, Q.,
Jiang, M.,
Hannig, J.,
Marron, J.:
Angle-based joint and individual variation explained.
Journal of multivariate analysis
166,
241–265
(2018)




[Fox and Roberts2012]fox2012tutorial

Fox, C.W.,
Roberts, S.J.:
A tutorial on variational Bayesian inference.
Artificial intelligence review
38,
85–95
(2012)




[Gavish and Donoho2017]gavish2017optimal

Gavish, M.,
Donoho, D.L.:
Optimal shrinkage of singular values.
IEEE Transactions on Information Theory
63(4),
2137–2152
(2017)




[Gaynanova and
  Li2019]gaynanova2019structural

Gaynanova, I.,
Li, G.:
Structural learning and integrative decomposition of multi-view data.
Biometrics
75(4),
1121–1132
(2019)




[Hastie and Mazumder2021]softImpute

Hastie, T.,
Mazumder, R.:
softImpute: Matrix Completion Via Iterative Soft-Thresholded SVD.
(2021).
R package version 1.4-1.
<https://CRAN.R-project.org/package=softImpute>




[Hastie et al.2021]KNN

Hastie, T.,
Tibshirani, R.,
Narasimhan, B.,
Chu, G.:
Impute: Impute: Imputation for Microarray Data.
(2021).
R package version 1.68.0




[Josse and Sardy2016]josse2016adaptive

Josse, J.,
Sardy, S.:
Adaptive shrinkage of singular values.
Statistics and Computing
26,
715–724
(2016)




[Kolda and Bader2009]kolda2009tensor

Kolda, T.G.,
Bader, B.W.:
Tensor decompositions and applications.
SIAM review
51(3),
455–500
(2009)




[Kurucz et al.2007]kurucz2007methods

Kurucz, M.,
Benczúr, A.A.,
Csalogány, K.:
Methods for large scale SVD with missing values.
In: Proceedings of KDD Cup and Workshop,
vol. 12,
pp. 31–38
(2007)




[Lock et al.2013]lock2013joint

Lock, E.F.,
Hoadley, K.,
Marron, J.S.,
Nobel, A.B.:
Joint and Individual Variation Explained (JIVE) for integrated
  analysis of multiple data types.
The Annals of Applied Statistics
7(1),
523
(2013)




[Lock et al.2022]lock2020bidimensional

Lock, E.F.,
Park, J.Y.,
Hoadley, K.:
Bidimensional linked matrix factorization for pan-omics pan-cancer
  analysis.
Annals of Applied Statistics
16(1),
193–215
(2022)




[Liu and Zhang2019]StructureMC

Liu, Y.,
Zhang, A.:
StructureMC: Structured Matrix Completion.
(2019).
R package version 1.0.
<https://CRAN.R-project.org/package=StructureMC>




[Mazumder et al.2010]mazumder2010spectral

Mazumder, R.,
Hastie, T.,
Tibshirani, R.:
Spectral regularization algorithms for learning large incomplete
  matrices.
The Journal of Machine Learning Research
11,
2287–2322
(2010)




[Nakajima and
  Sugiyama2014]nakajima2014analysis

Nakajima, S.,
Sugiyama, M.:
Analysis of empirical map and empirical partially bayes: Can they be
  alternatives to variational Bayes?
In: Artificial Intelligence and Statistics,
pp. 20–28
(2014).
PMLR




[Nakajima et al.2013]nakajima2013global

Nakajima, S.,
Sugiyama, M.,
Babacan, S.D.,
Tomioka, R.:
Global analytic solution of fully-observed variational Bayesian
  matrix factorization.
Journal of Machine Learning Research
14(1),
1–37
(2013)




[Nakajima et al.2012]nakajima2012perfect

Nakajima, S.,
Tomioka, R.,
Sugiyama, M.,
Babacan, S.:
Perfect dimensionality recovery by variational Bayesian pca.
In: Pereira, F.,
Burges, C.J.,
Bottou, L.,
Weinberger, K.Q. (eds.)
Advances in Neural Information Processing Systems,
vol. 25
(2012)




[O'Connell and Lock2016]oconnell2016

O'Connell, M.J.,
Lock, E.F.:
R.JIVE for exploration of multi-source molecular data.
Bioinformatics
32(18),
2877–2879
(2016)




[Park and Lock2020]park2020integrative

Park, J.Y.,
Lock, E.F.:
Integrative factorization of bidimensionally linked matrices.
Biometrics
76(1),
61–74
(2020)




[Rudelson and
  Vershynin2010]rudelson2010non

Rudelson, M.,
Vershynin, R.:
Non-asymptotic theory of random matrices: extreme singular values.
In: Proceedings of the International Congress of Mathematicians 2010
  (ICM 2010) (In 4 Volumes) Vol. I: Plenary Lectures and Ceremonies Vols.
  II–IV: Invited Lectures,
pp. 1576–1602
(2010).
World Scientific




[Shabalin and
  Nobel2013]shabalin2013reconstruction

Shabalin, A.A.,
Nobel, A.B.:
Reconstruction of a low-rank matrix in the presence of Gaussian
  noise.
Journal of Multivariate Analysis
118,
67–76
(2013)




[Schouteden
  et al.2014]schouteden2014performing

Schouteden, M.,
Van Deun, K.,
Wilderjans, T.F.,
Van Mechelen, I.:
Performing disco-sca to search for distinctive and common information
  in linked data.
Behavior research methods
46(2),
576–587
(2014)




[Samorodnitsky
  et al.2024]samorodnitsky2022bayesian

Samorodnitsky, S.,
Wendt, C.H.,
Lock, E.F.:
Bayesian simultaneous factorization and prediction using multi-omic data.
Computational Statistics & Data Analysis
197, 107974 (2024) 




[TCGA Research Network
  et al.2012]cancer2012comprehensive

TCGA Research Network, :
Comprehensive molecular portraits of human breast tumors.
Nature
490(7418),
61
(2012)




[Tang and Allen2021]tang2021integrated

Tang, T.M.,
Allen, G.I.:
Integrated principal components analysis.
Journal of Machine Learning Research
22(198),
1–71
(2021)




[Wang and Stephens2021]wang2021empirical

Wang, W.,
Stephens, M.:
Empirical Bayes matrix factorization.
Journal of Machine Learning Research
22(120),
1–40
(2021)




[Yang and Michailidis2016]yang2015non

Yang, Z.,
Michailidis, G.:
A non-negative matrix factorization method for detecting modules in
  heterogeneous omics multi-modal data.
Bioinformatics
32(1),
1–8
(2016)




[Yi et al.2023]yi2023hierarchical

Yi, S.,
Wong, R.K.W.,
Gaynanova, I.:
Hierarchical nuclear norm penalization for multi-view data integration.
Biometrics
(2023)









 Paper Title: EUDA: An Efficient Unsupervised Domain Adaptation via Self-Supervised Vision Transformer.








Preprint


EUDA: An Efficient Unsupervised Domain Adaptation via Self-Supervised Vision Transformer
    Ali Abedi, Graduate Student Member, IEEE,
        Q. M. Jonathan Wu, Senior Member, IEEE,
        Ning Zhang, Senior Member, IEEE,
        Farhad Pourpanah, Senior Member, IEEE
A. Abedi, Q. M. J. Wu, and N. Zhang are with the Department of Electrical and Computer
Engineering, University of Windsor, Windsor, ON N9B 3P4, Canada (e-mail:
abedi3@uwindsor.ca, jwu@uwindsor.ca, ning.zhang@uwindsor.ca).
F. Pourpanah is with the Department of Electrical and
Computer Engineering, Queen’s University, Kingston, ON K7L 3N6, Canada (e-mail: f.pourpanahnavan@queensu.ca).
Manuscript
    August 5, 2024
=================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Unsupervised domain adaptation (UDA) aims to mitigate the domain shift issue, where the distribution of training (source) data differs from that of testing (target) data. 
Many models have been developed to tackle this problem, and recently vision transformers (ViTs) have shown promising results. However, the complexity and large number of trainable parameters of ViTs restrict their deployment in practical applications. This underscores the need for an efficient model that not only reduces trainable parameters but also allows for adjustable complexity based on specific needs while delivering comparable performance.
To achieve this, in this paper we introduce an Efficient Unsupervised Domain Adaptation (EUDA) framework.
EUDA employs the DINOv2, which is a self-supervised ViT, as a feature extractor followed by a simplified bottleneck of fully connected layers to refine features for enhanced domain adaptation. Additionally, EUDA employs the synergistic domain alignment loss (SDAL), which integrates cross-entropy (CE) and maximum mean discrepancy (MMD) losses, to balance adaptation by minimizing classification errors in the source domain while aligning the source and target domain distributions. The experimental results indicate the effectiveness of EUDA in producing comparable results as compared with other state-of-the-art methods in domain adaptation with significantly fewer trainable parameters, between 42% to 99.7% fewer. This showcases the ability to train the model in a resource-limited environment. The code of the model is available at: https://github.com/A-Abedi/EUDA.




Unsupervised domain adaptation, vision transformers, Max mean discrepancy, self-supervised Learning













§ INTRODUCTION

Unsupervised domain adaptation (UDA) has shown promising results in addressing the domain shift problem, where the distribution of the training data (source domain) differs from that of the test data (target domain), and knowledge transfer <cit.>. The domain shift problem can significantly degrade the performance of conventional deep neural networks (DNNs) when applied directly to the target domain <cit.>. UDA overcomes this issue by leveraging unlabeled data from the target domain to align feature distributions and enhance model adaptation <cit.>.
Additionally, traditional DNNs often rely on a large number of annotated samples, which limits their applicability in conditions where labeling is expensive, such as autonomous driving <cit.>. UDA solves this problem by transferring knowledge from previously labeled data (source domain) to unlabeled data in the target domain, thereby enhancing model performance without the need for extensive new annotations. 

Existing UDA methods can be classified into: (i) adversarial methods, which use generative adversarial networks to generate domain-invariant features <cit.>; (ii) reconstruction methods, which employ encoder-decoder structures to learn invariant features by reconstructing domains <cit.>; (iii) transformation methods, which optimize inputs through transformations to enhance adaptation <cit.>; and (iv) discrepancy methods, which align domain features through statistical metrics like maximum mean discrepancy (MMD) <cit.> and correlational alignment (CORAL) <cit.>. 
Regardless of which UDA method is chosen, the backbone of the network for feature extraction plays a crucial role. The backbone is responsible for extracting features from the input data, and the more generalized and domain-invariant these features are, the better the UDA results will be. Recently, vision transformers (ViTs) <cit.>  have revolutionized computer vision by adapting the transformer architecture, originally designed for natural language processing <cit.>, to process images as sequences of patches. This approach enables ViTs to capture global image dependencies, offering a more comprehensive understanding than the local feature extraction typical of convolutional neural networks (CNNs) <cit.>. This capability makes ViTs well-suited for complex tasks such as object recognition and scene understanding <cit.> and particularly effective in UDA <cit.>. 

The success of ViTs in large-scale pre-training also shows significant performance improvements on benchmarks, particularly in UDA, making the integration of self-supervised learning (SSL) strategies effective <cit.>. SSL allows ViTs to learn detailed representations from vast, unlabeled datasets, enhancing generalization and robustness across various visual domains <cit.>. Among these methods, DINOv2 <cit.> stands out for its ability to extract general-purpose features <cit.>. This positions DINOv2 as a powerful tool for advancing UDA, highlighting its transformative potential in modern visual computing.

Despite the promising performance of the state-of-the-art UDA models, they often follow complex frameworks. Particularly those utilizing ViT architectures as backbone, which involve a significant amount of learnable parameters. These parameters need powerful computational resources for training. This complexity not only increases the cost of deployment but also limits their applicability in environments with limited resources.
Therefore, there is a critical need for UDA models that are inherently simpler and have fewer trainable parameters. Such models would ensure more stable training and better generalization, and offer the versatility to adjust their complexity based on specific task demands and the complexity of various applications. By achieving this, we can facilitate more efficient domain adaptation across a wider range of practical settings, making advanced UDA techniques more accessible and sustainable.

In this paper, we introduce a novel Efficient Unsupervised Domain Adaptation framework, known as EUDA, that utilizes the capabilities of self-supervised ViTs to address the challenges of UDA. Specifically, EUDA integrates DINOv2 as a feature extractor with a simple yet effective bottleneck consisting of fully connected layers. 
Using the fully connected layers aims to refine the features extracted by DINOv2 for effective domain adaptation. Additionally, inspired by <cit.> EUDA uses the synergistic domain alignment loss (SDAL), which combines cross-entropy (CE) loss and MMD loss. This hybrid loss function effectively balances the adaptation by minimizing classification errors on the source domain while aligning the distributions between the source and target domains, as shown in Fig. <ref>. By leveraging the self-supervised pre-training of DINOv2 and the computational efficiency of the MMD, our approach offers a promising solution to UDA. It reduces the dependency on large, complex models and extensive computational resources, making it not only effective but also suitable for real-world applications where adaptability and efficiency are crucial. 
To the best of our knowledge, this is the first application of DINOv2 within a UDA framework, marking a significant advancement in leveraging self-supervised learning techniques for addressing domain adaptation challenges.



In summary, the main contributions of this paper are as follows:


    
  * To the best of our knowledge, we are the first to apply DINOv2, a self-supervised vision transformer, for UDA, utilizing its strength to generate robust, domain-invariant features.
    
  * We propose EUDA (Efficient Unsupervised Domain Adaptation), a novel framework that leverages self-supervised ViTs and MMD to tackle the challenges of UDA. EUDA features a flexible architecture with a simple bottleneck of fully connected layers, which require significantly fewer parameters to be tuned and can be adjusted according to the complexity of the problem. 
    
  * We perform comprehensive experiments and ablation studies to validate the effectiveness of our proposed method on multiple benchmarks.


The rest of this paper is organized as follows: Section <ref> reviews UDA, ViTs, and self-supervised learning approaches within ViTs. Section <ref> details the proposed EUDA model. The experimental results across various configurations are presented in Section <ref>. Finally, Section <ref> concludes the paper with insights and implications of our findings.



§ RELATED WORK




 §.§ Unsupervised Domain Adaptation


As stated in the introduction, the domain shift problem is the major concern in UDA. Over the years, numerous methods have been developed to tackle this issue, aiming to enable effective model performance on the target domain without relying on labeled data <cit.>. 
Adversarial methods use adversarial training to create domain-invariant features. For example, DANN <cit.> leverages a gradient reversal layer that inverts the gradient sign during training. ADDA <cit.> trains a source encoder on labeled images, and mixes source and target images to confuse the discriminator. Adversarial methods are computationally intensive, often exhibit unstable training processes, and do not always guarantee accurate feature mapping.

Transformation methods enhance domain adaptation by pre-processing input samples to optimize their condition for model training. DDA <cit.> preprocesses input data to align signal-to-noise ratios and reduce domain shifts, while TransPar <cit.> applies the lottery ticket hypothesis to identify and adjust transferable network parameters for better cross-domain generalization. These approaches are simple and effective but may not capture all domain differences.

Reconstruction methods utilize an encoder-decoder setup to harmonize features across domains by reconstructing target domain images from source domain data. MTAE <cit.> utilized a multitask autoencoder to reconstruct images from multiple domains. DSNs <cit.> enhance model performance by dividing image representations into domain-specific and shared subspaces. This improves generalization and surpasses other adaptation methods. They usually face challenges with high computational costs, training instability, and potential overfitting to the source domain.

Discrepancy methods have emerged as particularly effective for UDA. DAN <cit.> embeds task-specific layer representations into a reproducing kernel Hilbert space (RKHS) and uses MMD to explicitly match the mean embeddings of different domain distributions. WeightedMMD <cit.> introduces a weighted MMD that incorporates class-specific auxiliary weights to address class weight bias in domain adaptation. This approach optimizes feature alignment between source and target domains by considering class prior distributions. Joint adaptation networks <cit.> align the joint distributions of multiple domain-specific layers across domains using a Joint MMD criterion to improve domain adaptation by considering the combined shift in input features and output labels.



 §.§ Vision Transformer


Transformers, initially introduced by Vaswani et al. <cit.>, have demonstrated exceptional performance across various language tasks. The core of their success lies in the attention mechanism, which excels at capturing long-range dependencies. 
ViT <cit.> represents a groundbreaking approach to applying transformers in vision tasks. It treats images as sequences of fixed-size, non-overlapping patches. Unlike CNNs that depend on inductive biases such as locality and translation equivariance, ViT leverages the power of large-scale pre-training data and global context modeling. ViT offers a straightforward yet effective balance between accuracy and computational efficiency <cit.>.

In the context of UDA, ViTs have demonstrated remarkable potential. TVT <cit.> introduces a transferability adaptation module to guide the attention mechanism and a discriminative clustering technique to enhance feature diversity. CDTrans <cit.> consists of a triple-branch structure with weight-sharing and cross-attention to align features from source and target domains, alongside a two-way center-aware pseudo labeling strategy to improve label quality. WinTR <cit.> uses two classification tokens within a transformer model to learn distinct domain mappings with domain-specific classifiers. This enhances the cross-domain knowledge transfer through source-guided label refinement and single-sided feature alignment. PMTrans <cit.> combines patches from both domains using game-theoretical principles, mixup losses, and attention maps for effective domain alignment and feature learning.

While these methods have shown promising performance in solving UDA problems, they typically rely on complex architectures with extensive trainable parameters and sophisticated training regimes, including multi-branch transformers, cross-attention, adversarial training, game-theoretical principles, and mixup losses. Furthermore, these models generally require training the entire network, resulting in a substantial computational burden. As a result, achieving promising outcomes necessitates extensive training on large-scale models, limiting their practical applicability in resource-constrained environments <cit.>.



 §.§ Self-supervised Vision Transformer

SSL has revolutionized the field of computer vision by enabling models to learn effective representations from unlabeled data, eliminating the dependency on large annotated datasets. These models learn representations by performing pre-text tasks, such as rotation prediction <cit.> and image colorization <cit.>, and then apply the learned representations to downstream tasks.
In the context of ViTs, SSL has been pivotal in enhancing their ability to extract robust, domain-invariant features. Jigsaw-ViT <cit.> integrates the jigsaw puzzle-solving problem into vision transformer architectures for improving image classification. EsViT <cit.> utilizes a multi-stage Transformer architecture to reduce computational complexity and introduces a novel region-matching pre-training task.


Among recent advancements in SSL for ViTs, DINO <cit.> and DINOv2 <cit.> have notably enhanced SSL by scaling up ViTs to effectively match representations across different views of the same image. With improvements like automatic data curation and innovative loss functions, DINOv2 excels in stability and efficiency. It can learn domain-invariant features essential for image classification and other vision tasks. Its capacity to generate robust feature maps makes DINOv2 an excellent choice for a feature extractor and representation generator. Because of these properties, we use DINOv2 in this research to significantly boost performance and generalization across various domains.




§ METHODOLOGY








 §.§ Problem Formulation

UDA aims to learn a function f: 𝒳→𝒴 that performs well on an unlabeled target domain by leveraging information from a labeled source domain. Let 𝒳 and 𝒴 denote the input and label spaces, respectively, 𝒟_s = {(x_i^s, y_i^s)}_i=1^n_s indicates the source domain data, where x_i^s ∈𝒳 and y_i^s ∈𝒴 represent the i^th input-output pairs, and 𝒟_t = {x_j^t}_j=1^n_t indicates the target domain data, where x_j^t ∈𝒳 is the j^th input samples without labels, n_s and n_t are the number of samples in the source and target domains, respectively. The goal of UDA is to train a model on labeled source data 𝒟_s and unlabeled target data 𝒟_t in such a way that it performs well on the predicting target data labels y_i^t.





 §.§ Model Overview

Fig. <ref> shows an overview of our framework for addressing UDA problems. 
The model receives images from labeled source and unlabeled target domains. It then extracts features using a pre-trained self-supervised ViT, while its weights are frozen to ensure stability and efficiency. The extracted features from both domains are fed into a bottleneck composed of multiple fully connected layers to refine and condense the information.
The bottleneck's output serves a dual purpose. Firstly, it inputs into the classification head to compute the CE loss on source domain.
Secondly, it contributes to the computation of the MMD loss to minimize the distance between source and target domain distributions in RKHS to align their feature representations. The pseudocode for our training procedure of EUDA is outlined in Algorithm <ref>. In the following subsections, we present different components of our model in detail.





 §.§ Feature Extractor

In our effort to leverage the capabilities of ViTs for domain adaptation, we adopt DINOv2 <cit.> as a self-supervised pre-trained model for feature extraction. 
It utilizes self-distillation to derive insights from unlabeled data autonomously. Central to DINOv2's design is its twin-network structure, which includes a student and a teacher network. Both networks employ the same underlying architecture based on ViTs. During training, these networks process 
different augmentations of the same image. They aim to extract consistent features regardless of the input variations.

During the training phase (see Fig. <ref>), the student network's parameters are continually updated, while the teacher network's parameters are progressively updated through an exponential moving average of the student's parameters. This ensures that the teacher model remains robust and generalizable. Moreover, DINOv2 uses registers <cit.> to improve the performance and interpretability of ViTs by addressing the problem of artifacts in feature maps, commonly observed in both supervised and self-supervised ViT networks. Registers are additional tokens added to the input sequence of ViTs to absorb high-norm token computations that typically occur in low-information areas of images. 

In our study, we leverage DINOv2 as the primary feature extractor due to its robust training on a large-scale, diverse dataset through self-supervised learning. To enhance efficiency, we freeze the model's parameters. This approach reduces the computational burden and significantly decreases the number of trainable parameters. This makes our method notably more efficient compared to other UDA techniques that require extensive training. Consequently, our streamlined model is well-suited for deployment in real-world scenarios and on-edge devices, where computational resources are often limited. Fig. <ref> shows the attention map of the pre-trained DINOv2 base model without any fine-tuning. This highlights its robustness across four different domains of the office-home <cit.> dataset. This demonstrates the precision and accuracy of the features extracted by DINOv2, underscoring the effectiveness of the EUDA feature extraction approach.

Using DINOv2 as the feature extractor significantly enhances our model by employing its robust, self-supervised pre-training to extract general-purpose features from images. The pre-trained DINOv2, with its weights frozen, ensures the extraction of high-quality features and reduces the number of trainable parameters. It can greatly simplify integration and adaptation in resource-limited settings, providing a strong foundation for effective domain adaptation.





 §.§ Bottleneck


The bottleneck component in our architecture consists of a series of fully connected layers. At the core of our model, the bottleneck output serves two primary functions. Firstly, it feeds into a classification head composed of a simple linear layer. This setup utilizes minimal computational resources to classify images efficiently. Secondly, the output acts as the feature vector for calculating the MMD loss, which aims to minimize the distance between source and target feature vectors. This facilitates effective domain adaptation.

The design of this dual-function bottleneck simplifies our model architecture and enhances its generalization capabilities across various domains. Its straightforward structure helps prevent overfitting, a frequent challenge in domain-specific applications, ensuring robustness for real-world deployments. Additionally, the minimalistic approach in the bottleneck design allows for easier maintenance and adaptability, which is crucial for meeting the dynamic requirements of domain adaptation tasks.

The bottleneck follows a simple yet efficient design for flexible adjustments in the model’s complexity. This flexibility allows us to adjust the model to meet specific performance needs or computational limits, which is particularly useful in settings with restricted resources.



 §.§ Synergistic Domain Alignment Loss

The SDAL is a composite loss function designed for the UDA models. It combines the strengths of CE loss and MMD loss. Formally defined as:

    λℒ_CE + (1-λ) ℒ_MMD,

where ℒ_CE and ℒ_MMD are the CE and MMD losses, respectively, and λ is a tunable hyperparameter that balances the influence of each loss component, it allows for a flexible adjustment according to specific domain adaptation needs. 

CE loss is a widely used loss function in classification tasks.  
The CE loss increases as the predicted probability diverges from the actual label, thus providing a robust metric for optimizing classification models. The mathematical formulation of CE loss for a multi-class classification task is given by:

    ℒ_CE = -1/N∑_i=1^N ∑_c=1^C y_i,clog(ŷ_i,c),

where N is the number of samples, C is the number of classes, y_i,c is a one-hot vector indicating whether class label c is the correct classification for the i^th sample, and ŷ_i,c is the predicted probability of the i^th sample belonging to class c. This formula penalizes the deviation of each predicted class probability from the actual class labels.


MMD loss is defined as the squared distance between the mean embeddings of features from the two domains in an RKHS <cit.>. The MMD loss can be written as:

    ℒ_MMD = MMD^2(𝒟_s, 𝒟_t) = 
    1/n_s∑_i=1^n_sϕ(x_i^s) - 1/n_t∑_j=1^n_tϕ(x_j^t) ^2_ℋ,

where n_s and n_t denote the number of samples in the source and target domains, respectively, x_i^s and x_j^t are the data samples from these domains, and ϕ: 𝒳→ℋ is a feature mapping function that projects the data into RKHS ℋ. By minimizing the MMD loss, we aim to align the statistical properties of the source and target domains.






The loss defined in Eq. (<ref>) leverages the labeled data in the source domain to fine-tune the classification performance via CE loss while aligning the distribution of the source and target domain features through MMD. 
The synergy between these two loss components enhances the model's ability to perform effectively across different domains. This makes SDAL suitable for real-world applications where domain shift is a significant challenge.
Additionally, minimizing the distance between the source and target domain distributions ensures that the feature representations from both domains are closely aligned, facilitating smoother and more reliable domain adaptation. Empirical studies have shown that MMD effectively reduces domain shift and improves model performance across various tasks <cit.>.



§ EXPERIMENTAL STUDIES





 §.§ Datasets

We examine the effectiveness of our proposed method across three benchmarks. These datasets include: 
Office-31 <cit.>, which consists of 4,652 images from three domains: Amazon, Webcam, and DSLR, across 31 categories; 
Office-Home <cit.>, which consists of 15,500 images spread over four domains: Art, Clipart, Products, and Real World, within 65 categories;
VisDA-2017 <cit.>, which includes 280,000 images from four domains: Caltech-256, ImageNet, ILSVRC2012, and PASCAL VOC 2012 as the source domain and real images as the target, belonging to 12 categories; and DomainNet <cit.>, which consists of 48,129 images from six domains: Clipart, Real, Sketch, Infograph, Painting, and Quickdraw, across 345 categories. 











 §.§ Evaluation

To evaluate the effectiveness of our UDA model, following the procedure in <cit.>, We train our model by alternately using each domain as the source with labeled data and another as the target with unlabeled data, then evaluate using labeled data from the target domain. This procedure is repeated until all domains have been used as the source domain. This setup ensures that the model's ability to generalize to new, unseen environments is properly tested. The primary metric for evaluation is accuracy, specifically how accurately the model classifies new samples when trained on the source domain and tested on the target domain. This metric provides a clear measure of the model's performance in bridging the gap between disparate data distributions of the source and target domains.



 §.§ Baselines

We compare our model against a broad spectrum of state-of-the-art models. We categorize these models into ResNet- and ViT-based models. 
The ResNet-based models include RevGrad <cit.>, CDAN <cit.>, TADA <cit.>, SHOT <cit.>, JAN <cit.>, BNM <cit.>, MCD <cit.>, SWD <cit.>, and DTA <cit.>. While ViT-based models include CDTrans <cit.>, PMTrans <cit.>, and TVT <cit.>.



 §.§ Implementation Details

In our domain adaptation model, we designed the bottleneck component with varying complexities to assess how architectural depth influences feature processing. Our configurations ranged from a simple single-layer network with 256 neurons (designated S for Small) to more complex setups like 2048-1024-512-256 (B for Base model), 4096-2048-1024-512-256 (L for Large model), and 8192-4096-2048-1024-512-256 (H for Huge model). We also employed different sizes of the DINOv2 model, base and large, as feature extractors to examine their effects on domain adaptation performance. We used a naming convention in XY format for clarity, where X represents the feature extractor size (B for base and L for large) and Y the bottleneck size. For instance, BB indicates that both the feature extractor and bottleneck are base size.

To optimize domain alignment and classification accuracy, we experimented with different values of the hyperparameter λ, and based on the experiments conducted on Section <ref>, the value of 0.7 was selected for further experiments. Adjustments were made to batch sizes and learning rates according to the model size to ensure computational efficiency on a 1080Ti GPU, with batch sizes set to 32, 16, and 8, and learning rates starting at 3e-2 and gradually decreasing.






 §.§ Comparison with the baseline models

Performance. Tables <ref>, <ref>, <ref> and <ref> show the accuracy rates of EUDA and baseline models on Office-Home, Office-31, VISDA-2017 and DomainNet, respectively. 
As can be seen, our model consistently outperformed all ResNet-based models across all datasets. Specifically, for the Office-Home dataset, our LL model surpassed CDTrans and TVT while delivering comparable results to PMTrans. For the Office-31 dataset, our model exceeded the performance of TVT and matched that of CDTrans and PMTrans. In the case of the VISDA-2017 dataset in Table <ref>, our model performed closely to other SOTA models. For the DomainNet dataset, our findings were particularly striking. Despite the large number of classes the dataset contains, our smallest model configuration, the BS, outperformed most of the SOTA models, including CDTrans, and delivered results comparable to PMTrans. 




Learnable parameters. Table <ref> compares the number of learnable parameters of our proposed EUDA model with baseline models.  EUDA requires significantly fewer trainable parameters compared to ViT-based baseline models while still achieving competitive performance. Specifically, our best model uses approximately 83% fewer parameters for the Office-Home and Office-31 datasets, 43% fewer parameters for the VISDA-2017 dataset, and an impressive 99.7% fewer parameters for the DomainNet dataset. This substantial reduction in model complexity highlights the efficiency of our approach, making it more suitable for deployment in resource-constrained environments without compromising accuracy. 
This indicates a higher efficiency in our modeling approach. EUDA employs a pre-trained DINOv2 model with frozen parameters as a feature extractor, i.e., there are no learnable parameters in this component, and learnability is confined to the bottleneck layers, the normalization layers associated with the feature extractor, and the classification heads. This streamlines the training process and reduces computational overhead, making EUDA particularly suitable for applications where resource efficiency is critical. This efficiency combined with the demonstrated effectiveness of our model in adapting to various domains underscores the practical advantages of our approach in real-world settings.




 §.§ Ablation Study

We test the effectiveness of the SDAL loss on the performance of EUDA model.
To achieve this, we remove MMD loss from Eq. (<ref>), i.e., the model only relies on the source domain and excludes learning from the target domain data. In Table <ref>, we assessed the performance of our BS and BL configurations on the Office-Home dataset using a source-only approach. The results clearly show that incorporating SDAL led to 1.2% and 3.5% improvements in the BS and BL configurations, respectively. Similarly, on the Office-31 dataset (see Table <ref>), our BB configuration improved by +0.4, and BL improved by +0.6 when using MMD loss. For the VisDA-2017 dataset (see Table <ref>), implementing SDAL with our BB and BL models perform approximately 8.1% and 7.4%, respectively better than only using CE loss.



 §.§ Sensitivity Analysis



  §.§.§ Effect of λ


Table <ref> shows the impact of λ in Eq. (<ref>) on the BS configuration for the Office-Home dataset.  In this experiment, we used four different values for λ: 0.3, 0.5, and 0.7. As can be seen, λ = 0.7 managed to produce the best results. The effectiveness of λ = 0.7 comes from its balanced approach that minimizes domain discrepancies through MMD loss while maintaining classification accuracy, ensuring robust performance across various domain shifts.



  §.§.§ Performance of the EUDA Framework Under Various Configurations

In this experiment, we test our model in different configurations across various datasets. Our goal is to find which model best suits each dataset. Our findings indicate that while more complex models perform better on complex datasets, our simpler models, which have significantly fewer trainable parameters, can also achieve comparable results. This flexibility allows users to adjust the model's complexity to match their datasets' specific requirements. This adaptive capability is a distinctive feature of our approach which provides a unique advantage by offering a scalable solution that adjusts to varying data complexities without sacrificing performance. 

Table <ref> demonstrates our model’s performance on the Office-Home dataset with B and L feature extractors and S, B, L, and H bottleneck configurations. Our testing on Office-Home helped us identify optimal configurations, which reducing the need for extensive trials on other datasets. The L feature extractor was notably effective due to its ability to handle the significant domain variance within the dataset, and benefits from higher-dimensional features that capture more informative details. The LL configuration provides the best balance of complexity and performance.

Table <ref> shows the results of Office-31 dataset using B and L feature extractors and B and L bottleneck configurations. It can be seen that LL configuration produces the best results. Insights from the Office-Home tests informed the decision not to use the H bottleneck, as higher complexity had not resulted in improved performance in previous tests.

For the VisDA-2017 dataset (see Table <ref>), the BH configuration stood out, particularly suited to managing the transition from simulation to reality. This confirms the benefit of a more complex bottleneck in handling extensive domain shifts between real and simulation data. This emphasizes the importance of matching architectural choices to specific dataset challenges.









While we have conducted extensive testing across multiple configurations and datasets, time constraints limited the breadth of our experiments. We therefore encourage researchers and practitioners to further explore various bottleneck configurations to meet the specific demands and complexities of their datasets, enabling customized solutions that optimally address their unique challenges.



§ CONCLUSION


In this paper, we highlighted the potential of using a self-supervised pre-trained ViT for UDA by introducing a versatile framework that maintains simplicity, efficiency, and adaptability to ensure its applicability in practical scenarios. Specifically, we leveraged DINOv2, a self-supervised learning method in ViTs,
to extract general features, and employed a simple yet effective bottleneck of fully connected layers to refine the extracted features.
Additionally, we utilized the MMD loss to effectively align the source and target domains. 
Our model produces comparable results to state-of-the-art UDA methods
with significantly fewer trainable parameters. This makes our method particularly suitable for real-world applications, including on-edge devices.
Our proposed framework demonstrates promising results, achieving top-tier performance with 43% to 99.7% fewer trainable parameters across benchmark datasets compared to other methods. In our future research, we will explore additional UDA approaches based on self-supervised pre-trained ViT backbones and expand the applications of UDA in fields such as Autonomous Vehicles and other demanding areas where UDA is crucial.




IEEEtran



    [
    < g r a p h i c s >
]Ali Abedi (Graduate Student Member, IEEE) received his B.Sc. in Computer Software Engineering in 2019 and completed his M.Sc. in Artificial Intelligence Engineering at the University of Isfahan in 2022. 
    He has accumulated extensive academic and industrial experience in the fields of artificial intelligence and machine learning. Currently, he is pursuing a Ph.D. in Electrical and Computer Engineering at the University of Windsor. His primary research interests include deep learning, machine learning, and computer vision, with a particular focus on transformers architecture and vision transformers.



    [
    < g r a p h i c s >
]Q. M. Jonathan Wu (Senior Member, IEEE) received the Ph.D. degree in electrical engineering from the University of Wales, Swansea, U.K., in 1990. He was affiliated with the National Research Council of Canada for ten years beginning, in 1995, where he became a Senior Research Officer and a Group Leader. He is currently a Professor with the Department of Electrical and Computer Engineering, University of Windsor, Windsor, ON, Canada. He has authored or coauthored more than 300 peer-reviewed papers in computer vision, image processing, intelligent systems, robotics, and integrated microsystems. His research interests include 3-D computer vision, active video object tracking and extraction, interactive multimedia, sensor analysis and fusion, and visual sensor networks. Prof. Wu held the Tier 1 Canada Research Chair in automotive sensors and information systems from 2005 to 2019. He is an Associate Editor for IEEE TRANSACTIONS ON CYBERNETICS, the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, the Journal of Cognitive Computation, and Neurocomputing. He was on technical program committees and international advisory committees for many prestigious conferences. He is a Fellow of the Canadian Academy of Engineering.



    [
    < g r a p h i c s >
]Ning Zhang (Senior Member, IEEE) is an Associate Professor in the Department of Electrical and Computer Engineering at University of Windsor, Canada. He received the Ph.D degree in Electrical and Computer Engineering from University of Waterloo, Canada, in 2015. After that, he was a postdoc research fellow at University of Waterloo and University of Toronto, respectively. His research interests include connected vehicles, mobile edge computing, wireless networking, and machine learning. He is a Highly Cited Researcher. He serves as an Associate Editor of IEEE Transactions on Mobile Computing, IEEE Internet of Things Journal, IEEE Transactions on Cognitive Communications and Networking, and IEEE Systems Journal; and a Guest Editor of several international journals, such as IEEE Wireless Communications, IEEE Transactions on Industrial Informatics, and IEEE Transactions on Intelligent Transportation Systems. He also serves/served as a TPC chair for IEEE VTC 2021 and IEEE SAGC 2020, a general chair for IEEE SAGC 2021, a track chair for several international conferences and workshops. He received 8 Best Paper Awards from conferences and journals, such as IEEE Globecom and IEEE ICC.



    [
    < g r a p h i c s >
]Farhad Porpanah (Senior Member, IEEE) received the Ph.D. degree in computational intelligence from the University of Science Malaysia (USM), Malaysia, in 2015. He is currently a Postdoctoral Research Fellow with the Department of Electrical and Computer Engineering, Queens University, Kingston, ON, Canada. Prior to his current position, he was an Associate Research Fellow with the Department of Electrical and Computer Engineering, University of Windsor, Windsor, ON, Canada. From 2019 to 2021, he held the position of Associate Research Fellow with the College of Mathematics and Statistics, Shenzhen University (SZU), Shenzhen, China. Before his tenure at SZU, he was a Postdoctoral Research Fellow with the Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. His research interests include machine learning and deep learning, driven by an emphasis on creating smart environments that improve the quality of life for the elderly population.




 Paper Title: Deep Fréchet Regression.


*These authors contributed equally to this work


#1 1




1
  
  
  
  
  
  
  
  
  
    
  Deep Regression
    Su I Iao^*, Yidong Zhou^*, Hans-Georg Müller

  Department of Statistics, University of California, Davis
    June 11, 2024
=============================================================================================================

 

0
  

    Deep Regression


   

§ ABSTRACT 

Advancements in modern science have led to the increasing availability of non-Euclidean data in metric spaces. This paper addresses the challenge of modeling relationships between non-Euclidean responses and multivariate Euclidean predictors. We propose a flexible regression model capable of handling high-dimensional predictors without imposing parametric assumptions. Two primary challenges are addressed: the curse of dimensionality in nonparametric regression and the absence of linear structure in general metric spaces. The former is tackled using deep neural networks, while for the latter we demonstrate the feasibility of mapping the metric space where responses reside to a low-dimensional Euclidean space using manifold learning. We introduce a reverse mapping approach, employing local regression, to map the low-dimensional manifold representations back to objects in the original metric space. We develop a theoretical framework, investigating the convergence rate of deep neural networks under dependent sub-Gaussian noise with bias. The convergence rate of the proposed regression model is then obtained by expanding the scope of local regression to accommodate multivariate predictors in the presence of errors in predictors. Simulations and case studies show that the proposed model outperforms existing methods for non-Euclidean responses, focusing on the special cases of probability measures and networks.



Keywords:  curse of dimensionality, deep learning, regression, non-Euclidean data, Wasserstein space. 


1.9 


§ INTRODUCTION





Regression analysis serves as the cornerstone of statistical methodology, providing a fundamental framework for modeling complex relationships between response and predictor variables. The demand for comprehensive and versatile regression techniques has soared, driven by the prevalence of non-Euclidean responses. These responses encompass a broad spectrum of data types, ranging from networks <cit.>, covariance matrices <cit.> and trees <cit.> to distributions <cit.> and other complex structures.

Real-world applications across various domains underscore the need for advanced regression methods. For instance, in transportation science, understanding the evolution of transportation networks in response to factors like weather conditions and public health data is of paramount interest <cit.>. These and related scenarios are covered by a general approach where one views the responses as random objects situated in metric spaces <cit.>.  Challenges that we address in this paper include the absence of linear structure and the potential high dimensionality of multivariate predictors.

Regression with metric-space valued responses is thus of paramount interest. Early explorations include regression for non-Euclidean data using Euclidean embedding with distance matrices <cit.> and a Nadaraya-Watson kernel regression approach <cit.>. A regression approach was introduced more recently <cit.>. It extends linear and nonparametric regression techniques to accommodate metric-space valued responses through conditional means. However, these regression models either rely on strict assumptions akin to classical linear models for scalar responses or suffer from the curse of dimensionality in analogy to traditional nonparametric regression.

In practical applications, the need often arises to flexibly model metric-space valued responses with potentially high-dimensional predictors, so that these limitations prevent application of these methods for real-world challenges. Several recent works considered extending sufficient dimension reduction approaches <cit.>, single index models <cit.>, and principal component regression <cit.> to regression to address high-dimensionality of the predictors. These methods, while capable of reducing predictor dimensionality, are confined to the construction of a universal kernel or strong assumptions similar to those for classical single index and linear models.

To address these limitations, we harness deep learning <cit.>, taking advantage of its capability of handling high-dimensional predictors without compromising model flexibility. Deep learning is known for its capacity to automatically extract features from complex data, but has not been developed so far for non-Euclidean responses. The proposed regression model utilizes manifold learning <cit.> for the observed metric-space valued responses, generating low-dimensional representations. Manifold learning is a nonlinear dimensionality reduction technique that has proven useful for dimensionality reduction of infinite-dimensional functional data <cit.>, where data are assumed to lie on a low-dimensional manifold that is unknown. Indeed, many data residing in metric spaces, including parametric families of probability distributions or stochastic block models for networks, are expected to have such a low-dimensional manifold structure; we exploit such structure for the proposed statistical modeling. 

We deploy a deep neural network that takes the observed predictors as input and produces the low-dimensional manifold representation as output. One key innovation of the proposed approach is that we demonstrate that local regression can be utilized to map the low-dimensional manifold representation to the original metric space so that the final output resides in the metric space where the random objects are situated.  This map, which has not yet been considered in the manifold learning literature for metric-space valued data, is crucial for the implementation of a regression model in statistics, as it is mandatory for interpretability to predict the random objects residing in metric spaces, rather than their low-dimensional representations. This comprehensive framework makes it possible to handle high-dimensional predictors while concurrently accommodating the complexities intrinsic to metric-space valued responses.

Our key contributions are as follows. First, we utilize deep learning for regression, highlighting its efficacy as a powerful tool for the regression of metric-space valued responses. Second, our model can handle high-dimensional predictors for metric-space valued data that allow a manifold representation. Third, we investigate the convergence rate of deep neural networks in the presence of dependent sub-Gaussian noise accompanied by bias. This analysis is conducted under a well-established general composition assumption on the regression function <cit.>, which necessitates an extension of the existing theoretical framework due to the presence of dependence and bias. Although <cit.> introduced the convergence rate of deep neural networks with dependent and non-central response variables, their findings are limited to local dependence, while  
the dependence considered here does not rely on any specific form and is not restricted to local dependence. Fourth, we demonstrate how local regression can be used to map the low-dimensional representation back to the original metric space, for which we establish convergence in the presence of errors in predictors. Fifth, we examine the asymptotic properties of the proposed regression model, including its rate of convergence.  Lastly, we showcase results from various simulation settings and two real-data applications, demonstrating the superior performance of the proposed regression model compared to existing approaches.

The structure of the paper is as follows. In Section <ref>, we introduce notation and provide background on deep neural networks and local regression. The proposed regression model is introduced in Section <ref>, with the asymptotic convergence rate established in Section <ref>. Simulation results for distributional data are presented in Section <ref>. The proposed framework is illustrated in Section <ref> using networks arising from the New York yellow taxi records. Finally, we conclude with a brief discussion in Section <ref>. Auxiliary results and proofs as well as additional simulations for networks and a second data application for age-at-death distributions of human mortality are provided in the Supplementary Material.



§ PRELIMINARIES




 §.§ Notation
 
Let ℝ represent the set of real numbers and ℝ_+ denote the set of all positive real numbers. The set of all positive natural numbers is denoted as ℕ_+. Throughout this work, plain letters refer to (random) scalars and bold letters refer to (random) vectors. For a vector v, we define v^⊕0=1, v^⊕1=v, v^⊕2=vv^. We let · denote a norm and set ·=·_2 (the Euclidean norm for vectors) by default unless stated otherwise. Additionally, ·_0 denotes the number of nonzero entries of a matrix or vector, and ·_∞ represents the sup-norm for vectors, matrices, or functions. The notation a ≲ b implies the existence of a positive constant C such that a ≤ Cb. Moreover, a≍ b if both a≲ b and b≲ a. Finally, a ∧ b=min{a, b}, a ∨ b=max{a, b}.




 §.§ Deep neural networks


We provide an overview of deep neural networks (DNNs) as function approximations. The focus lies on the nonparametric regression model with p-dimensional covariates in the unit hypercube, represented by observations (X_i, U_i)∈ [0,1]^p×ℝ. Consider the following regression model


    U_i = g_0(X_i) + ϵ_i,    i=1,…,n,

where g_0 is an unknown function and ϵ_i is independent and identically distributed (i.i.d.) noise with zero mean and finite variance.


A DNN with L hidden layers and layer width 𝐩=(p_0, …, p_L, p_L+1)^ is defined recursively, forming a composite function g: ℝ^p_0↦ℝ^p_L+1,

    g(x)=W_L g_L(x)+b_L ,
       g_L(x)=σ{W_L-1 g_L-1(x)+b_L-1}, …, g_1(x)=σ(W_0 x+b_0),

where W_l∈ℝ^p_l+1× p_l is a weight matrix and b_l∈ℝ^p_l+1 is a shift vector for l=0,…,L. The rectifier linear unit (ReLU) activation function σ=max{0, x} <cit.> operates component-wise, that is, σ{(x_1,…,x_p_l)^}=(σ(x_1),…,σ(x_p_l))^. We focus on the ReLU activation function for its widespread use, empirical success, and theoretical support <cit.>. Observe that p_0=p and p_L+1=1 in the case of nonparametric regression with p-dimensional covariates. 







Given L ∈ℕ_+, 𝐩∈ℕ_+^L+2, s ∈ℕ_+ and a constant D>0, in analogy to <cit.>, we consider a class of sparse neural networks

    𝒢(L, s, 𝐩, D)={   g  of the form (<ref>): max_l=0,…,L{W_l_∞,b_l_∞}≤ 1, 
       ∑_l=1^LW_l_0+b_l_0 ≤ s,g_∞≤ D}.

We focus on DNNs with bounded weight matrices and shift vectors. This choice is motivated by empirical observations where the learned matrices and vectors tend to remain relatively small, especially when initialized with modest-sized matrices and vectors. Additionally, the consideration of sparse neural networks arises from the practical issue of overfitting deep feedforward neural networks with fully connected layers. This concern is addressed by pruning weights, reducing the total number of nonzero parameters, and sparsely connecting layers <cit.>.







To estimate the regression function g_0, the basic paradigm is to minimize the empirical risk

    ĝ = _g∈𝒢1/n∑_i=1^n{U_i - g(X_i)}^2.

Certain restrictions on the regression function g_0 are necessary to study the asymptotic properties of the estimator ĝ. Consider the Hölder class of smooth functions, 
where the ball of β-Hölder functions with radius M>0 is

    ℋ_p^β(𝔻, M)=   {g: 𝔻⊂ℝ^p ↦ℝ: 
       ∑_α:|α|<β∂^α g_∞+∑_α:|α|=⌊β⌋sup _x, y∈𝔻, x≠y|∂^α g(x)-∂^α g(y)|/x-y_∞^β-⌊β⌋≤ M},

where ⌊β⌋ denotes the largest integer strictly smaller than β, ∂^α=∂^α_1⋯∂^α_p with α=(α_1, …, α_p)^, and |α|=∑_k=1^p α_k. Let q ∈ℕ, β=(β_0, …, β_q)^∈ℝ_+^q+1 and 𝐝= (d_0, …, d_q+1)^∈ℕ_+^q+2, 𝐝̃=(d̃_0, …, d̃_q)^∈ℕ_+^q+1 with d̃_j ≤ d_j, j=0, …, q. We assume that the regression function g_0 belongs to a composite smoothness function class:

    ℋ(q, β, 𝐝, 𝐝̃, M)=    {g=g_q ∘⋯∘ g_0: g_j=(g_j 1, …, g_j d_j+1)^ and 
        g_jk∈ℋ_d̃_j^β_j([a_j, b_j]^d̃_j, M),  for some |a_j|,|b_j| ≤ M} .

Functions in this class are characterized by two kinds of dimensions, 𝐝 and 𝐝̃, where the latter represents the intrinsic dimension of the function. For example, if

    g(x)= g_31(   g_21[g_11{g_01(x_1) + g_02(x_2)} + g_12{g_03(x_3) + g_04(x_4)}] 
    
        +    g_22[g_13{g_05(x_5)+ g_06(x_6)} + g_14{g_07(x_7) + g_08(x_8)}]),  x∈[0,1]^8

and g_i j are twice continuously differentiable, then smoothness β=(2,2,2,2)^, dimensions d= (8,8,4,2,1)^ and d̃=(1,1,1,1)^. The composite smoothness class ℋ(q, β, 𝐝, 𝐝̃, M) contains a rich set of classical smoothness classes and has been widely adopted <cit.>.

Writing 

    κ_n=max _j=0, …, q n^-β̃_j /(2 β̃_j+d̃_j) with β̃_j=β_j ∏_k=j+1^q(β_k ∧ 1),
 we require the following common assumption on the structure of the DNN model <cit.>.


    
  * D≥max{M,1}, L=O(log n), s=O(n κ_n^2 log n) and n κ_n^2 ≲min_l=1, …, Lp_l ≤max_l=1, …, Lp_l ≲ n.

Assumption <ref> characterizes the flexibility of the neural network family 𝒢(L, s, p, D) as per (<ref>). While more flexible neural networks (with increased depth and width) are capable of achieving smaller approximation errors <cit.>, they often introduce larger estimation errors. Thus, Assumption <ref> provides a balance between approximation and estimation errors.













 §.§ Local regression

Let (Ω, d) be a totally bounded metric space with distance d: Ω×Ω↦ [0,∞) and 𝒯 be a closed interval in ℝ. 
Consider a random pair (Z^0, Y) with a joint distribution F supported on the product space 𝒯^r×Ω. We denote the marginal distributions of Z^0 and Y as F_Z^0 and F_Y, respectively, and the conditional distribution of Y given Z^0 as F_Y|Z^0. The mean and variance of random objects in metric spaces <cit.>, as generalizations of usual notions of mean and variance, are defined as

    y_⊕=_y∈ΩE{d^2(Y, y)},   V_⊕=E{d^2(Y, y_⊕)},

where the existence and uniqueness of the minimizer depend on the structural properties of the underlying metric space and are guaranteed for Hadamard spaces.

The conditional mean of Y given Z^0=z, corresponding to the regression function, is

    v(z)=_y∈Ω Q(y, z),   Q(·, z)=E{d^2(Y, ·)|Z^0=z}.

The lack of an algebraic structure in general metric spaces poses challenges in modeling and estimating the regression function v(z). To address this challenge, previous work <cit.> suggested leveraging the algebraic structure within the predictor space 𝒯^r. Let K be an r-dimensional kernel corresponding to a symmetric r-dimensional probability density function and write K_h(z)=h^-rK(H^-1z) with H=hI_r for a suitably chosen bandwidth h. Extending local linear regression to metric-space valued responses <cit.>, local regression (LFR) models conditional mean v(z) as a weighted mean,

    v_h(z)=_y∈ΩQ_h(y, z),   Q_h(·, z)=E{w(Z^0, z, h)d^2(Y, ·)}.

Here 

    w(Z^0, z, h)=1/μ_0-μ_1^μ_2^-1μ_1K_h(Z^0-z){1-μ_1^μ_2^-1(Z^0-z)},

where μ_j=E{K_h(Z^0-z)(Z^0-z)^⊕ j} for j=0, 1, 2. For an i.i.d. sample {(Z^0_i, Y_i)}_i=1^n, the corresponding empirical version is

    ṽ_h(z)=_y∈ΩQ̃_h(y, z),  Q̃_h(·, z)=1/n∑_i=1^nw̃(Z^0_i, z, h)d^2(Y_i, ·).

Here 

    w̃(Z^0_i, z, h)=1/μ̃_0-μ̃_1^μ̃_2^-1μ̃_1K_h(Z^0_i-z){1-μ̃_1^μ̃_2^-1(Z^0_i-z)},

where μ̃_j=n^-1∑_i=1^nK_h(Z^0_i-z)(Z^0_i-z)^⊕ j for j=0, 1, 2.



§ METHODOLOGY



Consider a totally bounded metric space (Ω, d) with distance d:Ω×Ω↦ [0,∞), and let ℳ⊂Ω be a manifold isomorphic to a subspace of the Euclidean space ℝ^r. Let (X, Y) be a random pair in ℝ^p×ℳ, where X, without loss of generality, is assumed to reside in the unit hypercube [0, 1]^p. Suppose {(X_i, Y_i)}_i=1^n are n independent realizations of (X, Y). Let r be the intrinsic dimension of the manifold ℳ. There exists a bijective representation map of the manifold ℳ:

    ψ:ℳ↦ℝ^r.
 
Algorithms such as ISOMAP <cit.>, locally linear embedding <cit.>, Laplacian eigenmaps <cit.>, or diffusion maps <cit.> are commonly employed to estimate the representation map ψ. Here we choose the ISOMAP algorithm; see <cit.> for more details. The manifold representation is useful across many scenarios, for example when one has latent parametric families of probability distributions or stochastic block models for networks, where the low-dimensional manifold closely aligns with the parameters of the underlying model <cit.>.


We propose a unified framework, the deep regression (DFR), to model the relationship between the metric-space valued response Y and a possibly high-dimensional multivariate predictor X, as summarized in Figure <ref>.






The regression function

    m=v∘g_0:[0, 1]^p↦Ω

exhibits a composite structure, where g_0=(g_01, …, g_0r)^ and v correspond to nonparametric regression using DNNs and local regression, respectively. When the intrinsic dimension is r>1, 
we fit one DNN for each coordinate. 


The proposed framework includes the ISOMAP representation ψ which however is not known and must be estimated from the data, yielding ψ̂. 
We postulate that the manifold can be well identified at the sample points through the ISOMAP algorithm. 

    
  *  Consider Z^0_i=(Z_i1^0, …, Z_ir^0)^=ψ(Y_i) and Z_i=(Z_i1, …, Z_ir)^=ψ̂(Y_i). For each j=1, …, r, there exists a function π_j:[0,1]^p× n↦ℝ such that
    
    Z_ij-Z_ij^0=π_j(X_1,…,X_n)(ϵ_ij+u_nj),   i=1,…,n,

    where the bias u_nj→ 0 as n→∞, and {ϵ_ij}_i=1^n are i.i.d. (mean zero) sub-Gaussian random variables with parameter 1 and independent of {X_i}_i=1^n, i.e., P(|ϵ_ij|≥ t)≤ 2exp{-t^2}. Furthermore, there exists a constant C_π_j such that 
    
    C_π_j = sup_{x_1,…,x_n}|π_j (x_1,…,x_n) | < ∞.


Assumption <ref> is not overly restrictive 
as it allows for dependence between the estimated low-dimensional representations Z_ij and includes potential bias u_nj that may be induced by the ISOMAP algorithm. The dependence among Z_ij across different i is quantified by π_j with an upper bound C_π_j that is specific to each coordinate. This assumption aligns with similar considerations made in <cit.> when employing DNNs to analyze time series data. The sub-Gaussian noise ϵ_ij reflects randomness associated with the manifold estimation process. 


We employ DNNs that take the observed predictors X_i as input and produce the estimated low-dimensional representations Z_i as output. For each j=1, …, r, by Assumption <ref> and the fact that Z_ij^0=g_0j(X_i) we may model the relationship between Z_ij and X_i as

    Z_ij = g_0j(X_i) + π_j(X_1,…,X_n)(ϵ_ij+u_nj).


Unlike the prevalent focus on independent Gaussian/sub-Gaussian noise without bias in the existing literature on DNNs <cit.>, the noise π_j(X_1,…,X_n)(ϵ_ij+u_nj) considered in (<ref>) exhibits dependence across different i and is accompanied by a vanishing bias u_nj, characterized by the randomness of a sub-Gaussian variable. Such generalizations necessitate a nontrivial expansion of the current theoretical framework for DNNs <cit.>; this extension is provided below as  Theorem <ref>.  
The estimation of g_0j is performed by minimizing the empirical risk 

    ĝ_j = _g∈𝒢1/n∑_i=1^n {Z_ij - g(X_i)}^2.




The proposed framework also deploys local regression using the  fitted value of DNNs Ẑ_i = ĝ(X_i)={ĝ_1(X_i),…,ĝ_r(X_i)}^ in lieu of the unobservable Z^0_i as predictors. This leads to an errors-in-variables version of  local regression,

    v̂_h(z)=_y∈ΩQ̂_h(y, z),  Q̂_h(·, z)=1/n∑_i=1^nŵ(Ẑ_i, z, h)d^2(Y_i, ·),

where 

    ŵ(Ẑ_i, z, h)=1/μ̂_0-μ̂_1^μ̂_2^-1μ̂_1K_h(Ẑ_i-z){1-μ̂_1^μ̂_2^-1(Ẑ_i-z)},

with μ̂_j=n^-1∑_i=1^nK_h(Ẑ_i-z)(Ẑ_i-z)^⊕ j for j=0, 1, 2.

Consequently, the regression function m as per (<ref>), relating the predictor X to the metric-space valued response Y, is estimated as

    m̂=v̂_h∘ĝ

with v̂_h and ĝ=(ĝ_1, …, ĝ_r)^ defined in (<ref>) and (<ref>). This comprehensive framework empowers us to effectively handle potentially high-dimensional predictors while simultaneously addressing the inherent complexities associated with metric-space valued responses. Further details are provided in Algorithm <ref>.



§ ASYMPTOTIC PROPERTIES
















We first derive the convergence rate of DNN estimates (<ref>), extending the existing theoretical framework for DNNs to accommodate dependent noise and bias.  
We then delve into the convergence rate of the local regression estimate (<ref>), considering multivariate predictors contaminated by errors induced by DNNs. Finally, the convergence rate of the deep regression estimate as per (<ref>) is established by combining the convergence rates of DNNs and local regression. For DNN, we require <ref> and the following common assumption <cit.>.

    
    
  * The target function g_0j belongs to the Hölder class ℋ (q,β,𝐝,𝐝̃,M) defined in (<ref>) for j = 1,…, r.
    

Assumption <ref> states that the spaces of g_0j is the Hölder class ℋ (q,β,𝐝,𝐝̃,M) which encompasses a rich class of smooth functions <cit.>; see Section <ref> for more details. The following result formalizes the convergence rate of DNN estimates (<ref>).

 





    Consider the nonparametric regression model (<ref>).
    
    Under <ref>, <ref> and <ref>, for any i=1,…,n and j = 1,…,r, there exists an estimator ĝ_j in (<ref>) such that
    
    E[{ĝ_j(X_i) - g_0j(X_i)}^2] = O(κ_n^2 log^3 n+|u_nj|)

    and
    
    E[{ĝ_j(X) - g_0j(X)}^2] = O(κ_n^2 log^3 n+|u_nj|)

    where  X is a new realization of the predictor that is independent of the sample {(X_i, Y_i)}_i=1^n and κ_n is defined in (<ref>). It immediately follows that
    
    E(Ẑ_i - Z_i^0^2)= O(κ_n^2 log^3 n+u^2_n)

    and 
    
    E(Ẑ - Z^0^2)= O(κ_n^2 log^3 n+u^2_n),

    where Ẑ=(ĝ_1(X),…,ĝ_r(X))^, Z^0 = (g_01(X),…,g_0r(X))^, u_n^2=max_j=1, …, r|u_nj| and g_0j are the components of g_0 as defined in (<ref>).

All proofs are provided in the Supplementary Material. In contrast to the prevailing emphasis on DNNs with independent Gaussian/sub-Gaussian noise, Theorem <ref> establishes the convergence rate of DNNs in the presence of dependent sub-Gaussian noise coupled with vanishing bias. This extension may also be useful for other work in the area of  DNN. Note that the convergence rates in Theorem <ref> depend on the bias term u_nj, smoothness β, and intrinsic dimension 𝐝̃ of functions g_0j as per (<ref>), rather than the predictor dimension p. As a result, the proposed method can circumvent the curse of dimensionality, exhibiting a faster convergence rate particularly when the intrinsic dimension 𝐝̃ remains relatively low. For instance, disregarding the bias term momentarily, consider the true function g_0j structured in (<ref>). For such a target function traditional smoothing methods are fraught with a slow convergence rate of order n^-2/p+4 = n^-1/6, whereas the proposed DNN approach has a convergence rate of order n^-2/5log^3/2 n, reaching the optimal one-dimensional nonparametric regression convergence rate up to a polylogarithmic factor. This accelerated convergence rate effectively mitigates the curse of dimensionality inherent in nonparametric regression.








































Next, we show the convergence rate for errors-in-variables multivariate local regression, for which we require the following assumptions.

    
  *  The minimizers v(z), v_h(z), ṽ_h(z) and v̂_h(z) for local regression as per (<ref>), (<ref>), (<ref>), and (<ref>) exist and are unique, the last two almost surely. Additionally, for any ε>0,
    
        
  * inf_d{v(z), y}>ε[Q(y, z)-Q{v(z), z}]>0,
        
  * lim inf_h→0inf_d{v_h(z), y}>ε[Q_h(y, z)-Q_h{v_h(z), z}]>0,
        
  * P(inf_d{ṽ_h(z), y}>ε[Q̃_h(y, z)-Q̃_h{ṽ_h(z), z}]>0)→1,
        
  * P(inf_d{v̂_h(z), y}>ε[Q̂_h(y, z)-Q̂_h{v̂_h(z), z}]>0)→1.
    
    
    
  *  Let B_δ(y)⊂Ω be the ball of radius δ centered at y and N{ε, B_δ(y), d} be its covering number using balls of size ε (see Definition <ref> in Supplementary Material <ref>). Then for any y∈Ω, 
    
    ∫_0^1[1+log N{δε, B_δ(y), d}]^1/2dε=O(1)  as δ→0.

    
    
  * There exists η_1, η_2, η_3>0, C_1, C_2, C_3>0 and γ_1, γ_2, γ_3>1 such that
    
    
  * inf_d{v(z), y}<η_1[Q(y, z)-Q{v(z), z}-C_1d{v(z), y}^γ_1]≥0,
    
  * lim inf_h→0inf_d{v_h(z), y}<η_2[Q_h(y, z)-Q_h{v_h(z), z}-C_2d{v_h(z), y}^γ_2]≥0,
    
  *  P(inf_d{ṽ_h(z), y}<η_3[Q̃_h(y, z)-Q̃_h{ṽ_h(z), z}-C_3d{ṽ_h(z), y}^γ_3]≥0)→1,
    
    
    
    
    
    



Assumption <ref> is a standard requirement to ensure the consistency of M-estimators, as outlined by <cit.>. We note that the existence and uniqueness of the minimizers in <ref> is guaranteed for the case of Hadamard spaces, the curvature of which is bounded above by 0 <cit.>. Assumption <ref> entails conditions on the metric entropy of Ω to control its size. 
In Proposition <ref>, it constrains the bracketing integral used to derive the tail bound, following Theorem 2.14.16 in <cit.>. 
The curvature assumption in <ref> regulates the behavior of (Q_h - Q), (Q̃_h - Q_h), and (Q̂_h - Q̃_h) near their minima, which makes it possible to derive convergence rates. 

These assumptions stem from empirical process theory and are frequently employed in the literature concerning statistical analysis for metric-space valued data <cit.>. 

We now provide some statistically relevant examples of metric spaces, for which we demonstrate in Section <ref> of the Supplementary Material that these metric spaces satisfy  Assumptions <ref>–<ref>; see Propositions <ref>, <ref> and <ref>.
















	Let Ω be the space of probability distributions on a closed interval of ℝ with finite second moments, equipped with the Wasserstein metric d_𝒲 where
	
    d_𝒲^2(μ, ν)=∫_0^1{F_μ^-1(p)-F_ν^-1(p)}^2dp

	for any two probability distributions μ, ν∈Ω. The Wasserstein space (Ω, d_𝒲) satisfies Assumptions <ref>–<ref> with γ_1=γ_2=γ_3=2.



	Let Ω be the space of graph Laplacians of undirected weighted networks with a fixed number of nodes m and bounded edge weights equipped with the Frobenius metric d_F. The space (Ω, d_F) satisfies Assumptions <ref>–<ref> with γ_1=γ_2=γ_3=2.



	Let Ω be the space of m-dimensional covariance matrices with bounded variances or correlation matrices equipped with the Frobenius metric d_F. The space (Ω, d_F) satisfies Assumptions <ref>–<ref> with γ_1=γ_2=γ_3=2.














We next derive the convergence rate of multivariate local regression, extending previous results for local regression (<ref>) with one-dimensional predictors. The proof relies on M-estimation techniques arising from empirical process theory. The kernel and distributional assumptions <ref> and <ref> listed in the Appendix are standard for local regression. In the following, r is as in (<ref>) the intrinsic dimension of the manifold representation (<ref>).


If <ref>, <ref>, <ref> (i)–(ii) and <ref> (i)–(ii) hold, then

    d{v_h(z), v(z)}=O(h^2/(γ_1-1))

as h→0. If furthermore nh^r→∞, then

    d{ṽ_h(z), v_h(z)}=O_p{(nh^r)^-1/{2(γ_2-1)}}.


In general, the convergence rates are determined by the local geometry near the minimum as quantified in <ref>. For metric spaces in Examples <ref>, <ref> and <ref>, γ_1=γ_2=2 and convergence rates of the bias and stochastic deviation are d{v_h(z), v(z)}=O(h^2) and d{ṽ_h(z), v_h(z)}=O_p{(nh^r)^-1/2}, respectively. With h=n^-1/(4+r), the multivariate local regression estimate achieves the rate d{ṽ_h(z), v(z)}=O_p(n^-2/(4+r)), corresponding to the well-known optimal rate for standard local linear regression.

The low-dimensional representations Z^0_i in (<ref>) are unobservable, necessitating the use of (<ref>) in practice, where the estimated representations Ẑ_i obtained through DNNs are substituted. Consequently, errors are introduced in the predictors of local regression, leading to the need to address the errors-in-variables problem. The following theorem characterizes the impact of these errors on the estimation.



Suppose <ref>, <ref>, <ref> hold. If Ẑ_i-Z^0_i=O_p(ζ_n) for i=1, …, n, Ẑ-Z^0=O_p(ζ_n), nh^r→∞, and h^-r-2ζ_n→0, then it holds for ṽ_h(·) and v̂_h(·) as per (<ref>) and (<ref>) that

    d{v̂_h(Ẑ), ṽ_h(Z^0)}=O_p{(h^-r-1ζ_n)^1/(γ_3-1)}.



For metric spaces in Examples <ref>, <ref> and <ref>, γ_3=2 and the convergence rate provided in Proposition <ref> reduces to O_p{(h^-r-1ζ_n). Combining Theorems <ref> and Proposition <ref> and <ref>, we have the following Theorem.


If <ref>, <ref>–<ref>, <ref>, <ref>, <ref>–<ref> hold and furthermore nh^r→∞, h^-r-2ζ_n→0, one has

    d{m̂(X), m(X)}=O_p{h^2/(γ_1-1)+(nh^r)^-1/{2(γ_2-1)}+(h^-r-1ζ_n)^1/(γ_3-1)},

where X is a new predictor independent of the sample {(X_i, Y_i)}_i=1^n and ζ_n=κ_nlog^3/2n+u_n.



The initial two terms h^2/(γ_1-1), (nh^r)^-1/{2(γ_2-1)} in Theorem <ref> give  the convergence rate of multivariate local regression, as outlined in Proposition <ref>, representing the optimal rate when Z_i^0, ψ, and g_0 are known. The final term, (h^-r-1ζ_n)^1/(γ_3-1), originates from Theorem <ref> and Proposition <ref>, reflecting errors introduced during the estimation of low-dimensional representations of metric-space valued responses (ISOMAP), DNNs, and the subsequent errors-in-variables problem.



If one assumes E{sup_i=1,…,n(Ẑ_i - Z_i^0)^2}=O(τ_n^2), the dependence of the convergence rate presented in Proposition <ref> on the intrinsic dimension of metric-space valued responses r would vanish. Specifically, the convergence rate in Proposition <ref> would be further improved to (h^-1τ_n)^1/(γ_3-1), leading to an enhanced convergence rate of the proposed model as h^2/(γ_1-1)+(nh^r)^-1/2(γ_2-1)+(h^-1τ_n)^1/(γ_3-1). The proof is provided in the Supplementary Material <ref>. While the primary focus of the current work is to devise a general framework for handling multivariate predictors and metric-space valued responses, it would be intriguing for future work to derive such a supremum rate for DNNs when dealing with dependent sub-Gaussian noise coupled with bias.  



§ IMPLEMENTATION AND SIMULATIONS




 §.§ Implementation details

The optimization problem in (<ref>) was addressed within Pytorch framework, where we opted for the Adam optimizer <cit.>, a widely used choice in deep learning known for its robust performance across diverse models and datasets. The weight matrices W_l and shift vectors b_l of the function g∈𝒢 as per (<ref>) were initialized using the default random initialization provided by Pytorch.

Manifold learning (ISOMAP) and DNNs involve tuning parameters such as the number of nearest neighbors k in the Dijkstra algorithm <cit.>,  the number of hidden layers L, the number of neurons p_l in each hidden layer, dropout rate, and learning rate. For simplicity, we maintained the same number of neurons in each hidden layer. The dropout rate <cit.> is the rate of randomly excluding neurons during training. The learning rate <cit.> determines the step size for gradient descent in the Adam algorithm. Tuning these parameters can be accomplished through a grid search, assessing empirical risk over a held-out validation set after each training iteration. We set aside 20% of the training set as the validation set in each run. The training process stops when the empirical risk on the validation set ceases to reliably improve.




















 §.§ Simulations for probability distributions

To assess the performance of the proposed model, we report the results of simulations for various settings. The random objects we consider first are univariate probability distributions equipped with the Wasserstein metric; see Example <ref>. The Wasserstein space is a geodesic metric space related to optimal transport <cit.> and is increasingly applied in various research domains, including population pyramids <cit.>, financial returns <cit.>, and multi-cohort studies <cit.>, among others <cit.>. Additional simulations for networks can be found in Supplementary Material <ref>. Despite the simulated networks not residing on a 2-dimensional manifold, the proposed model exhibits remarkable robustness as the sample size increases.

Consider the true regression function, represented as a quantile function, to be

    m(X) = E(η|X)+E(σ|X) Φ^-1(·),

which corresponds to a Gaussian distribution with mean and standard deviation depending on X. The distribution parameters of m(X) are generated conditionally on X, where the mean η and the standard deviation σ are assumed to follow a normal distribution and a Gamma distribution, respectively.

We consider sample sizes n=100,200,500,1000, with Q=500 Monte Carlo runs. In each Monte Carlo run, predictors X_i ∈ℝ^9, i=1, …, n, are independently sampled as

    X_i1∼ U(-1,0), X_i2∼ U(0,1), X_i3∼ U(1,2)
       
            X_i4∼ N(0,1), X_i5∼ N(-10,3), X_i6∼ N(10,3)
       
            X_i7∼Bernoulli(0.6), X_i8∼Bernoulli(0.7), X_i9∼Bernoulli(0.3).






For each i, the mean and standard deviation of the Gaussian distribution are generated conditionally on X_i as

    η_i|X_i∼ N(μ, 0.5^2), σ_i|X_i ∼Gamma(θ^2, θ^-1) where 
       μ=3 X_i8{sin (π X_i1)+cos(π X_i2)}  + X_i7(5X_i4^2+X_i5) ,
       θ=[3+0.5 X_i8{sin (π X_i1) +cos(π X_i2)} + X_i7|5X_i4^2 + X_i5|].






The corresponding distributional response is then constructed as Y_i = η_i + σ_i Φ^-1. 

To simulate practical scenarios where direct observations of probability distributions are not available, but rather independent data samples generated by the corresponding distribution are obtained, we independently sample 100 observations {y_ij}_j=1^100 from each distributional response Y_i. Consequently, one must initially estimate the distributional response Y_i from the random sample {y_ij}_j=1^100, introducing a bias to the regression model. This practical consideration aligns with prior research <cit.>, which addresses the issue by adopting the empirical measure in place of the unobservable distributional response Y_i. For the implementation of local regression using empirical measures, we follow the algorithm outlined in <cit.>. The bandwidths for the local regression in each Monte Carlo run are chosen as 10% of the range of the intermediate estimates Ẑ_i = ĝ(X_i).

For the qth Monte Carlo run out of Q=500 runs, with m̂_q(·) denoting the fitted regression function, the quality of the estimation is quantified by the mean squared prediction error (MSPE),

    MSPE_q = 1/100∑_i=1^100d_𝒲^2{m̂_q(X_i^test), m(X_i^test)},

where {X_i^test}_i=1^100 denote out-of-sample predictors and the metric d_𝒲 is the Wasserstein metric for probability distributions. The average quality of the estimation over the Q=500 Monte Carlo runs is quantified  by the average mean squared prediction error (AMSPE)

    AMSPE=1/Q∑_q=1^QMSPE_q.




























To assess the validity of the manifold assumption, Figure <ref>(a) presents a scatter plot depicting the two-dimensional representations of distributional responses {Y_i}_i=1^n alongside their true parameters (mean and standard deviation) for a sample size of n=500. The plot reveals a distinctive horseshoe shape in the two-dimensional representations, implying that the distributional responses adhere to a lower-dimensional manifold. Intriguingly, the two-dimensional representations closely align with the two-dimensional parameters of the distributional responses. This observation suggests that for distributional responses ISOMAP effectively identifies suitable low-dimensional representations corresponding to the latent parameter space. 


We conduct comparisons with global regression (GFR) <cit.> and local regression with sufficient dimension reduction (SDR) <cit.>. Figure <ref>(b) summarizes MSPEs for all Monte Carlo runs and various sample sizes using the proposed model, GFR and SDR. The MSPE decreases with increasing sample size, indicating the convergence of the proposed model to the target. Notably, the proposed model exhibits superior performance over GFR and SDR across all sample sizes, including the case of n=100. In Table <ref>, we present the AMSPE for different sample sizes. The proposed model (DFR) is seen to fare best across different methods. Despite the conventional need for large sample sizes in the performance of deep neural networks, the proposed model surprisingly demonstrates robustness for small sample sizes, potentially owing to its flexibility in accommodating complex regression relationships.




















































































§ DATA APPLICATION





    
    
    
    
    
    
    

















































Yellow and green taxi trip records in New York City (NYC), including pick-up and drop-off dates, locations, trip distances, payment methods, and passenger counts, are available at <https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page>. Additionally, we collect NYC weather history, including daily average temperature, humidity, wind speed, pressure, and total precipitation from <https://www.wunderground.com/history/daily/us/ny/new-york-city/KLGA/date>. The objective is to predict transport networks constructed from taxi trip records using relevant predictors. Given the potential influence of weather conditions on travel plans, the variability in travel patterns across different days of the week, and the impact of daily trip features on the taxi system, we construct a fifteen-dimensional predictor set including daily weather information, indicators for days of the week or holiday, and daily trip features averaged over each day; see Table <ref>.












We analyze yellow taxi trip records in Manhattan, excluding islands, and divide the 66 taxi zones into 13 regions based on preprocessing procedures outlined in <cit.>. While <cit.> focused on the effect of COVID-19 on the transport network, our study investigates the pre-COVID transport network. Therefore, we limit our analysis to a period of 1092 days from January 1, 2017, to December 31, 2019, excluding 3 outlier days.





The main interest regarding these traffic records is the transport network that represents daily passenger movement between the 13 regions. To this end, we build daily undirected networks with nodes standing for the 13 regions and edge weights representing the number of passengers traveling between the regions. Each of these networks is uniquely associated with a 13 × 13 graph Laplacian. The two-dimensional representations of 1092 graph Laplacians using MDS are shown in Figure <ref> which suggest a clear separation among Monday to Thursday, Friday or Saturday, and Sunday or holiday in the second dimension. Additionally, as the first dimension increases, the total ridership also increases. The scatter plot exhibits a horseshoe shape, indicating that the manifold assumption is plausible within this context.



The proposed model was applied to model the relationship between daily graph Laplacians and the fifteen-dimensional predictors. The MSPE was calculated using 10-fold cross-validation, averaged over 100 runs. The proposed method achieves better prediction performance, resulting in a 46% and 55% improvement in prediction accuracy compared to GFR <cit.> and SDR <cit.>, respectively.

To further investigate the effect of daily total precipitation on taxi traffic, we predicted networks with varying total precipitation levels and fixed all other covariates at their median levels. Figure <ref> shows the predicted networks at total precipitation levels of 0, 2, and 6 inches on Monday to Thursday, Friday or Saturday, and Sunday or holiday. We observed that edge weights decrease on Sunday or holiday compared with those on Monday to Saturday, regardless of total precipitation levels. This suggests that more people tend to stay at home on Sunday and holiday. Across different precipitation levels, the taxi traffic from Monday to Thursday is more concentrated in areas 105, 106, 107, and 108, which are primarily residential areas with popular destinations such as Penn Station and Grand Central Terminal. Conversely, taxi traffic on Friday and Saturday is more diffuse, spreading into areas 102, 103, and 104, which are home to popular bars, restaurants, chain stores, and high-end art galleries and museums. As the total precipitation increases, taxi traffic decreases, irrespective of the day of the week. When the total precipitation reaches 6 inches, corresponding to a rainstorm, the remaining taxi traffic for all three categories of days is primarily concentrated in areas 105, 106, 107, and 108. This may be related to unavoidable activities such as work-related events. However, discretionary travel plans, primarily observed in areas 102, 103, and 104, are more likely to be canceled due to adverse weather conditions.






































































































































































































§ DISCUSSION









The proposed deep regression model introduces a novel approach for analyzing metric-space responses using multivariate predictors. Extensive simulations and applications across diverse scenarios, including networks and probability distributions, highlight the versatility of the model. 
The proposed approach is flexible and requires no specific model assumptions. 
Key contributions include an in-depth exploration of the convergence rate of deep neural networks under dependent sub-Gaussian noise with bias and an extension of the scope of local regression to handle multivariate predictors, including the convergence rate in an errors-in-variables regression framework. The proposed approach can be easily adapted to various manifold learning methods, such as locally linear embedding, Laplacian eigenmaps and diffusion maps.

A limitation lies in the assumption of the existence of a low-dimensional representation of random objects. However, our simulations and applications demonstrate the broad applicability of the proposed method, showcasing robust performance even when the intrinsic dimension of metric-space valued responses is not low.
Open problems for future research include the supremum rate of DNN convergence when dealing with dependent sub-Gaussian noise with bias and the case of high-dimensional predictors.   















§ ACKNOWLEDGMENTS


This research was supported in part by NSF grant DMS-2310450.



§ APPENDIX A. KERNEL AND DISTRIBUTIONAL ASSUMPTIONS


    
  *  The r-dimensional kernel K(·) satisfies ∫_ℝ^rK(u)u_j^4du<∞ and ∫_ℝ^rK^2(u)u_j^6du<∞ for j=1, …, r and u=(u_1, …, u_r)^, and is Lipschitz continuous with compact support [-1, 1]^r. 


    
  *  Both the marginal density f_Z^0(·) of Z^0 and the conditional densities f_Z^0| Y(·, y) of Z^0 given Y=y exist and are twice continuously differentiable, the latter for all y∈Ω, and sup_z, y|(∂^2 f_Z^0| Y/∂z^2)(z, y)|<∞. In addition, for any open set U⊂Ω, ∫_UdF_Y| Z^0(z, y) is continuous as a function of z.



chicago







Supplementary Material
















 §.§ Additional Background

[Covering number]
    Let (Ω, d) be a metric space. Let S ⊆Ω be a subset and ε>0. The ε-covering number of S, denoted by N(ε, S, d), is the minimal number of balls of radius ε needed to cover the set S, i.e.,

    N(ε, S, d):=min{k: There exist y_1, …, y_k ∈Ω such that S ⊆⋃_i=1^k B_ε(y_i)},

where B_ε(y)={y' ∈Ω: d(y, y') ≤ε} denotes the closed ε-ball centered at y ∈Ω.
































 §.§ Verification of Model Assumptions






	
    The Wasserstein space (Ω, d_𝒲) defined in Example <ref> satisfies Assumptions <ref>–<ref>.


    As discussed in Example <ref> of <cit.>, the Wasserstein space (Ω, d_𝒲) satisfies the first three parts of <ref>, <ref>, and the first two parts of <ref> with γ_1=γ_2=2. It suffices to show <ref> (iv) and <ref> (iii). For any probability distribution y∈Ω, let F_y^-1 be the corresponding quantile function. Let ⟨·, ·⟩, ·_L^2, and d_L^2(·, ·) be the inner product, norm, and distance on the Hilbert space L^2(0, 1). For any y∈Ω, the map from y to F_y^-1 is an isometry from Ω to the subset of L^2(0, 1) formed by equivalence classes of left-continuous nondecreasing functions on (0, 1). The Wasserstein space Ω can thus be viewed as a subset of L^2(0, 1), which has been shown to be convex and closed <cit.>.
    
    Let
    
    B̂(z)=1/n∑_i=1^nŵ(Ẑ_i, z, h)F_Y_i^-1.

    Since 
    
    Q̂_h(y, z)   =1/n∑_i=1^nŵ(Ẑ_i, z, h)d_𝒲^2(Y_i, y)
       =1/n∑_i=1^nŵ(Ẑ_i, z, h)[d_L^2^2{F_Y_i^-1, B̂(z)}+d_L^2^2{B̂(z), F_y^-1}+
       2⟨ F_Y_i^-1-B̂(z), B̂(z)-F_y^-1⟩_L^2]
       =Q̂_h(B̂(z), z)+d_L^2^2{B̂(z), F_y^-1}+
       2/n∑_i=1^nŵ(Ẑ_i, z, h)⟨ F_Y_i^-1-B̂(z), B̂(z)-F_y^-1⟩_L^2

    and
    
    1/n∑_i=1^nŵ(Ẑ_i, z, h)⟨ F_Y_i^-1-B̂(z), B̂(z)-F_y^-1⟩_L^2
       =⟨1/n∑_i=1^nŵ(Ẑ_i, z, h)F_Y_i^-1-B̂(z), B̂(z)-F_y^-1⟩_L^2
       =⟨B̂(z)-B̂(z), B̂(z)-F_y^-1⟩_L^2
       =0,

    One has
    
    Q̂_h(y, z)=Q̂_h(B̂(z), z)+d_L^2^2{B̂(z), F_y^-1},

    whence
    
    v̂_h(z)=_y∈ΩQ̂_h(y, z)=_y∈Ωd_L^2^2{B̂(z), F_y^-1}.

    Then by the convexity and closedness of the Wasserstein space, the minimizer v̂_h(z) exists and is unique for any z∈ℝ^r. Hence <ref> (iv) is satisfied.
    
    In view of 
    
    1/n∑_i=1^nw̃(Z_i^0, z, h)=1,

    one can similarly show that 
    
    ṽ_h(z)=_y∈ΩQ̃_h(y, z)=_y∈Ωd_L^2^2{B̃(z), F_y^-1},

    where
    
    B̃(z)=1/n∑_i=1^nw̃(Z_i^0, z, h)F_Y_i^-1.

    Observe that ṽ_h(z), viewed as the best approximation of B̃(z) in Ω, is characterized by 
    
    ⟨B̃(z)-F_ṽ_h(z)^-1, F_y^-1-F_ṽ_h(z)^-1⟩_L^2≤0,  for all y∈Ω.

    It follows that
    
    Q̃_h(y, z)   =1/n∑_i=1^nw̃(Z_i^0, z, h)d_𝒲^2(Y_i, y)
       =Q̃_h{ṽ_h(z), z}+d_𝒲^2{ṽ_h(z), y}+
       2/n∑_i=1^nw̃(Z_i^0, z, h)⟨ F_Y_i^-1-F_ṽ_h(z)^-1, F_ṽ_h(z)^-1-F_y^-1⟩_L^2
       =Q̃_h{ṽ_h(z), z}+d_𝒲^2{ṽ_h(z), y}+2⟨B̃(z)-F_ṽ_h(z)^-1, F_ṽ_h(z)^-1-F_y^-1⟩_L^2
       ≥Q̃_h{ṽ_h(z), z}+d_𝒲^2{ṽ_h(z), y}

    for all y∈Ω. Therefore, <ref> (iii) holds for the Wasserstein space (Ω, d_𝒲) for any η_3>0, C_3=1 and γ_3=2.

    
    
    
    
    




    The metric space (Ω, d_F) defined in Example <ref> satisfies Assumptions <ref>–<ref>, where Ω is the space of graph Laplacians of undirected weighted networks with a fixed number of nodes m and bounded edge weights.


    
    As discussed in Theorem 3 of <cit.>, the space of graph Laplacians (Ω, d_F) satisfies the first three parts of <ref>, <ref>, and the first two parts of <ref> with γ_1=γ_2=2. It suffices to show <ref> (iv) and <ref> (iii). Let ⟨·, ·⟩_F and ·_F be the Frobenius inner product and norm on Ω. Define
    
    B̂(z)=1/n∑_i=1^nŵ(Ẑ_i, z, h)Y_i,  B̃(z)=1/n∑_i=1^nw̃(Z_i^0, z, h)Y_i.

    Similar to the proof of Proposition <ref>, one can show that
    
    v̂_h(z)=_y∈Ωd_F^2{B̂(z), y},  ṽ_h(z)=_y∈Ωd_F^2{B̃(z), y}.

    Then by the convexity and closedness of the space of graph Laplacians, the minimizer v̂_h(z) exists and is unique for any z∈ℝ^r. Hence <ref> (iv) is satisfied.
    
    Observe that ṽ_h(z), viewed as the best approximation of B̃(z) in Ω, is characterized by
    
    ⟨B̃(z)-ṽ_h(z), y-ṽ_h(z)⟩_F≤0,  for all y∈Ω.

    It follows that
    
    Q̃_h(y, z)   =1/n∑_i=1^nw̃(Z_i^0, z, h)d_F^2(Y_i, y)
       =Q̃_h{ṽ_h(z), z}+d_F^2{ṽ_h(z), y}+
       2/n∑_i=1^nw̃(Z_i^0, z, h)⟨ Y_i-ṽ_h(z), ṽ_h(z)-y⟩_F
       =Q̃_h{ṽ_h(z), z}+d_F^2{ṽ_h(z), y}+2⟨B̃(z)-ṽ_h(z), ṽ_h(z)-y⟩_F
       ≥Q̃_h{ṽ_h(z), z}+d_F^2{ṽ_h(z), y}

    for all y∈Ω. Therefore, <ref> (iii) holds for the space of graph Laplacians (Ω, d_F) for any η_3>0, C_3=1 and γ_3=2.




    The metric space (Ω, d_F) defined in Example <ref> satisfies Assumptions <ref>–<ref>, where Ω is the space of m-dimensional covariance matrices with bounded variances or correlation matrices.

The proof of Proposition <ref> is similar to that of Proposition <ref> and is thus omitted.




























 §.§ Proof of Theorem <ref>







To establish the convergence of estimation error, <cit.> applied an oracle-type inequality in his Theorem 2, supported by Lemma 4. In contrast to <cit.> where the prediction error was studied assuming independent Gaussian noise with mean 0, here we consider the case of dependent sub-Gaussian noise with possible bias that vanishes asymptotically as in (<ref>). Additionally, we investigate both out-of-sample and in-sample errors, given that the output of deep neural networks serves as the input for local regression. Understanding the in-sample error is crucial for examining the convergence rate of local regression. Revisiting Lemma 4 of <cit.>, we provide here a parallel result, Lemma <ref>, that is suitable for the type of dependent sub-Gaussian noise with bias that we assume for our data.  Theorem <ref> then follows along similar arguments as in Theorem 1 and Corollary 1 of <cit.>. 




    Let ξ_1, …, ξ_M be sub-Gaussian random variables with mean 0 and a common parameter C, then 
    E(max_l=1,…,M ξ_l^2)≲log M.




By the sub-Gaussian assumption, we have P(|ξ_l|≥√(t))≤ 2 e^-t/C^2. For any u>0, we can apply the union bound and observe the following:

    E(max_l=1,…,Mξ_l^2)    ≤ u + ∫_u^∞ P(max_l=1,…,Mξ_l^2 ≥ t) dt
       ≤ u + ∑_l=1^M∫_u^∞ P(|ξ_l| ≥√(t)) dt
       ≤ u + 2M∫_u^∞ e^-t/C^2dt
        = u + 2MC^2e^-u/C^2.

By selecting u≍log M, we conclude that 

    E(max_l=1,…,Mξ_l^2)≲log M.




Consider the p-variate nonparametric regression model (<ref>) with unknown regression function g_0j. Let g̃_j be any estimator taking values in 𝒢=𝒢(L, s, 𝐩, D) as per (<ref>) and N_δ=N(δ,𝒢,·_∞) be the covering number of 𝒢 using balls of size δ. Define

    Δ_n= E[1/n∑_i=1^n {Z_ij - g̃_j(X_i)}^2 - inf_g∈𝒢1/n∑_i=1^n{Z_ij-g(X_i)}^2]

and assume g_0j_∞≤ D. Then for any i=1, …, n and j=1, …, r, we have

    E[{g̃_j(X_i) - g_0j(X_i)}^2]≲inf_g∈𝒢E[{g(X)-g_0j(X)}^2] + log N_δ/n + Δ_n + δ + |u_nj|

and

    E[{g̃_j(X) - g_0j(X)}^2]≲inf_g∈𝒢E[{g(X)-g_0j(X)}^2] + log N_δ/n + Δ_n + δ + |u_nj|,

with X being an independent copy of X_1.








Define g^2_n = 1/n∑_k=1^n g(X_k)^2. Since E[{g̃_j(X_i) - g_0j(X_i)}^2]≤ 4 D^2 and E[{g̃_j(X) - g_0j(X)}^2]≤ 4 D^2, the inequality in (<ref>) and (<ref>) hold trivially if log N_δ≥ n. In the following, we focus on the case log N_δ < n.

Similar to Lemma 4 of <cit.>, the proof is divided into three parts which are denoted by (I)–(III).

    
  * Relate the in-sample error E[{g̃_j(X_i) - g_0j(X_i)}^2] and predicted error E[{g̃_j(X) - g_0j(X)}^2] to E(g̃_j-g_0j_n^2) via the inequality

    E[{g̃_j(X_i) - g_0j(X_i)}^2] ≲ E(g̃_j-g_0j_n^2) + log N_δ/n + δ .


    E[{g̃_j(X) - g_0j(X)}^2] ≲ E(g̃_j-g_0j_n^2) + log N_δ/n + δ .


    
  * For any estimator g̃_j taking values in 𝒢,

    | E{1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) g̃_j(X_k)}| ≲√(E(g̃_j-g_0j_n^2)  log N_δ/n) + δ


    
  * Show that

    E(g̃_j-g_0j_n^2) ≲inf _g ∈𝒢 E[{g(X)-g_0j(X)}^2]+log N_δ/n+Δ_n + δ + |u_nj|.



The inequality in (<ref>) and (<ref>) follow from (I) and (III).

(I): Since the inequality (<ref>) has been shown by Lemma 4(I) in <cit.>, we only need to show the inequality (<ref>). Given a minimal δ-covering of 𝒢, denote the centers of the balls by g_l. By construction, there exists a l^* such that g̃-g_l^*_∞≤δ. Without loss of generality, one can assume that g_l_∞≤ D by the definition of 𝒢. It follows that 

    |E[{g̃_j(X_i)-g_0j(X_i)}^2] - E{g̃_j-g_0j_n^2}|
       = |E[1/n∑_k=1^n{g̃_j(X_i)-g_0j(X_i)}^2 - {g̃_j(X_k)-g_0j(X_k)}^2]|
        = |1/n∑_k=1^n E[{g̃_j(X_i)-g_l^*(X_i)}^2 + {g_l^*(X_i) - g_0j(X_i)}^2 
        + 2{g̃_j(X_i)-g_l^*(X_i)}{g_l^*(X_i) - g_0j(X_i)} - {g̃_j(X_k)-g_l^*(X_k)}^2 
        - {g_l^*(X_k) - g_0j(X_k)}^2 - 2{g̃_j(X_k)-g_l^*(X_k)}{g_l^*(X_k) - g_0j(X_k)}]|
       ≲ E{|1/n∑_k=1^n f_l^*(X_k, X_i)| } + δ

where f_l^*(X_k, X_i)={g_l^*(X_i)-g_0j(X_i)}^2-{g_l^*(X_k)-g_0j(X_k)}^2. Furthermore, define R_l^*=D√(n^-1log N_δ)∨√(E[{g_l^*(X_i)-g_0j(X_i)}^2]). Using g_l^*-g̃_j_∞≤δ and the triangle inequality, we have R_l^*≤ D√(n^-1log N_δ)+√(E[{g̃_j(X_i)-g_0j(X_i)}^2]) +δ. Similarly, we can define f_l and R_l in the same way with l^* being replaced by l.

Set U=√(E[{g̃_j(X_i)-g_0j(X_i)}^2]) and T=max_l=1,…,N_δ|∑_i=1^n f_l(X_k, X_i) /(R_l D)|, we have

    |E[{g̃_j(X_i)-g_0j(X_i)}^2] - E[g̃_j-g_0j_n^2]| 
       ≲ E{|1/n∑_k=1^n f_l^*(X_k, X_i)/(R_l^* D)| R_l^*} + δ
       ≲1/nE{T (U + √(n^-1log N_δ) + δ) } + δ
       ≲1/n√(E(T^2))√(E(U^2)) + 1/n(√(n^-1log N_δ) + δ)E(T) + δ,

where the last inequality follows from the Cauchy-Schwarz inequality. Observe that E[f_l(X_k, X_i)]=0,|f_l(X_k, X_i)| ≤ 4 D^2 and

    {f_l(X_k, X_i)}    =2  [{g_l(X_i)-g_0j(X_i)}^2] ·1_k  i
       ≤ 2 E[{g_l(X_i)-g_0j(X_i)}^4] 
       ≤ 8 D^2 R_l^2 .

Up to now, all the settings match Lemma 4(I) of <cit.>. Therefore, using the same technique, we can show that

    E[{g̃_j(X_i) - g_0j(X_i)}^2] ≲ E(g̃_j-g_0j_n^2) + log N_δ/n + δ .










































(II): For any estimator g̃_j taking values in 𝒢, using Jensen's inequality and Assumption <ref>, we have

    |E{∑_k=1^n ϵ_kjπ_j(X_1,…,X_n){g̃_j(X_k)-g_l^*(X_k)}}| 
       ≤ C_π_jδ∑_k=1^n E(|ϵ_kj|) 
       ≤ n δ C_π_j√(E(ϵ_1j^2))
       ≤ n δ C_π_j.


Since 

    E{ϵ_kjπ_j(X_1,…,X_n) g_0j(X_k)}
       =E[E{ϵ_kjπ_j(X_1,…,X_n) g_0j(X_k) | X_1, …, X_n}]
       =0,

we also find

    |E[1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) g̃_j(X_k)]| 
        =|E[1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) {g̃_j(X_k)-g_0j(X_k)}]| 
       ≤|E[1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) {g̃_j(X_k)-g_l^*(X_k) } ]|
       + |E[1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) {g_l^*(X_k) - g_0j(X_k)}]| 
       ≤δ C_π_j+|1/√(n)E[∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) {g_l^*(X_k) - g_0j(X_k)}/√(n)g_l^*-g_0j_ng_l^*-g_0j_n ]| 
       ≤δ C_π_j+1/√(n)E{|ξ_l^*|(g̃_j-g_0j_n+δ)},

with

    ξ_l=∑_k=1^n ϵ_kjπ_j(X_1,…,X_n){g_l(X_k)-g_0j(X_k)}/√(n)g_l-g_0j_n .

Given X_1, …, X_n, ξ_l are sub-Gaussian with common parameter C_π_j. It follows from Lemma <ref> that E(ξ_l^*^2) ≤ E(max_l=1, …, N_δξ_l^2) ≲log N_δ. Using Cauchy-Schwarz inequality,

    E{|ξ_l^*|(g̃_j-g_0j_n+δ)}≲√(log N_δ){√(E(g̃_j-g_0j_n^2))+δ}.

Together with (<ref>) and (<ref>), the inequality in (II) follows.

(III): For any fixed g ∈𝒢, it follows from the definition of Δ_n that

    E[1/n∑_k=1^n{Z_kj-g̃_j(X_k)}^2] ≤ E[1/n∑_k=1^n{Z_kj-g(X_k)}^2]+Δ_n.
 
Using the inequality in (II), we have

    E(g̃_j-g_0j_n^2)     = E[1/n∑_k=1^n {g̃_j(X_k) - Z_kj + (ϵ_kj+u_nj)π_j(X_1,…,X_n)}^2]
       = E[1/n∑_k=1^n {g̃_j(X_k) - Z_kj}^2 + 1/n∑_k=1^n (ϵ_kj+u_nj)^2π_j^2(X_1,…,X_n) 
        + 2/n∑_k=1^n (ϵ_kj+u_nj)π_j(X_1,…,X_n) {g̃_j(X_k) - Z_kj}]
       ≤Δ_n + E[1/n∑_k=1^n {g(X_k) - Z_kj}^2 + 1/n∑_k=1^n (ϵ_kj+u_nj)^2π_j^2(X_1,…,X_n) 
        + 2/n∑_k=1^n (ϵ_kj+u_nj)π_j(X_1,…,X_n) {g̃_j(X_k) - Z_kj}]
       ≤Δ_n + E(g-g_0j_n^2) + E{2/n∑_k=1^n (ϵ_kj+u_nj)^2π_j^2(X_1,…,X_n)}
        + E[2/n∑_k=1^n (ϵ_kj+u_nj)π_j(X_1,…,X_n) {g̃_j(X_k) - Z_kj - g(X_k) + g_0j(X_k)}]
        = Δ_n +  E[{g(X)-g_0j(X)}^2] 
        + E[2/n∑_k=1^n (ϵ_kj+u_nj)π_j(X_1,…,X_n) {g̃_j(X_k) - g(X_k)}]
       ≲Δ_n + E[{g(X)-g_0j(X)}^2]
        + E{1/n∑_k=1^n ϵ_kjπ_j(X_1,…,X_n) g̃_j(X_k)} + |u_nj| 
       ≲Δ_n +  E[{g(X)-g_0j(X)}^2]
        +√(E{g̃_j-g_0j_n^2}log N_δ/n) + δ + |u_nj|.


Finally, applying (C.4) in <cit.>, the result follows.



Using Lemma <ref> to replace Lemma 4 of <cit.>, 




the remaining proof follows the proof of Theorem 1 and Corollary 1 in <cit.>. Let g̃_j = ĝ_j be the minimizer of empirical risk (<ref>), one has Δ_n = 0. Then for any i=1,…,n and j=1,…,r, we have

    E[{ĝ_j(X_i)-g_0j(X_i)}^2] ≲ |u_nj| + κ_n^2 log^3 n

and

    E[{ĝ_j(X)-g_0j(X)}^2] ≲ |u_nj| + κ_n^2 log^3 n.

The result follows.




































































 §.§ Proof of Proposition <ref>


We present the elementary results of an auxiliary Lemma <ref> and its proof, extending the well-known results from <cit.>. The key quantities of interest are μ_j = E{K_h(Z^0-z)(Z^0-z)^⊕ j}, τ_j(y) = E{K_h(Z^0-z)(Z^0-z)^⊕ j|Y=y} and the estimators μ̃_j=1/n∑_i=1^n K_h(Z^0_i-z)(Z^0_i-z)^⊕ j, for j=0,1,2. These quantities are fundamental in local linear smoother, as they relate to the bias and variance properties of local regression.


    Suppose <ref> and <ref> hold. Then,

    μ_0     = f_Z^0(z) K_10 + O(h^2),
    μ_1     = h^2  K_12∂ f_Z^0(z)/∂z + O(h^3 1),
    μ_2     = h^2 f_Z^0(z) K_12 + O(h^4 11^),

where K_1j = ∫_ℝ^r K(u) u^⊕ j du for j=0,2. Furthermore, we have

    μ̃_j=μ_j+O_p{(h^2 j-r n^-1)^1 / 21^⊕ j}
 and

    τ_0(y)     = f_Z^0|Y(z,y) K_10 + O(h^2),
    τ_1(y)     = h^2 K_12∂ f_Z^0|Y(z,y)/∂z + O(h^31),
    τ_2(y)     = h^2 f_Z^0|Y(z,y) K_12 + O(h^4 11^),

where the order term is uniform over y ∈Ω.



By definition, we have

    μ_j     = E{K_h(Z^0-z)(Z^0-z)^⊕ j}
        = ∫ h^-rK{H^-1(x-z)}  (x-z)^⊕ j f_Z^0(x) dx,


    τ_j(y)     = E{K_h(Z^0-z)(Z^0-z)^⊕ j|Y=y}
        = ∫ h^-rK{H^-1(x-z)}  (x-z)^⊕ j f_Z^0|Y(x,y) dx.

The statements regarding μ_j and τ_j(y) follow from Assumptions <ref> and <ref> using a second-order Taylor expansion of the densities f_Z^0 and f_Z^0|Y.

    μ_0     = ∫ h^-rK{H^-1(x-z)}f_Z^0(x) dx
        = ∫ K(u)f_Z^0(z + Hu) du
        = f_Z^0(z) K_10 + h (∂ f_Z^0(z)/∂z)^ K_11 + O(h^2)
        = f_Z^0(z) K_10 + O(h^2).


    μ_1     = ∫ h^-rK{H^-1(x-z)}  (x - z) f_Z^0(x) dx
        = h ∫u K(u) f_Z^0(z + Hu) du
        = h { f_Z^0(z) K_11 + h  K_12∂ f_Z^0(z)/∂z + O(h^2)}
        = h^2  K_12∂ f_Z^0(z)/∂z + O(h^3 1)).


    μ_2     = ∫ h^-rK{H^-1(x-z)}  (x - z)(x - z)^ f_Z^0(x) dx
        = h^2 ∫uu^ K(u) f_Z^0(z + Hu) du
        = h^2{f_Z^0(z) K_12 + h  ∫uu^u^∂ f_Z^0(z)/∂z K(u)du + O(h^2 11^)}
        = h^2 f_Z^0(z) K_12 + O(h^4 11^).


    τ_0(y)     = ∫ h^-rK{H^-1(x-z)} f_Z^0|Y(x,y) dx
        = ∫ K(u)f_Z^0|Y(z + Hu,y) du
        = f_Z^0|Y(z,y) K_10 + h (∂ f_Z^0|Y(z,y)/∂z)^ K_11 + O(h^2)
        = f_Z^0|Y(z,y) K_10 + O(h^2).


    τ_1(y)     = ∫ h^-rK{H^-1(x-z)}  (x-z) f_Z^0|Y(x,y) dx
        = h ∫u K(u)f_Z^0|Y(z + Hu,y) du
        = h{f_Z^0|Y(z,y) K_11 + h K_12∂ f_Z^0|Y(z,y)/∂z  + O(h^21)}
        = h^2 K_12∂ f_Z^0|Y(z,y)/∂z + O(h^31).


    τ_2(y)     = ∫ h^-rK{H^-1(x-z)}  (x - z)(x - z)^ f_Z^0|Y(x,y) dx
        = h^2 ∫uu^ K(u) f_Z^0|Y(z+Hu,y) du
        = h^2{f_Z^0|Y(z,y) K_12 + h  ∫uu^u^∂ f_Z^0|Y(z,y)/∂z K(u)du + O(h^2 11^)}
        = h^2 f_Z^0|Y(z,y) K_12 + O(h^4 11^).

Next, by definition

    μ̃_j=1/n∑_i=1^n K_h(Z^0_i-z)(Z^0_i-z)^⊕ j.

Note that E(μ̃_j)=μ_j and

    E{K_h^2(Z^0_i-z)(Z_il-z_l)^2 j}=h^2 j-r∫ K(u)^2 u_l^2 j f_Z^0(z+Hu) d u=O(h^2 j-r),

where Z_il is the lth element of the random vector Z^0_i and z_l is the lth element of the vector z. It follows that (μ̃_j)=O(h^2 j-r n^-11^⊕ j), proving the result for the μ̃_j.



    Suppose <ref> and <ref> hold and furthermore h → 0, n h^r →∞. Then d{ṽ_h(z), v_h(z)}=o_p(1).



The symbol ⇝ denotes weak convergence and the notation l^∞(Ω) denotes the space of bounded functions on Ω. According to Corollary 3.2.3 in <cit.> and Assumption <ref> (i), it is sufficient to 

demonstrate the convergence of sup_y∈Ω|Q̃_h(y,z)-Q_h(y,z)| to zero in probability. Achieving this requires showing Q_h(·,z)-Q̃_h(·,z) ⇝ 0 in l^∞(Ω) and then applying Theorem 1.3.6 of <cit.>. Finally, by Theorem 1.5.4 and 1.5.7 in <cit.>, we can establish weak convergence by showing the following:


  
  * Q̃_h(y,z)-Q_h(y,z)=o_p(1) for all y∈Ω and
  
  * Q̃_h(·,z)-Q_h(·,z) is asymptotically equicontinuous in probability, i.e., for all ϵ,η>0, there exists δ>0 such that
  
    lim sup_n P(sup _d(y_1, y_2)<δ|{Q̃_h(y_1,z)-Q_h(y_1,z)} - {Q̃_h(y_2,z)-Q_h(y_2,z)}|>ϵ)<η.


To begin with <ref>, recall that 

    w(Z^0_i, z, h)=1/σ_0^2K_h(Z^0_i-z)[1-μ_1^μ_2^-1(Z^0_i-z)]

and 

    w̃(Z^0_i, z, h)=1/σ̃_0^2 K_h(Z^0_i-z)[1-μ̃_1^μ̃_2^-1(Z^0_i-z)],
 
where σ_0^2 = μ_0 - μ_1^μ_2^-1μ_1 and σ̃_0^2 = μ̃_0 - μ̃_1^μ̃_2^-1μ̃_1. Then, one observes that

    Q̃_h(y,z)-Q_h(y,z) =    1/n∑_i=1^n{w̃(Z^0_i, z, h)-w(Z^0_i, z, h)} d^2(Y_i, y) + 
       1/n∑_i=1^n[w(Z^0_i, z, h) d^2(Y_i, y)-E{w(Z^0_i, z, h) d^2(Y_i, y)}].

Observe that w̃(Z^0_i, z, h)-w(Z^0_i, z, h)=W_0 n K_h(Z^0_i-z)+K_h(Z^0_i-z)W_1 n(Z^0_i-z), where

    W_0 n=1/σ̃_0^2-1/σ_0^2,    W_1 n=μ̃_1^μ̃_2^-1/σ̃_0^2-μ_1^μ_2^-1/σ_0^2.

Using the results of Lemma <ref> and Sherman-Morrison formula, it follows that 

    μ̃_2^-1μ̃_1 - μ_2^-1μ_1 
    
            =    [μ_2+O_p{(h^4-r n^-1)^1/211^}]^-1[μ_1+O_p{(h^2-r n^-1)^1/21}] - μ_2^-1μ_1 
    
            =    [μ_2^-1 - μ_2^-1O_p{(h^4-r n^-1)^1/211^}μ_2^-1/1 + 1^μ_2^-11]
            [μ_1+O_p{(h^2-r n^-1)^1/21}]- μ_2^-1μ_1 
     
            =    O_p{(nh^2+r)^-1 / 21}.

Furthermore, it is easy to get σ̃^2_0 - σ_0^2=O_p{(n h^r)^-1 / 2}. Therefore, W_0 n=O_p{(n h^r)^-1 / 2} and W_1 n=O_p{(nh^2+r)^-1 / 21}. Since for j =0, 1, we have

    E{K_h(Z^0_i-z)(Z_il-z_l)^j d^2(Y_i, y)}=O(h^j), 
        E{K_h^2(Z^0_i-z)(Z_il-z_l)^2 j d^4(Y_i, y)}=O(h^2 j-r),


it follows that the first term in (<ref>) is O_p((n h^r)^-1 / 2). We find that E{w^2(Z^0_i,z, h)}=O(h^-r), so the second term in (<ref>) is O_p{(n h^r)^-1 / 2} as well. Then, we show that Q̃_h(y,z)-Q_h(y,z)=o_p(1) for any y ∈Ω and any z∈ℝ^r, since n h^r →∞. 

Moving on to <ref>, for any y_1,y_2∈Ω,

    |{Q̃_h(y_1,z)-Q_h(y_1,z)} - {Q̃_h(y_2,z)-Q_h(y_2,z)}|
    ≤    |Q̃_h(y_1,z)-Q̃_h(y_2,z)| + |Q_h(y_1,z)-Q_h(y_2,z)|
    ≤   1/n∑_i=1^n|w̃(Z^0_i, z, h)||d(Y_i, y_1)-d(Y_i, y_2)||d(Y_i, y_1)+d(Y_i, y_2)| 
        + |E[w(Z^0_i, z, h){d(Y_i, y_1)+d(Y_i, y_2)}{d(Y_i, y_1)-d(Y_i, y_2)}]|
    ≤    2 diam(Ω) d(y_1,y_2) 1/n∑_i=1^n {|w̃(Z^0_i, z, h)| + E|w(Z^0_i, z, h)|}
    
    =    O_p{d(y_1,y_2)},

since E{|w(Z^0_i, z, h)|}=O(1), E{w^2(Z^0_i, z, h)}=O(h^-1), and n^-1∑_i=1^n|w̃(Z^0_i, z, h)|=O_p(1). Therefore,

    sup_d(y_1,y_2)<δ|{Q̃_h(y_1,z)-Q_h(y_1,z)} - {Q̃_h(y_2,z)-Q_h(y_2,z)}| = O_p(δ),

which implies <ref>. These show that d{ṽ_h(z),v_h(z)} = o_p(1).



Firstly, we prove (<ref>). Similar to the proof of Theorem 3 in <cit.>, we can show 

    d F_Y|Z^0(z,y)/d F_Y(y) = f_Z^0|Y(z,y)/f_Z^0(z)

for all z given that f_Z^0(z)>0. Then by Lemma <ref>,

    ∫ w(x,z,h)d F_Z^0|Y(x|y)    = τ_0(y) -μ_1^μ_2^-1τ_1(y)/μ_0 -μ_1^μ_2^-1μ_1 = f_Z^0|Y(z,y)/f_Z^0(z) + O(h^2)

where the error term is uniform over y∈Ω. Hence,

    Q_h(ω,z)     = ∫ d^2(y,ω)w(x,z,h)dF_Z^0,Y(x,y) = ∫ w(x,z,h)dF_Z^0|Y(x,y) d^2(y, ω) dF_Y(y)
       = ∫f_Z^0|Y(z,y)/f_Z^0(z)  d^2(y, ω) dF_Y(y) + O(h^2) = ∫ d^2(y, ω) d F_Y|Z^0(z,y) + O(h^2)
       = Q(ω, z) + O(h^2),

where the error term is now uniform over ω∈Ω. By the first two parts of Assumption <ref> and employing proof by contradiction, we then have d{v_h(z), v(z)}=o(1) as h=h_n→ 0.

Next, define r_h=h^-γ_1/γ_1-1 and set 

    S_j,h(z)={y: 2^j-1 < r_h d{y, v(z)}^γ_1/2≤ 2^j}.

Let I denote the indicator function. Then, for any M>0, there exists a>0 such that, for large n,

    I[r_h d{v_h(z), v(z)}^γ_1/2>2^M]     = ∑_j ≥ M I{v_h(z)∈ S_j,h(z)}
       ≤∑_j ≥ M I(sup_y∈ S_j,h(z) [Q_h{v(z),z} - Q_h(y,z)] ≥ 0).

By the first two parts of Assumption <ref>, for every j involved in the sum, we have, for every y∈ S_j,h(z),

    Q{v(z),z} - Q(y,z) ≲ -d{v(z),y}^γ_1≲ -2^2(j-1)/r_h^2.

Define V(y,z) = Q_h(y,z) - Q(y,z) and we have

    sup_y∈ S_j,h(z)| V(y,z) - V{v(z),z}| 
    ≥   sup_y∈ S_j,h(z)[ Q_h{v(z),z} - Q_h(y,z)] -sup_y∈ S_j,h(z)[ Q{v(z),z} - Q(y,z)].


Hence,

    I[r_h d{v_h(z), v(z)}^γ_1/2>2^M]    ≤∑_j ≥ M I[sup_y∈ S_j,h(z) |V(y,z) - V{v(z),z}| ≥2^2(j-1)/r_h^2]
       ≤∑_j ≥ M I{a h^2 (2^j/r_h)^2/γ_1≥2^2(j-1)/r_h^2}
       ≤∑_j ≥ M I{4a h^2 (2^2j/γ_1-2j/r_h^2/γ_1-2)≥ 1}
       ≤ 4a ∑_j ≥ M2^2 j(1-γ_1) / γ_1/r_h^2(1-γ_1) / γ_1 h^-2≤ 4a ∑_j ≥ M(1/4^(γ_1-1) / γ_1)^j,

which converges since γ_1>1. Thus, for some M>0, we have

    d{v_h(z), v(z)}≤ 2^2M/γ_1  h^2/(γ_1-1)

for large n and hence

    d{v_h(z), v(z)}=O(h^2/(γ_1-1))

for large n, so we showed (<ref>).

Secondly, we prove (<ref>). Define T_n(y,z)=Q̃_h(y,z)-Q_h(y, z). Letting

    D_i(y, z)=d^2(Y_i, y)-d^2{Y_i, v_h(z)},

we have

    |T_n(y,z)-T_n{v_h(z),z}| ≤   |1/n∑_i=1^n{w̃(Z^0_i, z, h)-w(Z^0_i, z, h)} D_i(y, z)|  
        +|1/n∑_i=1^n[w(Z^0_i, z, h) D_i(y, z)-E{w(Z^0_i, z, h) D_i(y, z)}]|.


Since W_0 n and W_1 n from (<ref>) are O_p{(n h^r)^-1/2} and O_p{(n h^2+r)^-1 / 21}, respectively, and using the fact that |D_i(y, z)| ≤ 2 diam(Ω) d(y, v_h(z)), the first term on the right-hand side of (<ref>) is O_p[(n h^r)^-1/2 d{y, v_h(z)}], where the O_p term is independent of Ω and v_h(z). Thus, we can define

    B_R={sup _d(y, v_h(z))<δ|1/n∑_i=1^n{w̃(Z^0_i, z, h)-w(Z^0_i, z, h)} D_i(y, z)| ≤ R δ(n h^r)^-1 / 2}

for R>0, so that P(B_R^c) → 0.

Next, to control the second term on the right-hand side of (<ref>), define the functions g_y: ℝ^r ×Ω↦ℝ by

    g_y(x, w)=1/σ_0^2 K_h(x-z){1-μ_1^μ_2^-1(x-z)} d^2(y, w),

and the corresponding function class

    ℱ_n δ={g_y-g_v_h(z): d{y, v_h(z)}<δ}.

An envelope function for ℱ_n δ is

    F_n δ(z)=2 diam(Ω) δ/σ_0^2 K_h(z-x)|1-μ_1^μ_2^-1(z-x)|,

and E{F_n δ^2(Z^0)}=O(δ^2 h^-r). According to Theorem 2.7.17 of <cit.> and Assumption <ref>, we know the bracketing integral of the class ℱ_nδ

    J_[](ϵ, ℱ_nδ,·_Z^0,Y,2)=∫_0^ϵ√(1+log N_[](t{E(F_nδ^2)}^1/2, ℱ_nδ,·_Z^0,Y,2)) d t

is O(1), where N_[](t{E(F_nδ^2)}^1/2, ℱ_nδ,·_Z^0,Y,2) denotes the minimum number of t{E(F_nδ^2)}^1/2-brackets needed to cover the function class ℱ_nδ (see Definition 2.1.6 of <cit.>) and ·_Z^0,Y,2 denotes the norm that for any f∈ℱ_nδ, f_Z^0,Y,2=[E{f(Z^0,Y)^2}]^1/2. Using these facts together with Theorems 2.14.16 of <cit.> and Assumption <ref>, for small δ,

    E(sup _d{y, v_h(z)}<δ|1/n∑_i=1^n w(Z^0_i, z, h) D_i(y, z)-E{w(Z^0_i, z, h) D_i(y, z)}|)
       ≤J_[](ϵ, ℱ_nδ,·_Z^0,Y,2)  [E{F_n δ^2(Z^0)}]^1/2/√(n)
       =O(δ(n h^r)^-1 / 2).

Combining this with (<ref>) and the definition of B_R,

    E(I_B_Rsup _d{y, v_h(z)}<δ|T_n(y,z)-T_n{v_h(z),z}|) ≤a δ/(n h^r)^1 / 2,

where I_B_R is the indicator function for the set B_R and a is a constant depending on R and the entropy integral in Assumption <ref>.

To finish, set r_n=(n h^r)^γ_2/4(γ_2-1) and define

    S_j, n(z)={y: 2^j-1< r_n d{y, v_h(z)}^γ_2/2≤ 2^j} .

Choose η_2 satisfying Assumption <ref> (ii) and such that Assumption <ref> is satisfied for any δ<η_2. Set η=(η_2 / 2)^γ_2 / 2. Denote events A_η_2 = {d{ṽ_h(z),v_h(z)}>η_2/2} and C_M = {r_n d{ṽ_h(z),v_h(z)}^γ_2/2>2^M)}. If r_n d{ṽ_h(z),v_h(z)}^γ_2/2 is larger than 2^M for a given integer M, then ṽ_h(z) is in one of the shells S_j,n(z) with j>M such that the supremum of the map y↦Q̃_h{v_h(z),z} - Q̃_h(y,z) over this shell is nonnegative by the property of ṽ_h(z).

    P(C_M) ≤    P(B_R^c) + P(A_η_2) + P(B_R∩ A_η_2^c∩ C_M)
    ≤    P(B_R^c) + P(A_η_2)  + ∑_j≥ M
     2^j≤ r_n ηP({sup_y∈ S_j,n(z)| Q̃_h{v_h(z), z} - Q̃_h(y, z)≥ 0 |}∩ B_R),

where the first term goes to zero clearly and the second term goes to zero by Lemma <ref>. Choose η_2>0 small enough that Assumption <ref> (ii) holds for every d{y,v_h(z)}≤η and (<ref>) holds for every δ≤η. Then for every j involved in the sum, we have, for every y∈ S_j,n(z),

    Q_h(v_h(z),z) - Q_h(y,z)≲ - d{v_h(z),y}^γ_2≲ - 2^2(j-1)/r_n^2.

In terms of T_n(y,z) = Q̃_h(y,z) - Q_h(y,z), by Markov's inequality, we have

    P(C_M) ≤    P(B_R^c) + P(A_η_2) 
        + ∑_j≥ M
     2^j≤ r_n ηP({sup_y∈ S_j,n| T_n(y,z) - T_n{v_h(z),z}|≳2^2(j-1)/r_n^2}∩ B_R)
    ≲   ∑_j≥ M
     2^j≤ r_n η(2^j/r_n)^2/γ_2 (nh^r)^-1/2/2^2j/r_n^2
    ≲   ∑_j≥ M2^2j(1-γ_2)/γ_2/r_n^2(1-γ_2)/γ_2√(nh^r)
    
         =    ∑_j≥ M (1/4^(γ_2-1)/γ_2)^j,

which converges since γ_2>1. Hence, as M→∞,

    P(d{ṽ_h(z),v_h(z)}>(2^M/r_n)^2/γ_2)→ 0.

Therefore,

    d{ṽ_h(z),v_h(z)} = O_p{(n h^r)^-1/{2(γ_2-1)}}.





 §.§ Proof of Proposition <ref>















































Suppose <ref> holds. If Ẑ_i-Z^0_i=O_p(ζ_n) for i=1, …, n, Ẑ-Z^0=O_p(ζ_n), nh^r→∞, and h^-r-2ζ_n→0, then the weight functions as defined in (<ref>) and (<ref>) satisfy

    1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}=O_p(h^-r-1ζ_n).



Recall that 

    ŵ(Ẑ_i, Ẑ, h)=1/σ̂^2K_h(Ẑ_i-Ẑ)[1-μ̂_1^μ̂_2^-1(Ẑ_i-Ẑ)],

where 

    σ̂^2=μ̂_0-μ̂_1^μ̂_2^-1μ̂_1,  μ̂_j=1/n∑_i=1^nK_h(Ẑ_i-Ẑ)(Ẑ_i-Ẑ)^⊕ j, j=0, 1, 2

and 

    w̃(Z_i^0, Z^0, h)=1/σ̃^2K_h(Z^0_i-Z^0)[1-μ̃_1^μ̃_2^-1(Z^0_i-Z^0)],

where 

    σ̃^2=μ̃_0-μ̃_1^μ̃_2^-1μ̃_1,  μ̃_j=1/n∑_i=1^nK_h(Z^0_i-Z^0)(Z^0_i-Z^0)^⊕ j, j=0, 1, 2

with K_h(z)=h^-rK{H^-1z}. 

Define

    ρ_j(x, z)=K_h(x-z)(x-z)^⊕ j,  x, z∈𝒯^r.

By Assumption <ref>, K(·) is Lipschitz continuous. For any x_1, x_2, z_1, z_2∈𝒯^r, one has

    |ρ_0(x_1, z_1)-ρ_0(x_2, z_2)|   =|K_h(x_1-z_1)-K_h(x_2-z_2)|
       ≲ h^-r-1x_1-z_1-(x_2-z_2)
       ≤ h^-r-1(x_1-x_2+z_1-z_2).

For j=1, 2, it hold that

    ρ_j(x_1, z_1)-ρ_j(x_2, z_2)   =K_h(x_1-z_1)(x_1-z_1)^⊕ j-K_h(x_2-z_2)(x_2-z_2)^⊕ j
       ≤K_h(x_1-z_1)(x_1-z_1)^⊕ j-K_h(x_2-z_2)(x_1-z_1)^⊕ j+
       K_h(x_2-z_2)(x_1-z_1)^⊕ j-K_h(x_2-z_2)(x_2-z_2)^⊕ j
       ≤(x_1-z_1)^⊕ j·|K_h(x_1-z_1)-K_h(x_2-z_2)|+
       K_h(x_2-z_2)·(x_1-z_1)^⊕ j-(x_2-z_2)^⊕ j.

Using the Lipschitz continuity of K(·) by Assumption <ref> and the fact that K_h(x-z)=0 if any component of |x-z| exceeds h, for j=1, 2 one has

    ρ_j(x_1, z_1)-ρ_j(x_2, z_2)   ≲ h^j-r-1x_1-z_1-(x_2-z_2)+
       h^j-1x_1-z_1-(x_2-z_2)
       ≲ h^-r-1+jx_1-z_1-(x_2-z_2)
       ≤ h^-r-1+j(x_1-x_2+z_1-z_2).

One thus has

    μ̂_j-μ̃_j   =1/n∑_i=1^nρ_j(Ẑ_i, Ẑ)-1/n∑_i=1^nρ_j(Z^0_i, Z^0)
       ≤1/n∑_i=1^nρ_j(Ẑ_i, Ẑ)-ρ_j(Z^0_i, Z^0)
       ≲1/n∑_i=1^nh^-r-1+j(Ẑ_i-Z^0_i+Ẑ-Z^0).

We conclude that μ̂_j-μ̃_j=O_p(h^-r-1+jζ_n) for j=0, 1, 2. Since nh^r→∞, by Lemma <ref> one has μ̃_0=O_p(1), μ̃_1=O_p(h^2), and μ̃_2=O_p(h^2). It follows that 

    |σ̂^2-σ̃^2|   =|(μ̂_0-μ̂_1^μ̂_2^-1μ̂_1)-(μ̃_0-μ̃_1^μ̃_2^-1μ̃_1)|
       =O_p(h^-r-1ζ_n)

and 

    μ̂_1^μ̂_2^-1/σ̂^2-μ̃_1^μ̃_2^-1/σ̃^2=O_p(h^-r-2ζ_n),

Observe that 

    ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)   ={σ̂^-2-σ̃^-2}ρ_0(Z^0_i, Z^0)+σ̂^-2{ρ_0(Ẑ_i, Ẑ)-ρ_0(Z^0_i, Z^0)}-
       {μ̂_1^μ̂_2^-1/σ̂^2-μ̃_1^μ̃_2^-1/σ̃^2}ρ_1(Z^0_i, Z^0)-
       μ̂_1^μ̂_2^-1/σ̂^2{ρ_1(Ẑ_i, Ẑ)-ρ_1(Z^0_i, Z^0)}

and E{ρ_j(Z^0_i, Z^0)}=O(h^j1^⊕ j), E[{ρ_j(Z^0_i, Z^0)}^⊕ 2]=O(h^2j-r1^⊕ 2j) for j=0, 1. 
We have

    1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}=O_p(h^-r-1ζ_n).













    
    Suppose <ref> and the last two parts of <ref>. If Ẑ_i-Z^0_i=O_p(ζ_n) for i=1, …, n, Ẑ-Z^0=O_p(ζ_n), nh^r→∞, and h^-r-2ζ_n→0, then the following holds,
    
    d{v̂_h(Ẑ), ṽ_h(Z^0)}=o_p(1).







For any y∈Ω, observe that

    Q̂_h(y, Ẑ)-Q̃_h(y, Z^0)   =1/n∑_i=1^nŵ(Ẑ_i, Ẑ, h)d^2(Y_i, y)-1/n∑_i=1^nw̃(Z_i^0, Z^0, h)d^2(Y_i, y)
       =1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}d^2(Y_i, y)
       =O_p(h^-r-1ζ_n)

by Lemma <ref> and the fact that Ω is totally bounded. By the last two parts of Assumption <ref> and employing proof by contradiction, one concludes that d{v̂_h(Ẑ), ṽ_h(Z^0)}=o_p(1).






























Define V_n(y)=Q̂_h(y, Ẑ)-Q̃_h(y, Z^0). Letting D_i(y, Z^0)=d^2(Y_i, y)-d^2{Y_i, ṽ_h(Z^0)}, we have

    |V_n(y)-V_n{ṽ_h(Z^0)}|=|1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}D_i(y, Z^0)|.

By Lemma <ref>, and using the fact that |D_i(y, Z^0)|≤2diam(Ω)d{y, ṽ_h(Z^0)}, the right-hand side of (<ref>) is O_p[d{y, ṽ_h(Z^0)} h^-r-1ζ_n]. Thus, we can define

    B_R={sup_d{y, ṽ_h(Z^0)}<δ|V_n(y)-V_n{ṽ_h(Z^0)}|≤ Rδ h^-r-1ζ_n}

for R>0, so that P(B_R^C)→0.

Set t_n=(h^-r-1ζ_n)^-γ_3/2(γ_3-1). For each n, the metric space Ω (minus the point ṽ_h(Z^0)) can be partitioned into the “shells”

    S_j, n(Z^0)={y: 2^j-1<t_nd{y, ṽ_h(Z^0)}^γ_3/2≤2^j},

with j varying over the integers. Choose η=(η_3/2)^γ_3/2>0 small enough that Assumption <ref> (iii) holds and δ<η_3. If t_nd{v̂_h(Ẑ), ṽ_h(Z^0)}^γ_3/2 is larger than 2^M for a given integer M, then v̂_h(Ẑ) is in one of the shells S_j, n(Z^0) with j>M. In that case the supremum of the map y↦Q̂_h(ṽ_h(Z^0), Ẑ)-Q̂_h(y, Ẑ) over this shell is nonnegative by the optimality of v̂_h(Ẑ). Conclude that, for every η>0,

    P(t_nd{v̂_h(Ẑ), ṽ_h(Z^0)}^γ_3/2>2^M)
    ≤   P(B_R^C)+P(2d{v̂_h(Ẑ), ṽ_h(Z^0)}>η_3)
        + ∑_j>M
    2^j≤η t_nP({sup_y∈ S_j, n(Z^0)[Q̂_h{ṽ_h(Z^0), Ẑ}-Q̂_h(y, Ẑ)]≥0}∩ B_R),

where P(B_R^C)→0 as discussed previously and the second term goes to zero by Lemma <ref>. Then for every j involved in the sum, by Assumption <ref> (iii) we have, for every y∈ S_j, n(Z^0),

    Q̃_h{ṽ_h(Z^0), Z^0}-Q̃_h(y, Z^0)≲-2^2(j-1)/t_n^2.

In terms of the centered process V_n, by Markov's inequality the right-hand side of (<ref>) may be bounded by

    ∑_j>M
    2^j≤η t_nP({sup_y∈ S_j, n(Z^0)[V_n(y)-V_n{ṽ_h(Z^0)}]≳2^2(j-1)/t_n^2}∩ B_R)
       ≲∑_j>Mt_n^2/2^2j(2^j/t_n)^2/γ_3 h^-r-1ζ_n
       =∑_j>M(1/4^(γ_3-1)/γ_3)^j,

which converges to zero for every M=M_n→∞ since γ_3>1. Hence

    d{v̂_h(Ẑ), ṽ_h(Z^0)}=O_p(t_n^-2/γ_3)=O_p{(h^-r-1ζ_n)^1/(γ_3-1)}.



    P(inf_d{ṽ_h(Z^0), y}≥η_3[Q̃_h(y, Z^0)-Q̃_h{ṽ_h(Z^0), z}-C_3η_3^γ_3]≥0)→0.

For any y∈Ω, observe that

    |Q̂_h(y, Ẑ)-Q̃_h(y, Z^0)|   =|1/n∑_i=1^nŵ(Ẑ_i, Ẑ, h)d^2(Y_i, y)-1/n∑_i=1^nw̃(Z_i^0, Z^0, h)d^2(Y_i, y)|
       =|1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}d^2(Y_i, y)|
       =O_p(h^-r-1ξ_n)

by Lemma <ref> and the fact that Ω is totally bounded. It follows that

    Q̃_h{v̂_h(Ẑ), z}-Q̃_h{ṽ_h(Z^0), z}   =Q̃_h{v̂_h(Ẑ), z}-Q̂_h{v̂_h(Ẑ), z}+Q̂_h{v̂_h(Ẑ), z}-Q̃_h{ṽ_h(Z^0), z}
       ≤Q̃_h{v̂_h(Ẑ), z}-Q̂_h{v̂_h(Ẑ), z}+Q̂_h{ṽ_h(Z^0), z}-Q̃_h{ṽ_h(Z^0), z}
       =O_p(h^-r-1ξ_n).

where the inequality follows from the optimality of v̂_h(Ẑ), i.e., Q̂_h{ṽ_h(Z^0), z}≤Q̂_h{v̂_h(Ẑ), z}. Now Assumption <ref> indicates that

    d{v̂_h(Ẑ), ṽ_h(Z^0)}=O_p{(h^-r-1ξ_n)^1/γ_3},

the result follows.




 §.§ Proof of Theorem <ref>


Combining Theorems, <ref> and Proposition <ref>, and <ref>, it follows from the triangle inequality that

    d{m̂(X), m(X)}    = d{v̂_h(Ẑ), v(Z^0)}
       ≤ d{ṽ_h(Z^0), v(Z^0)}+d{v̂_h(Ẑ), ṽ_h(Z^0)}
        = O_p{h^2/(γ_1-1)+(nh^r)^-1/{2(γ_2-1)}+(h^-r-1ζ_n)^1/(γ_3-1)},

where Z^0 = g_0(X) and Ẑ = ĝ(X).






 §.§ Additional Proof

If we have the supremum rate, E(sup_i=1,…,nẐ_i-Z^0_i)=O(τ_n), we can substitute Lemma <ref> with Lemma <ref> below. This substitution would lead to an improvement in the convergence rate outlined in Proposition <ref>, resulting in (h^-1τ_n)^1/(γ_3-1). Furthermore, an enhanced convergence rate can be achieved in Theorem <ref>, expressed as h^2/(γ_1-1)+(nh^r)^-1/2(γ_2-1)+(h^-1τ_n)^1/(γ_3-1).



Suppose <ref> and <ref> hold. If E(sup_i=1,…,nẐ_i-Z^0_i)=O(τ_n), nh^r→∞ and h^-r-1τ_n→0, then the weight functions as defined in (<ref>) and (<ref>) satisfy


    1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}=O_p(h^-1τ_n).
















Recall that in Lemma <ref>, for any x_1, x_2, z_1, z_2∈𝒯^r, one has






    ρ_j(x_1, z_1)-ρ_j(x_2, z_2)≲ h^-r-1+j(x_1-x_2+z_1-z_2).

One thus has

    μ̂_j-μ̃_j   =1/n∑_i=1^nρ_j(Ẑ_i, Ẑ)-1/n∑_i=1^nρ_j(Z^0_i, Z^0)
       ≤1/n∑_i=1^nρ_j(Ẑ_i, Ẑ)-ρ_j(Z^0_i, Z^0)·[𝐈{ρ_j(Ẑ_i, Ẑ) 0} + 𝐈{ρ_j(Z^0_i, Z^0) 0}] 
       ≤1/n h^r+1-j ( sup_i=1,…,nẐ_i-Z_i^0 + Ẑ-Z^0)
       ∑_i=1^n[𝐈(Ẑ_i-Ẑ≤ h) + 𝐈(Z_i^0-Z^0≤ h)]
       ≲τ_n/n h^r+1-j∑_i=1^n [𝐈(Ẑ_i-Ẑ≤ h) + 𝐈(Z_i^0-Z^0≤ h) ].

Given Z^0, observe 

    1/nh^r∑_i=1^n 𝐈(Z_i^0-Z^0≤ h)P→ c_1 f_Z^0(Z^0)

for some constant c_1<∞, and that

    1/nh^r∑_i=1^n𝐈(Ẑ_i-Ẑ≤ h)
    ≤   1/nh^r∑_i=1^n𝐈(Z^0_i-Z^0≤ 3h) + 𝐈(Ẑ_i-Z_i^0≥ h) + 𝐈(Ẑ -Z^0≥ h)

converges to c_2 f(Z_i^0) in probability for some constant c_2<∞. Here, 

    1/n h^r∑_i=1^n𝐈(Ẑ_i-Z_i^0≥ h) and 1/nh^r∑_i=1^n𝐈(Ẑ - Z^0≥ h)
 are both o_p(1) since by the Markov inequality and Theorem <ref>,

    E{1/n h^r∑_i=1^n𝐈(Ẑ_i-Z_i^0≥ h)}≤1/n h^r+1∑_i=1^n EẐ_i-Z_i^0≤1/n∑_i=1^n h^-r-1τ_n → 0,
       E{1/n h^r∑_i=1^n𝐈(Ẑ - Z^0≥ h)}≤1/n h^r+1∑_i=1^n EẐ - Z^0≤1/n∑_i=1^n h^-r-1τ_n → 0.

This implies that μ̂_j-μ̃_j=O_p(h^j-1τ_n) for j=0, 1, 2. It follows that 

    |σ̂^2-σ̃^2|=O_p(h^-1τ_n)   and  μ̂_1^μ̂_2^-1/σ̂^2-μ̃_1^μ̃_2^-1/σ̃^2=O_p(h^-2τ_n).

Thus, due to E{ρ_j(Z^0_i, Z^0)}=O(h^j1^⊕ j) and E[{ρ_j(Z^0_i, Z^0)}^⊕ 2]=O(h^2j-r1^⊕ 2j) for j=0, 1, we have

    1/n∑_i=1^n{ŵ(Ẑ_i, Ẑ, h)-w̃(Z_i^0, Z^0, h)}=O_p(h^-1τ_n).














  * Framework:

    ℝ^pℝ^r[ISOMAP: ψ]LFR: ϕΩ


  * Data: {(X_i, Y_i)}_i=1^n, where X_i∈ℝ^p, Y_i∈Ω.


  * Proof outlines:

    
  * Deep learning: <cit.>.
    
  * Manifold learning: <cit.> and possibly Alvaro's manuscript.
    
  * Errors-in-variables: examine the difference in the weights; see Appendix F of <cit.>, and D.2, D.3 of <cit.>.
    
  * Multivariate local regression: <cit.>, <cit.>, and <cit.>.


  * Difficulties:

    
  * ISOMAP manifold coordinates are not directly observed: α_n = sup_1≤ i≤ nψ̂(Y_i)-ψ(Y_i). Any influence on the asymptotic property of DNN?
    
  * i.i.d. assumption is in fact not satisfied for DNN. Ẑ_i = ψ(Y_i) is not i.i.d.
    
    Ẑ_i = g_0(X_i) + ϵ_i

    
  * error term, ϵ_i, need to be i.i.d. standard normal.
    
  * The theory in <cit.> is only applicable for scalar responses. Need to think about the extension to vector responses. Use sequential learning.
    
  * Uniform rate of convergence for local regression seems not directly extendable to multivariate cases, as the rate in <cit.> is suboptimal compared to usual local linear regression.






 §.§ Additional Simulations for Networks


We consider graph Laplacians of undirected weighted networks with a fixed number of nodes m and bounded edge weights as a second type of metric-space valued responses; see Example <ref>. The space of graph Laplacians <cit.> is

    Ω={Y=(y_i j): Y=Y^T ; Y 1_m=0_m ;-W ≤ y_i j≤ 0  for i ≠ j},

where 1_m and 0_m are the m-vectors of ones and zeroes, respectively. We aim to construct a generative model that produces random graph Laplacians Y along with an Euclidean predictor X∈ℝ^9. 

Denote the half vectorization excluding the diagonal of a symmetric and centered matrix by vech, with inverse operation vech^-1. By the symmetry and centrality of graph Laplacians as per (<ref>), every graph Laplacian Y is fully determined by its upper (or lower) triangular part, which can be further vectorized into vech(Y), a vector of length d=m(m-1) / 2. The true regression function can thus be defined as

    m(X)=vech^-1[{-E(l_1|X), …, -E(l_d|X)}^].


To generate random response Y, each entry of the random vector (l_1, …, l_d)^ is first generated using beta distributions whose parameters depend on the predictor X. The random response Y is then generated conditional on X through an inverse half vectorization vech^-1 applied to (-l_1, …,-l_d)^. The space of graph Laplacians Ω is not a vector space. Instead, it is a bounded, closed, and convex subset in ℝ^m^2 of dimension m(m-1)/2. To ensure that the random response Y generated in simulations resides in L_m, the off-diagonal entries l_j, j = 1, …, d, need to be nonpositive and bounded below as per (<ref>). To this end, we consider

    Y=vech^-1(-l_1, …,-l_d),  where  l_j i.i.d.∼Beta(α_1, α_2),
       α_1 = X_8sin(π X_1) +(1-X_8)cos(π X_2),
       α_2 = X_4^2 X_7+X_5^2(1-X_7),

where components of the predictor X∈ℝ^9 are distributed as

    X_1 ∼ U(0,1), X_2 ∼ U(-1/2,1/2), X_3 ∼ U(1,2)
       X_4 ∼ N(0,1), X_5 ∼ N(0,1), X_6 ∼ N(5,5)
       X_7 ∼Ber(0.4), X_8 ∼Ber(0.3), X_9 ∼Ber(0.6).

The true regression function is thus

    m(X)=vech^-1(-α_1/(α_1 + α_2), …, -α_1/(α_1 + α_2)).




Similar to simulations for the distributional data, we investigated sample sizes n = 100, 200, 500, 1000, with Q=500 Monte Carlo runs and set r = 2 as the dimension of low-dimensional representations of random objects. We compare the proposed model with global regression (GFR) and sufficient dimension reduction (SDR). Figure <ref> displays the MSPE for various sample sizes across all Monte Carlo runs, while Table <ref> summarizes their averages. We observe that MSPE decreases with increasing sample size, highlighting the convergence of the proposed methods. Both the boxplot and the table illustrate the superior performance of the proposed model over GFR and SDR, even with small sample sizes. Despite the simulated network data not residing on a 2-dimensional manifold,

the proposed model exhibits remarkable robustness as the sample size increases. Indeed, as the sample size grows, the proposed model proves to be a superior method for handling multivariate predictors and metric-space valued responses.





 §.§ Data Application for Human Mortality Data


We illustrate the proposed method using the age-at-death distributions of 162 countries in 2015, obtained from United Nations databases (<http://data.un.org/>) and UN World Population Prospects 2022 Databases (<https://population.un.org/wpp/Download>). For each country and age, the life table contains the number of deaths aggregated every five years, which can be treated as histograms of the number of deaths at age, with bin widths equal to 5 years. We use the  package <cit.> to create the density by local linear smoother on the histograms and finally standardize the function by trapezoid integration. Mortality distribution in each country is influenced by various factors, such as economics, healthcare systems, and social and environmental factors. To investigate these potential factors, we consider nine predictors, as used in <cit.>, including demographic, economic, and environmental factors in 2015; see Table <ref>.




The MSPE was calculated through leave-one-out cross-validation. From Table <ref>, the proposed method achieves the most accurate prediction results, with a 16% and 3% improvement compared to GFR and SDR, respectively. These results demonstrate that despite the data not residing on a two-dimensional manifold and with a small sample size, the proposed method is superior and robust. 











Additionally, our proposed method also allows for the investigation of the effects of predictors. Here, we further investigate the effect of GDP and health expenditure on mortality distributions while holding other predictors constant at their median values. Figure <ref> (left) shows that the fitted age-at-death densities shift towards the right as GDP increases, indicating an increase in life expectancy. Additionally, the probability of death before age 5 declines with increasing GDP, suggesting a lower infant mortality rate. Similarly, as the percentage of health expenditure in GDP increases, the age-at-death densities also shift rightward, indicating improved life expectancy. Conversely, as the percentage of agricultural GVA increases, the age-at-death densities shift leftward, indicating a decrease in life expectancy. This trend is consistent with the observation that as a country develops, GDP increases and the agricultural share of GVA commonly decreases.



























































































































































































































 Paper Title: Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokens.




New lower bound for the optimal congruent geodesic ball packing density of screw motion groups in 𝐇^2×𝐑 space. 
[Mathematics Subject Classification 2010: 52C17, 52C22, 53A35, 51M20. Key words and phrases: Thurston geometries;  geometry; geodesic ball packing; tiling; space group;]
    Arnasli Yahya and Jenő Szirmai 

Department of Algebra and Geometry, Institute of Mathematics,

Budapest University of Technology and Economics, 

Műegyetem rkp. 3., H-1111 Budapest, Hungary 

arnasli@math.bme.hu, szirmai@math.bme.hu

    August 5, 2024
=========================================================================================================================================================================================================================================================================================




§ ABSTRACT 
  
  
  
  
  


While large language models (LLMs) are extensively used, there are raising concerns regarding privacy, security, and copyright due to their opaque training data, which brings the problem of detecting pre-training data on the table.
Current solutions to this problem leverage techniques explored in machine learning privacy such as Membership Inference Attacks (MIAs), which heavily depend on LLMs' capability of verbatim memorization. 
However, this reliance presents challenges, especially given 

the vast amount of training data and the restricted number of effective training epochs. 










In this paper, we propose an adaptive pre-training data detection method which alleviates this reliance and effectively amplify the identification. 
Our method adaptively locates surprising tokens of the input. 
A token is surprising to a LLM if the prediction on the token is "certain but wrong", which refers to low Shannon entropy of the probability distribution and low probability of the ground truth token at the same time.
By using the prediction probability of surprising tokens to measure surprising, the detection method is achieved based on the simple hypothesis that seeing seen data is less surprising for the model compared with seeing unseen data. 


The method can be applied without 
any access to the the pre-training data corpus or additional training like reference models.
Our approach exhibits a consistent enhancement compared to existing methods in diverse experiments conducted on various benchmarks and models, achieving a maximum improvement of 29.5%. We also introduce a new benchmark Dolma-Book developed upon a novel framework, which employs book data collected both before and after model training to provide further evaluation. 































§ INTRODUCTION


Large language models(LLMs) are experiencing immense success, with their remarkable performance and widespread popularity driving their research and deployment across a multitude of applications <cit.>. Alongside this, concerns regarding the potential negative impacts on privacy <cit.>, security <cit.> and copyright <cit.> of LLMs are also gathering widespread attention. 

Lacking of transparency regarding training data <cit.>, as one significant challenge, leads to difficulty in model evaluation <cit.>, and a series of problems including potential leaks of personal privacy data <cit.> and infringements upon copyrighted data <cit.>.


In tackling this challenge, a group of works is delving into the problem of pre-training data detection, which aims to determine whether the target LLM was pre-trained on a given input without prior knowledge of the pre-training data and only with black-box access to it. This is in line with Membership Inference Attacks (MIAs) <cit.>, a problem extensively studied in machine learning privacy and gaining attention under LLMs domain.
MIAs exploit the model's tendency to overfit the training data, resulting in low loss values for that data. Typically, prevalent strategies for pre-training data detection based on MIAs heavily rely on the verbatim memorization <cit.> capabilities of LLMs. However, this reliance poses significant challenges <cit.>, especially given the current scenario where pre-trained LLMs generally have dramatically large scale training sets and are only trained on the data for around one epoch.



In this work, we aim to exploring the issue from a different perspective: if the verbatim memorization of arbitrary pre-training data is unreliable for the model, is it possible for people to automatically locate representative places

of the input, which effectively amplify the distinctions between the model's seen data (i.e., pre-training data) and unseen data (i.e., non-training data)? 


Distinguished from prior works, our method wants to fill in this gap and to conduct detection through method that alleviates the reliance on the plain input memory as much as possible. 







We propose an adaptive pre-training data detection method , which locate surprising tokens of a given input and achieve detection based on the hypothesis that 
seeing seen data is less surprising for model when compared with seeing unseen data.


For the input, we first adaptively locate surprising tokens. The surprising tokens are inspired by Shannon Entropy<cit.> in information theory, empirical observations(Figure <ref>), and the nature of surprise. 
Surprising happens when people are sure about the answer of a question, but the revealed true answer is different from the answer in mind. 
We transfer this definition of surprise to LLMs, and define surprising token as tokens in the input where the LLM is sure about what the prediction should be, but the probability that the LLM assigned to the ground truth token is low.




We use Shannon entropy to measure how sure the LLMs think of what the next token is, as it measures the uncertainty of a distribution. 
Lower entropy means the probability distribution is more concentrated, and there’s less uncertainty about the outcome. 
Thus, surprising tokens refers to low Shannon entropy of the probability distribution and low probability of the ground truth token at the same time.
Further, to measure how surprising the LLM is, we use the average prediction probability of surprising tokens: 

the lower the average probability, the more surprising the LLM. Based on the hypothesis, LLM will have higher average probability on seen data compared with on unseen data. 
To summarize the process, given an input, our method first adaptively locates the surprising token of input, then calculate the average probability that the LLM assigned to the surprising tokens, finally detect whether the input is seen or not based on the average probability. Our approach can be applied without any prior knowledge about the pre-training data corpus or additional training like reference models.

We construct a new benchmark Dolma-Book, to enlarge the limited number of datasets for pre-training data detection problem. Dolma-Book is developed upon a novel LLM framework – Open Language Model (OLMo) <cit.>. We employs book data from Project Gutenberg collected before and after model training(i.e., based on a cutoff date) to provide further evaluation of the detection methods. Since the considerable length of the raw book data, we collect and category three datasets from the head, middle and tail segmentations of the book to serve for a better evaluation of detection performance across various text positions within the long-context book data. Procedure of data collection and categorization is automatic and convenient to extend for future studies. 
We empirically evaluate the performance of our approach across various language models and data benchmarks. 


Our approach consistently outperforms existing methods across three benchmarks and different families of LLMs. Further analysis demonstrate our method maintains advantages on deduplicated model and inputs of different length variations. 
Code and benchmark are available here: make public after acceptance.




    
    
    
    


We summarize our contributions below:

    
  * We propose to identify surprising tokens ( <ref>) instead of blindly believing in the model's memory of each individual token, due to the challenges and unreliability of verbatim memorization ( <ref>).
    
  * We construct an improved pre-training data detection approach ( <ref>) through adaptively surprising tokens identification and designed score computation.
    
    
    
  * We construct a book data benchmark for further pre-training detection problem ( <ref>) and provide extensive empirical demonstration of black-box detections for various LLMs and benchmarks ( <ref>). 







§ RELATED WORK




We discuss related works on pre-training data detection for large language models. The problem of detecting pre-training data aims to determine whether a language model was trained on the given text input when black-box access to the LLM without knowing the pre-training data. Its necessity has received increasing attention from various aspects. 

In the field of traditional machine learning privacy, there has been well explored research on membership inference <cit.>. Since the current problem is in line with it, some prior works use MIA technologies to solve detection problem in LLM domain. A simple and frequently employed method is the LOSS attack <cit.>, which utilizes the model's computed loss and classifies the inputs as part of the training set if their loss values fall below a specific threshold. 
Some work <cit.> claims this kind of attacks have high false positive rate (i.e., non-member samples are often erroneously predicted as members), 
and difficulty calibration <cit.> is proposed to overcome this. 

Since then, other reference-based methods <cit.> are proposed to consider the intrinsic complexity of the target data by calibrating the loss of target model with another reference model. These methods have to train other models on the same dataset or data from same distribution. Instead of training reference models, <cit.> craft textual neighboring samples through data augmentation and leverages an estimation of the loss curvature to detect membership. More recently, <cit.> selects tokens which have the lowest likelihood to compute the score for detection.

Applying MIAs in detecting pre-training data of LLMs has its own challenges.
Since traditional MIAs exploit the model's overfitting on the training data (i.e., model tends to obtain low loss values on training data), basically, the commonly-used methods 

heavily rely on LLMs' capabilities of verbatim memorization <cit.>. 
However, existing work <cit.> claims challenges in solving data detection problem simply based on memorization of input, 
as the standard practice of training LLMs is training model for one epoch with massive training corpus, which makes overfitting impractical.

Not relying on memorizing each word input but uses some narrow concepts to screen, <cit.> proposes name cloze for data archaeology to infer books that are known to ChatGPT/GPT-4. 
Although insightful design, the scenario is very limited. Usually, inferring membership based on named-entity is not robust due to the unsure information rarity. 

In parallel to pre-training data detection, there are some related but not completely overlapping issues. <cit.> study the data contamination problem, where the pre-training dataset contains various evaluation benchmarks. <cit.> work on the data extraction problem, which aim to recover individual training examples through prompting attacks, especially the personally identifiable information(PII) <cit.>. 
These methods have limited use in pre-training data detection problem, as they require the specific dataset structure or data types.











§ PRELIMINARY




We first define the terminology we use, formalize the problem, and state our assumptions about the knowledge and capabilities. 
We follow the standard definition of the existing pre-training data detection in LLMs:





  
Autoregressive language models. We focus on the autoregressive language models ℳ, which are trained to predict the next text token based on the input sequence of previous tokens (i.e., different from bidirectional masked language models such as BERT <cit.>). We use p(· | x_1, x_2, ..., x_i-1) to represent the probability distribution of an model ℳ on generating next token when given an input sequence x_1, x_2, ..., x_i-1. This probability distribution is among a token vocabulary, denoted as 𝒱, and for each v_j ∈𝒱, denote p(v_j| x_1, .., x_i-1) as the probability of next token to be v_j. 






  
Clarify names and definitions. We want to clarify the following names used in paper for better understanding. Given an input sequence x_1, x_2, …, x_N, each of x_i is called token (i.e., ground-truth token); index (i.e, 1,2,..N) is used to state the location of x_i. 
We call the possible generated tokens (i.e., ∀ v_j ∈𝒱) at a specific index as token candidates.














  
Problem definition. Given a data point 𝐱 = x_1, x_2,..,x_N and a pre-trained language model ℳ, the goal of pre-training data detection problem is to infer whether 𝐱 belongs to the training dataset 𝒟 for this model ℳ. 
The detection process ℱ is achieved by computing a score f(𝐱, ℳ) and decided through a case-dependent threshold λ. Formally, 𝐱: ℱ(f(𝐱, ℳ); λ) ↦{0, 1}. 
Following previous works, people care about the threshold-independent metric, area under the ROC curve, for the aggregate measure of performance. 




  
Knowledge and capabilities. We assume the detector can access to the output statistics of the model ℳ, i.e., probabilities, but no access to model parameters, gradients as well as the pre-training data 𝒟. Also, we assume no access to the pre-training data distribution D for any training of calibration model. 









§ ADAPTIVE PRE-TRAINING DATA DETECTION THROUGH SURPRISING TOKENS

In this section, we provide the motivation and detailed description of our adaptive pre-training data detection method , that using token probabilities at representative index for detection. 

Since current state-of-art pre-trained LLMs are trained with billions and trillions of data corpus but pre-train for very limited (i.e., only near-one) epoch, believing in the model’s memory of each individual token is not practical. Even the size of today's large models is far from enough to memorize verbatim. Because of the black-box, we don’t know exactly what the model remembers, however we can try to get clues from some details displayed by the model itself.



 §.§ Surprising Tokens Identification


Language models determine what the current token should be based on the preceding text, and entropy reveals the model's certainty in deciding the current token should be which one among the entire vocabulary. 

Shannon Entropy<cit.> is widely used as a measurement for the uncertainty within a distribution. Specifically, it is defined as H(P_X) = - Σ_x ∈𝒳 P_X(x) log P_X(x), where P_X represents a discrete probability distribution across the outcome space 𝒳, and its value falls between 0 and log|𝒳| (i.e., occurs when all outcomes are equally like). Generally, lower entropy means the probability distribution is more concentrated, and there's less uncertainty about the outcome; whereas higher entropy indicates a more spread-out distribution with greater uncertainty. Thus in context of language models, Shannon entropy can be used to evaluate the predictability or uncertainty of the text sequences. That is, a low entropy distribution in language models implies that there are not so much possible choices, leading to model's great certain for the current token. 

Based on these, we define surprising tokens as the tokens that 1) model is confident with the prediction of the current token location, which can be reflected from the low entropy value at the index; 2) the model has low probability for predicting this token to be the ground truth one. 
When these two conditions achieve at the same time, the model is confident that the ground truth token is not the current token. 
As a result, upon revealing the ground truth, the model will be surprising. Note that if the model itself is not very certain about what this position should be (i.e., it has a high entropy distribution), the low probability prediction of the ground truth token doesn't signify much, regardless of whether the data has been seen before or not.




 §.§ Adaptive Pre-training Data Detection



We introduce an adaptive pre-training data detection method  that using token probabilities at adaptively selected representative index (aka. surprising tokens) of a given input for detection. is based on the hypothesis that the seeing seen data will not be that surprising for the model when compared with seeing unseen data. 



Our inspiration is derived from the observations about the surprising tokens: we found that when select representative tokens based on the requirements described in <ref>, the average probability of ground truth tokens at the selected indexes of a seen data is higher than it of a unseen data. We empirically demonstrate this phenomenon in Figure <ref> using an example of GPT-Neo-2.7B model and the DM Math dataset. 

Figure <ref> shows the token entropy and ground truth token log probability for all tokens within given inputs, where we are hard to tell the difference between unseen inputs and seen inputs. We notice that in the left bottom part of the figure, which is the range for the surprising tokens, tokens from seen inputs and tokens from unseen inputs are more separable. Figure <ref> is drawn by limiting the token entropy and ground truth probability(i.e., the left bottom part of Figure <ref>). We can clearly see that the tokens from seen inputs have a larger prediction probability compared with tokens from unseen inputs. Therefore, we focus on surprising tokens and design .




is formulated as following. 
Denote a sequence of tokens, which is the input, as 𝐱=x_1, x_2, x_3, …, x_N. 
Denote a language model as ℳ, which takes some of the token sequence x_1, x_2, …, x_i-1 as input and predict the probability of the next token among the set of vocabulary 𝒱 at location i. 
We denote x_i as the ground truth token at position i. 
The probability distribution of each token position as a 1 ×|𝒱| vector P_i, where i indicates the index in the input sequence 𝐱. 
Each entry of the probability vector P_i, which represents the model's predicted probability on a specific token candidate, is calculated as P_i, v_j = p(v_j| x_<i; ℳ), ∀ v_j∈𝒱. 

Denote the ground truth token's log probability as L_i, with the probability vector, it can be calculate as L_i = log P_i, x_i.
We first obtain the entropy E_i at each index i, then select a subset of indexes where entropy is smaller than a certain value ε_e:


    E_i    = -∑_v_j∈𝒱P_i,v_jlogP_i,v_j, 
    
    S_e    = {i |  E_i < ε_e, ∀ i ∈{1, …, N}}








We further restrict set S_e by choosing indexes with low prediction probability for the ground truth token x_i, i.e., P_i, x_i is low. We define low probability in terms of k-th-percentile of log probabilities L_1, L_2, …, L_N. The k-th percentile is the value k/100 of the way from the minimum to the maximum. Denote this value as L^k, 


we have

    S_p = {i |  L_i < L^k, ∀ i ∈{1, …, N}}





Finally, for each given input sequence 𝐱 we adaptively locate the set of surpring tokens as S_e∩ S_p, and calculate the average log probability of the tokens in the joint of these two:

    (𝐱, ℳ) = 1/| S_e∩ S_p|∑_i∈ S_e∩ S_p L_i





To detect whether a piece of text 𝐱 is part of the pre-training data, we simply thresholding the value (𝐱, ℳ):

    ℱ((𝐱, ℳ); λ) = 1[ (𝐱, ℳ) ≥λ]




where λ is a case-dependent threshold. We care about the threshold-independent metric in <ref> same as previous works. 
The algorithm is summarized in Algorithm <ref> in Appendix <ref>.

















































§ EXPERIMENTS



We evaluate the performance of the proposed as well as prior detection baseline methods against large autoregressive models over three benchmarks targeting pre-training data detection. 
We start by describing the datasets, models and baselines used in our experiments.










 §.§ Setups






  
Datasets. We perform experiments on three benchmarks, including two existing benchmarks WikiMIA <cit.> and MIMIR <cit.>, as well as a new benchmark Dolma-Book that we developed upon a novel framework. 
WikiMIA <cit.> is a dataset that collect event pages from Wikipedia and distinguish training and non-training data based on a specific cutoff date. Moreover, WikiMIA[WikiMIA from HuggingFace. https://huggingface.co/datasets/swj0419/WikiMIA] provides categorized data based on different sentence lengths (i.e., 32, 64, 128, 256 words in one input separately), aiming to offer a detailed evaluation of varying input lengths.
MIMIR <cit.> is benchmark that built based on seven diverse data sources in the Pile dataset <cit.>. Training and non-training data in this benchmark are directly sampled from the train and test sets of the Pile respectively. Moreover, MIMIR[MIMIR from HuggingFace. https://huggingface.co/datasets/iamgroot42/mimir] provides data in different source domains with different n-gram overlap between the training and non-training data, and we keep all methods to use the ≤ 20% 7-gram overlap data groups in our evaluation since baseline methods show better performance. 





  
Dolma-Book Benchmark Construction. We construct a new benchmark Dolma-Book,  which is developed upon a novel LLM framework – Open Language Model (OLMo) <cit.>, to provide further evaluation of our method. 
Since it releases full pre-training data (i.e., Dolma) for the OLMo models, we could easily sample training data from the released data. 
However, we still need to collect non-training data for the detection problem. 
Therefore, to construct our Dolma-Book benchmark: Step 1. Collect non-training book data from Project Gutenberg, which is a repository of over 70 thousand public domain books that are not protected under U.S. copyright law. Since Dolma collected this archive in April 2023, we set January 1, 2024 as the cutoff date and extract all books after this date directly from the website https://www.gutenberg.org/gutenberg.org[Use GutenbergPy <cit.> to extract books.]. There are 978 books in total. 
Step 2. For the pre-training data, we randomly sample from the groups of book data within Dolma[Dolma from HuggingFace. https://huggingface.co/datasets/allenai/dolma] for the same number of books with the non-training ones. 
Step 3. The raw books are too long to be inputs for a benchmark, so we need to select parts of them. In order to further compare the detection performance of different text positions within the long-context raw book, we collect and category three datasets for both the training and non-training, called Dolma-Book-head, Dolma-Book-middle and Dolma-Book-tail. We first split each book into segmentations (i.e., after discarding the headers of the book) according to a fixed number of words (i.e., do 1024 and 512). Then we take the first segmentation, middle segmentation(i.e., length //2) and the last two segmentations into the datasets of head, middle and tail respectively. The procedure is conducted for both training and non-training data creation for further evaluations. 
The data construction pipeline including new data collection and categorization is automatic and is convenient to extend for future time.



  
Models. We conduct various experiments on eight large language models among four families. Given that Wikipedia often serves as part of the training corpus for multiple models, we evaluate the performance on WikiMIA against the suite of LLaMA <cit.> (7B, 13B), Pythia <cit.> (2.8B, 6.9B, 12B) and GPT-Neo <cit.> (2.7B) models. For Dolma-Book, we use the two different-size models (1B, 7B) provided by OLMo framework <cit.>. Then we use the suite of Pythia and GPT-Neo models for the MIMIR, since these models are trained on the Pile. We also evaluate on the deduped-Pythia model(Table. <ref>) to compare the performance. 



  
Evaluation Metrics. Consistent with previous works, we primarily report the AUC-ROC score (the area under ROC curve) for performance evaluation, and show the TPR@low% FPR scores. 



  
Baselines. We consider six baselines including the standard reference-free and reference-based methods to compare the performance. 
The LOSS Attack method (PPL) <cit.> uses the model's computed loss of an input as the perplexity to predict the membership. 
Neighbor method (Neighbor) <cit.> leverages an estimation of the loss curvature to detect membership. 
Min-K% Prob method (MinK) <cit.> selects k% tokens which have the lowest likelihood to compute the score for detection.
Reference-based method (Ref) proposed in <cit.> trains a small reference language model on the same data and uses the perplexity obtained from the reference model for calibration. 
In evaluation, we use the smaller version of each family as their reference model, i.e., LLaMA-160M for LLaMA models, Pythia-70M for Pythia models, GPT-Neo-125M for GPT-Neo-2.8B and OLMo-1B for OLMo-7B. 
Similar to reference-based method, Zlib method <cit.> uses zlib compression entropy as reference, and Lowercase <cit.> uses lowercased example perplexity as the reference. 












 §.§ Evaluation Results















Table. <ref>, <ref>, <ref> show the results of our method across three benchmarks on various models and the comparisons with the baseline methods. For all experiments in evaluation, we apply grid search for the best hyperparameters, i.e., ε_e is from 0.5 to 10 with step size 0.5, and k is from 10 to 100 with step size 10. Same with the baseline MinK, the hyperparameter can be determined by a held-out validation set. We show more analysis about our hyperparemeters in <ref>. For each of the baseline, we basically apply setups according to their papers and implementations. And for MinK, we report the results under the best parameter value for a fair comparison. 

The AUC-ROC results for WikiMIA benchmark is shown in Table. <ref>. In addition to different kinds of LLMs, we also evaluate the detection performance on various length of the input. Results of 32 and 128 words length are displayed here, and the results of the input length to be 64 and 256 are in Appendix. <ref>. 
As shown in the table, our method can achieve a huge enhancement with maximum of 29.5%, and consistently outperform baselines across different input lengths (average improvement of 11.4% for length-32 and average 6.4% for length-128).
In general, the performance increases as the input text length increases, except for LLaMA models which display a dramatic advantage in shorter inputs. Also, according to the results from the suite of Pythia models, the performance will get improved with larger size of models regardless of the input length. 

Table. <ref> shows the AUC-ROC results for our Dolma-Book benchmark. Given the considerable length of the raw book, we conduct separate comparisons of input data from the head, middle and tail segmentations(details in <ref>) of the book to show a better evaluation of detection performance across different text positions within the long-context book data. As shown in Table. <ref>, our method outperforms other baseline methods on both Dolma-Book-middle and Dolma-Book-tail with an improvement of 5.2% and 5.6% from the best baseline AUC scores respectively. 

For Dolma-Book-head, Our method performs the best except for Zlib. The head segmentation of a book could contains a certain degree of garbled characters (including formatting, list of names, table of contents, etc.), which is likely to make Zlib compression calibration effective.

Moreover, basically for all methods, the middle of book as input can obtain best detection performance, and tail gets the worst. 
Appendix. <ref> show more results about 512 word-length of inputs. 

Table. <ref> shows the results of using MIMIR benchmark. Among seven different datasets, our method basically outperforms other baselines or reach a comparable results (second best beyond Zlib for Github). For some datasets like DM Math, our method achieves an enhancement from the best previous method's (Lowercase) 92.8% to 94.2%, but we may also obtain modest improvement on some datasets such as HackerNews.



  
TPR@lower FPR. 
Following prior works, we additionally report the detection performance in terms of the true positive rates (TPR) under low false positive rates (FPR). Figure. <ref> shows the ROC curve of WiKIMIA data and LLaMA-13B model across various methods. Our method significantly improve existing approaches through varying threshold, e.g., we improve TPR from 0.275 to 0.431 at 5% FPR, and from 0.196 to 0.412 at 1% FPR. Table. <ref> - <ref> in Appendix. <ref> display the values of TPR@low% FPRs (i.e., 1%, 5% and 10%) for different benchmarks and LLMs. 
















































































 §.§ Ablation Studies




























  
Analysis about the hyperparameters. 




We have two parameters, one is the entropy threshold ε_e, one is the low probability threshold k-percentile. For giving more analysis, we show the heatmap of AUC scores in Figure <ref>. 
Figure <ref> shows the AUC score when using different entropy threshold and percentile pairs, with OLMo-7B model and Dolma-Book-middle dataset. We can find that the best AUC score happens when the ε_e = 2.5 and the k=40. We can see clearly that both entropy threshold and low ground truth probability are important: if only use low entropy threshold ε_e (the top row in figure), the AUC score could as low as 0.52 (with ε_e = 2.5 and k=100); if only use small k (the right most column), the AUC score is around 0.56 (with ε_e = 10.0 and k=10). 

Similar phenomena is shown from Figure <ref> in Appendix <ref> using Dolma-Book-tail dataset, which achieves best AUC when entropy threshold is low (ε_e=2.0) and k=30.







































  
Deduped model cannot decrease the advantage. 
To see the influence of duplicates in training data, we compare the performance of model trained on regular dataset and deduplicate dataset. We evaluate the performance of our method and baselines using Pythia-2.8B and deduped-Pythia-2.8B, see Table. <ref> in Appendix <ref>. For using the deduped model, our method also achieve the best results comparing with other baselines over various datasets (except for Github as second best after Zlib). Also, our method achieve similar performance on the original model and deduplicate model (i.e., non-deduped slightly higher in Wikipedia, ArXiv and DM Math). 








































  
Influence of data distribution shifts. 
Data distribution shifts could occur in current dataset for pre-training detection, as in many datasets, the unseen data consists of newly added data after the trained model was released. This brings a potential data distribution shifts. To see the influence of data distribution shifts, we evaluate the performance on temporal datasets of Wikipedia and ArXiv using the datasets provided by <cit.>. The temporal datasets contain data distribution shifts between training data and non-training data, as they are created with texts in different time periods. When comparing the results between with-distribution-shifts (i.e., temporal Wikipedia/ArXiv) and without-distribution-shifts (i.e., original Wikipedia/ArXiv), as shown in Table <ref> in Appendix <ref>, we see no consistent performance improvement on the temporal datasets compared with common ones, as Wikipedia-temporal improves a lot from 68.8% to 75.6% while ArXiv-temporal performs a negative impact from 79.3% to 73.4%. 



























§ CONCLUSION


We provide an adaptive pre-training data detection method, which alleviate the reliance of verbatim memorization and effectively amplify the identification. 
Our method adaptively identify surprising tokens of the input, and detect whether the input is seen or not based on the hypothesis that seen data is less surprising for the model compared with unseen data. We construct data benchmark Dolma-Book, and evaluate our method on our benchmark and other existing benchmarks.

Our work further suggests the impractical of simple application of traditional MIAs for pre-training data detection in LLMs, and we encourage the development of new strategies considering completly outside the verbatim memorization for achieving effective and robust pre-training data detection. We expect to see the positive social impact of our method and other work in line, especially in protecting private and copyright data. 














































 















 Paper Title: Neural Graph Matching for Video Retrieval in Large-Scale Video-driven E-commerce.















  JD.COM
  P.O. Box 1212

  
  China
  43017-6221




	JD.COM
	P.O. Box 1212
	
	
	China
	43017-6221




	JD.COM
	P.O. Box 1212
	
	
	China
	43017-6221





	JD.COM
	P.O. Box 1212
	
	
	China
	43017-6221






	National University of Singapore
	P.O. Box 1212
	
	
	China
	43017-6221




	JD.COM
	P.O. Box 1212
	
	
	China
	43017-6221














	§ ABSTRACT 
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
		With the rapid development of the short video industry, traditional e-commerce has encountered a new paradigm, video-driven e-commerce, which leverages attractive videos for product showcases  and provides both video and item services for users.
		Benefitting from the dynamic and visualized introduction of items, video-driven e-commerce has shown huge potential in stimulating consumer confidence and  promoting sales.
		In this paper, we focus on  the video retrieval  task, facing the following challenges: (1)  How to handle the heterogeneities  among users, items, and videos?  (2) How to mine the complementarity  between items and videos for  better user understanding?
		In this paper, we first leverage the dual graph to model the co-existing of user-video and user-item interactions in video-driven e-commerce and innovatively reduce user preference understanding to a graph matching problem. 
		To solve it, we further propose  a novel bi-level Graph Matching Network (GMN),  which mainly consists of node- and preference-level   graph matching.
		Given a user, node-level  graph matching aims to match videos and items, while preference-level graph matching aims to match multiple user preferences extracted from both videos and items.
		Then the proposed GMN can generate and improve user embedding by aggregating matched nodes or preferences from the  dual graph in a bi-level manner.
		Comprehensive  experiments show the superiority of the proposed GMN with significant improvements over   state-of-the-art approaches (e.g., AUC+1.9% and  CTR+7.15%). 
		We have developed it on a well-known video-driven  e-commerce platform, serving hundreds of millions of users every day.
		
	
	










<ccs2012>
 <concept>
  <concept_id>00000000.0000000.0000000</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>500</concept_significance>
 </concept>
 <concept>
  <concept_id>00000000.00000000.00000000</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>300</concept_significance>
 </concept>
 <concept>
  <concept_id>00000000.00000000.00000000</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>100</concept_significance>
 </concept>
 <concept>
  <concept_id>00000000.00000000.00000000</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>100</concept_significance>
 </concept>
</ccs2012>


[500]Do Not Use This Code Generate the Correct Terms for Your Paper
[300]Do Not Use This Code Generate the Correct Terms for Your Paper
Do Not Use This Code Generate the Correct Terms for Your Paper
[100]Do Not Use This Code Generate the Correct Terms for Your Paper











20 February 2007
[revised]12 March 2009
[accepted]5 June 2009




 Neural Graph Matching for Video Retrieval in Large-Scale Video-driven E-commerce
    Lei Su
    August 5, 2024
=================================================================================






 





§ INTRODUCTION








	
	
	
	
		
		
	
	







	In the era of the information explosion, the recommender system has become the most effective way to help users  discover what they are interested from enormous data.
	As two basic Internet applications, e-commerce and  content recommendation  both  provide  the recommender service but focus on either item recommendation or content (e.g., video and live) recommendation.  
	
	Recently, with the thriving of online applications, there is a surge of video-driven  e-commerce <cit.>,  which integrates  video    and e-commerce for better  services. Benefiting from   vivid and  attractive video shows, video-driven  e-commerce provides a new business paradigm which improves user stickiness and activeness, and has become a new driving force for e-commerce development.
	
	
	
	For example, TikTok and Kuaishou integrate e-commerce with videos, while
	Taobao and  JD.COM     leverage videos   to   improve the user experience in traditional e-commerce.   As shown in Figure <ref>(a),
	JD.COM first incentivizes authors to produce videos  for related items,  and then conducts video recommendations for users.
	Due to the large-scale video set,
	industrial recommendation systems usually adopt the classical two-stage architecture: retrieval and ranking <cit.>.
	Given a user, the retrieval stage  aims to find thousands of candidate videos from the whole video set, then  the  ranking stage aims to  precisely select   hundreds of videos from the retrieved candidate videos.
	
	



	
	
	
	
	
	
	
	
	
	
	
	





































	
	
	
	
		
	
	


	
	
	In this paper, we focus on the video retrieval stage with the following challenges:
	
	
(1) How to  handle the heterogeneities   among users, items, and videos?

The characteristics of items (e.g., price, sales, and shop) are markedly different from videos (e.g.,  resolution, duration, and author) and them aim to stimulate user consumption and attract users stay, respectively.


This disparity compels us to treat user-item interactions and user-video interactions with distinct considerations, given the unique characteristics each type of interaction manifests.

(2) How to mine the complementarity  between items and videos for better user understanding?

While at a cursory glance these interactions may seem discrete, they exhibit a strong complementary nature upon closer examination. As exemplified in Figure <ref>, when a user interacts with both an item i_1 symbolizing a keyboard and a corresponding video v_1, it reinforces the user's profound interest in keyboards. In a contrasting scenario, if a user interacts with a beer item i_4 but directs their video interactions exclusively towards devices, such as keyboards, it suggests a potential superficial interest in the beer item.









Based on the above analysis, when  performing video retrieval  in video-driven e-commerce, we need to address the following new requirements:

	
	
  * Heterogeneous Interaction Modeling.  
	
	Different from traditional e-commerce, the video-driven e-commerce consists of complex heterogeneous interactions among users, videos, and items.
	
	Intuitively, user interactions on videos and items will show similar patterns and preferences. 
	
	As shown in Figure <ref>(b), user u_1 who wants to buy a keyboard will watch related videos v1,v2 and   click item i_1.
	
	How to model the co-existing of heterogeneous interactions is the basis of user preference understanding and video retrieval.
	
	
  * Node-level Graph Matching. 
	
	
	
	As shown in Figure <ref>(b), video v_1 and item i_1 are both keyboard-related (high relevance) while video v_1 and item i_4 are belonging to different categories (low relevance).
	
	So, when jointly modeling heterogeneous interactions for user preference understanding, it is necessary to properly distinguish and match the relevance between videos and items.
	
	
  * Preference-level Graph Matching. 
	
	Intuitively, diverse user preferences extracted from both videos and items are similar to each other.
	
	As shown in Figure <ref>(b), items {i_1, i_2, i_3 }  indicate user preference on electronic products, which is similar to user preference on video {v_1, v_2}.
	
	But, item {i_4}  indicates user preference on food, which is different from his video preference.
	
	It is necessity to mine the  the differences and relevances for user preference understanding.
	
	Particularly, how to match multiple user preferences on both videos and items is still an open problem.
	
		























In this paper, we study a newly emerging  paradigm video-driven e-commerce,  and then  formulate it as the dual graph to model the co-existing of users, videos, and items as well as interactions among them. 
After that, we reduce the user preference understanding  in the dual graph to a graph matching problem. To solve it, we propose a novel Graph Matching Network (GMN), which mainly consists of node- and preference-level graph matchings, to improve user preference understanding. 
























The contributions of this work are summarized as follows:




•  To our knowledge,  this  is the first time that  leveraging the dual graph  to model  the newly emerging video-driven e-commerce,   and innovatively reduce  the user preference understanding  problem to a graph matching problem.

•  To solve this, we propose a novel  Graph Matching  Network,  called GMN, which mainly consists of node- and preference-level   graph matching. Specifically,   node-level   graph matching aims to mine the potential   relevance between  videos and items  in the dual graph.  Furtherly,    preference-level graph matching further captures the  relevance of user preferences extracted from both  videos and items.








•  Extensive experimental results on both offline and online data demonstrate the superiority of the proposed GMN. Currently, it has been deployed on one of the largest video-driven e-commerce, affecting hundreds of millions of users every day.






















§ RELATED WORK

Recommendation System (RS), which mainly focuses  on  user preference understanding and recommends diverse types of candidates (e.g., items and videos) to them, 
has accompanied  diverse paradigms, such as  item recommendation <cit.>, video  recommendation <cit.>, and business-driven recommendation  <cit.>. 

Real-world    RS usually consists of two stages: retrieval   <cit.>
and   ranking   <cit.>.
YoutubeDNN <cit.> retrieves videos for online video services, while MIND <cit.> retrieves items for e-commerce.




Unlike them, 

we study a newly emerging video-driven e-commerce, which models the co-existing of items and videos.

Cross-domain Recommendation (CDR) attempts to learn useful knowledge from the source domain   to help the target domain. 
Hu et al.  <cit.> propose CoNet  to perform 
dual transfer at the unit level.   <cit.> unify both source and target domains as
graph-structured data  and perform CDR via graph learning.


Significantly different from the above works, we 

we first model video-driven e-commerce 
as a dual graph and reduce it to a graph matching problem.






Graph Neural Networks  (GNNs) generalize deep learning to graph-structured data, where graph convolution <cit.> aims to aggregate  neighbors information  and update node embedding,
while graph pooling


<cit.>  aims to aggregate all node to update graph embedding.


Recently, many works <cit.> apply GNNs to      recommendation scenarios. 
SURGE <cit.> reduces   user behaviors into graphs   and then extracts users' core interests via cluster-aware   graph pooling. 
Furthermore, several works <cit.> generalize GNN to multiple interactions (or domains).
MGFN <cit.> proposes a multi-graph learning for cross domain video recommendation.


Different from the above works, we first propose a novel bi-level graph matching network  to align  user behaviors and  preferences on both videos  and items    for   retrieval.




























































































































































§ PRELIMINARIES






	Dual   Graph. 

The dual graph,  termed as 𝒢=(𝒢^U-V,𝒢^U-I), describes the co-existing of  user  behaviors on both videos and items in video-driven e-commerce. Here  𝒩=𝒩^U∪𝒩^V∪𝒩^I is the union set  of  User, Video, and Item,  respectively. Also,   ℰ=ℰ^U-V∪ℰ^U-I is the edge set, where	ℰ^U-V=⟨𝒩^U, 𝒩^V⟩ denotes User-Video interaction  and ℰ^U-I=⟨𝒩^U, 𝒩^I⟩ denotes User-Item interaction. 




Example. 
Figure <ref>(b) gives an illustrative example of the dual graph in video-driven e-commerce.
Given  a user u and the  dual graph  𝒢, we can obtain his neighbor sets, including 𝒩_u^U-V={ v|⟨ u, v ⟩∈ℰ^U-V} from 𝒢^U-V and
𝒩_u^U-I={ i|⟨ u, i ⟩∈ℰ^U-I} from 𝒢^U-I.
In the dual graph, user u_1's  neighbor sets  are  𝒩_u_1^U-I= {i_1,  i_2, i_3, i_4 }
and   𝒩_u_1^U-V= {v_1, v_2  }, respectively.
Furthermore, we also  extract user-centric  sub-graphs  for user u  from the dual graph, termed as    𝒢_u=(𝒢_u^U-V,𝒢_u^U-I).  
In detail, 𝒢_u^U-V  contains multi-hops neighbors of user u and himself (i.e., 𝒱_u^U-V = 𝒩_u^U-V∪ u), as well as induced edges ℰ_u^U-V. 
Note that,  dual graphs can be widely used in many other related fields, such as  the co-existing of user-music and user-movie, or even  the co-existing of user-item(old/warm domain) and user-item(new/cold domain).




Finally, we formulate the video retrieval problem in video-driven e-commerce as follows:


    max_v ∈𝒩^V Pr(v| u; 𝒢),

which aims to retrieve candidate videos v from the whole video set 𝒩^V for user u based on the dual graph 𝒢.




















§ THE PROPOSED MODEL



In this section, we present a novel  Graph Matching  Network, which aims the  to match the users’ behaviors as well as preferences via both node- and preference-level graph matching.



The overall framework of GMN is 
shown in Figure <ref>.
































 §.§ Embedding Initialization




In this section, we first initialize the node embeddings of users, videos, and items via   feature embeddings, and then update them via GNNs.






Taking   embedding initialization   of video v as an example,  we  have:



    𝐱_v= ||_f𝐱_v^f,


where || denotes the vector concatenation, 𝐱_v and 𝐱_v^f denote the initial embedding of video v and the  f-th feature embedding of video v, respectively. 












































After that, we further utilize two interaction-specific    GNNs (i.e., GNN^U-V and GNN^U-I) to update corresponding embeddings.


Given  video v and 𝒢^U-V,  GNN^U-V is able to update the video 𝐱_v^U-V embedding via aggregating its neighbors 𝒩_v^U-V, as follows:


    𝐱_v^U-V = GNN^U-V(𝐱_u| u ∈𝒩_v^U-V).



Note that,   we focus on bi-level graph matching and 


simply instantiate  GNN^U-V  as the  MeanPooling <cit.>. 
Similarly,  the embedding of item i via  GNN^U-I is 𝐱_i^U-I.
For user u, we can also get his embeddings on the  dual graph    𝐱_u^U-V and  𝐱_u^U-I.










 































































































































































































































Until now, 
a method way for
video retrieval   is  
to fuse user preferences on   videos and items i.e., 𝐱_u^U-V|| 𝐱_u^U-I, and then retrieve top videos based on the similarity of their embeddings, 

suffering from  the following weaknesses:

(1) In video-driven e-commerce,  video production  actually revolves around items   and  aims to promote the item purchase. Although videos and items are different types of nodes, there exists potential relevance between them.
(2) It only models each user as a single node embedding while fails to explicitly capture the  graph structure  surrounding users, which  also provides  valuable information for understanding user preferences. So, it is necessary to  encode the user-centric sub-graph as a whole for video retrieval.




 §.§ Node-level Graph Matching


In this section, we propose node-level graph matching to mine the  relevance between videos and items in the dual graph,  and then enhance their connections via  aggregating  cross-graph neighbors based on    the  matching scores.




To handle large-scale video-driven e-commerce, we 
first extract user-centric  sub-graphs (i.e.,  𝒢_u^U-V and 𝒢_u^U-I) for user u, 
and then actually perform the node-level graph matching GM_NL on them, shown as follows:


    𝐑^u_ = GM_NL(𝒢_u^U-V,𝒢_u^U-I).

Here   𝐑^u  ∈ℝ^|𝒱_u^U-V| × |𝒱_u^U-I| 
denotes the  personalized relevance   matrix of user u in   node-level.


Given a   user u, 

the  personalized   relevance   score r^u_v,i  between   video v ∈𝒢_u^U-V and   item i ∈𝒢_u^U-I is shown as follows:


    r^u_v,i
    =(𝐱_v^U-V)^T ·𝐌·𝐱_i^U-I.

Here (·)^T denotes the transposition operator. 

𝐌 is  a learnable metric matrix, harmonizing the embeddings of heterogeneous nodes (video v.s. item).


Obviously, the  personalized relevance  score  r^u_v,i is dynamically changed with regard to users, videos, and items.






After obtaining the personalized relevance  scores {r^u_v,i}, we normalize them via the softmax function to get the personalized matching scores s^u_v← i and s^u_v→ i, 
shown as follows:


    s^u_v← i=  softmax_i(r^u_v,i)= exp(r^u_v,i)/∑_i'∈𝒱_u^U-Iexp(r^u_v,i'),



    s^u_v→ i=  softmax_v(r^u_v,i)= exp(r^u_v,i)/∑_v'∈𝒱_u^U-Vexp(r^u_v',i).


Note that  the above matching scores are asymmetric since they are normalized on different node sets (i.e., 𝒱_u^U-I and 𝒱_u^U-V).
Given a user u,  the personalized matching score s^u_v← i
measures the    matching level from item i to video v.  
Intuitively, aggregating relevant items embeddings into video embeddings is able to  the alleviate the  heterogeneity and sparsity problem,  improving user preference understanding and  the effectiveness of video retrieval.








Taking personalized matching scores (e.g., {s^u_v← i})
as coefficients, we can properly summarize all information ∑_i ∈𝒱_u^U-I r^u_v← i·𝐱_i^U-I which needs to propagate from relevant items i ∈𝒢_u^U-I to video v,  and then  utilize it to enhance  the video embedding.




The final video embedding 𝐡_v^U-V of video v is shown as  follows:


    𝐡_v^U-V  =  𝐱_v^U-V||   ∑_i ∈𝒱_u^U-I r^u_v← i·𝐱_i^U-I  .


Also, we   conduct information propagation from    videos to items, and  learn  the final   embedding 𝐡_i^U-I  of item i.





Although node-level graph matching is able to learn the potential connections between videos and items and improves their embeddings via bi-directional information propagation, it only focuses on node-level modeling and largely ignores the user preferences inferred from user interactions.













 §.§ Preference-level Graph Matching






In this section, we introduce a novel method for preference-level graph matching. This method mines the relevance of user preferences for videos and items. Subsequently, this relevance is utilized in cross-graph propagation to refine preference embeddings.







  §.§.§ Extracting User Preferences via Dual Graph Pooling


Specifically, we propose a novel dual graph pooling method (i.e., GPN^U-V and GPN^U-I) to cluster user-centric sub-graphs (i.e.,  𝒢_u^U-V and 𝒢_u^U-I) as preference graphs, where each node indicates  a cluster of similar videos or items and  corresponding user preferences.





Given a user u and  𝒢_u^U-V, a graph pooling network   GPN^U-V is able to cluster all video embeddings  {𝐡_v^U-V   | v ∈𝒱_u^U-V} into k_1 preferences, and constructs the condensed preference graph 𝒵_u^U-V:




    𝒵_u^U-V = GPN^U-V(𝒢_u^U-V),





    {𝐳_u^1^U-V,  ⋯, 𝐳_u^k_1^U-V}= GPN^U-V(𝐡_v^U-V  ),

where 𝐳_u^k_1^U-V is the embedding of  the k_1-th user preference.



If we set k_1 > 1, then   GPN^U-V actually extracts   multiple  types of user preferences. 
Therefore, we simply instantiate GPN^U-Vas the MeanPooling <cit.>


Similarly, we use    GPN^U-I to extract   k_2   preference embeddings   {𝐳_u^1^U-I,   ⋯, 𝐳_u^k_2^U-I | u^k_2∈𝒵_u^U-I}    for user u  from 𝒢^U-I.







  §.§.§ Improving Preference Embedding via Preference-level Graph Matching















































Intuitively, user preferences on both videos and items also have potential relevance. 

So we further propose the preference-level graph matching GM_PL, which  learns the   preference-level   relevance   matrix  𝐏^u ∈ℝ^k_1 ×  k_2 between preference graphs (i.e.,  𝒵_u^U-V and 𝒵_u^U-I), shown as follows:


    𝐏^u_ = GM_PL(𝒵_u^U-V,𝒵_u^U-I),

where each element p^u_k_1,k_2 indicates the relevance between the k_1-th video preference  and  the  k_2-th item preference of user u,  as follows:


    p^u_k_1,k_2
    =(𝐳_u^k_1^U-V )^T ·𝐳_u^k_2^U-I.



Then, we also normalize  p^u_k_1,k_2 on 𝒵_u^U-V and 𝒵_u^U-I  via softmax, and get corresponding matching scores s^u_k_1← k_2 and s^u_k_1→ k_2. Then, the embeddings of preference nodes can be enhanced via bi-directional propagation, as follows:

    𝐞_u^k_1^U-V  =  𝐳_u^k_1^U-V||   ∑_u^k_2∈𝒵_u^U-I s^u_k_1← k_2·𝐳_u^k_2^U-I  ,



    𝐞_u^k_2^U-I  =  𝐳_u^k_2^U-I||   ∑_u^k_1∈𝒵_u^U-V s^u_k_1→ k_2·𝐳_u^k_1^U-V  .


Interactively performing graph pooling and graph matching on preference graphs for L times, we can  progressively compress the original graph into  a single  node  (i.e., k_1=k_2=1),  termed as 𝐞_u^U-I and 𝐞_u^U-I,  representing the overall user preferences.






 §.§ Prediction and Optimization



To comprehensively understand user preferences, we   fuse user preferences on both videos and items, and get 
the final user embedding 𝐳_u of user u, as follows:





    𝐳_u= MLP(𝐞_u^U-V || 𝐞_u^U-I|| 𝐱_u^U-V|| 𝐱_u^U-I).


Since the feature of   video  si  much simpler and more stable than   users, we only perform GMN for user modeling,  and the final embedding  𝐳_v of  video  v is actually 𝐱_v^U-V. Similar to previous works<cit.>,  we do not use a more sophisticated fusion network (e.g., attention-based fusion), so the effectiveness of bi-level graph matching can be clearly verified. 



































































The estimated preference ŷ_u,v of  user u  towards the target video v is   the   inner product of their embeddings:


    ŷ_u, v= (𝐳_u)^T ·𝐳_v.




The overall loss function is as follows:


    ℒ = ∑_(u, v, v_neg) ∈𝒟-lnσ( ŷ_u,v - ŷ_u,v_neg) + λ· ||Θ||^2_2,






where Θ denotes all trainable model parameters, and λ
controls the L2 regularization strength to prevent overfitting. The whole model can be optimized via back-propagation.








 §.§ Model Analysis

 Here we give the analysis of the proposed GMN as follows:
 
 •  Considering the   characteristics of video-driven e-commerce, 
 we first represent it as the dual graph, and then reduce user preference 
 understanding to a bi-level graph matching problem. 
 So, the proposed GMN focused on matching relevant nodes (e.g., videos and items) and preferences properly in a bi-level manner, rather than designing better GNN architectures to learn nodes and preferences embeddings.  Similar to previous works<cit.>, we adopt basic GNNs as backbones, so the effectiveness of  bi-level graph matching  could be clearly verified.
 
 


•  We  analyze  the time complexity of GMN in both the node- and preference-level graph matching. In the  node-level graph matching,  if we directly  match all nodes between   𝒢^U-V and 𝒢^U-I, the computational cost is  𝒪(|𝒩^V| · |𝒩^I|), which is unacceptable for real-world scenarios. Fortunately, we actually match  user-centric  sub-graphs (i.e.,  𝒢_u^U-V and 𝒢_u^U-I) , which significantly reduce the time complexity to    𝒪(|𝒢_u^U-V| · |𝒢_u^U-I|). Similar to the node-level, the    time complexity of preference-level graph matching is     𝒪(|𝒵_u^U-V| · |𝒵_u^U-I|).


•  Then, we further  analyze the space complexity of our GMN, which mainly depends on the learnable parameters in embedding matrixes, rather than deep neural networks. Assuming we have total N nodes and F features in the dual graphs, the space complexity of GMN with feature embedding is  𝒪(F · d), which is significantly less than ID embedding 𝒪(N · d). In summary, the proposed GMN is able to efﬁciently handle  large-scale e-commerce. 

 





























§ OFFLINE EVALUATIONS



















 §.§ Datasets.
 









Although video-driven e-commerce has grown rapidly on many platforms, related data is protected by privacy   and security policies.
Therefore, we sample   user logs  from a real-world video-driven e-commerce platform, and then extract two large-scale dual graph datasets from different time periods (7-days, and 30-days) for verification.








We   also use a public dataset MovieLens-1M released by GIFT<cit.> to mimic  video-driven e-commerce, which takes user-movie (warm) as user-item interaction and user-movie (cold) as user-video interaction, respectively.  





The details of the datasets  are shown in Table <ref>. 






















































 §.§ Baselines.

To validate the effectiveness of  GMN, 
we  first select single interaction  based methods (i.e., YoutubeDNN<cit.>, MIND<cit.>, and SURGE<cit.>) as well as  multiple interactions based methods (i.e., GIFT<cit.>, MGFN<cit.>, and CCDR<cit.>). 

Since MIND and SURGE cannot be directly applied to the dual graph, we also provide corresponding dual versions (i.e., MIND+ and SURGE+) for better performance. The baselines are shown below:










 



 
  * YoutubeDNN<cit.>: 

It is a classical model for large-scale   video retrieval, utilizing two-tower architecture to learn the embeddings of users and videos based on their features.




 
  * MIND/MIND+<cit.>: 

 MIND is a classical retrieval   model, which   utilizes a dynamic routing mechanism to extract   users' diverse interests from historical behaviors.


 Since MIND cannot directly apply to the dual graph, we extend MIND as MIND+ to learn diverse user interests from both user-video and user-item interactions,  and  then concatenate them as the final user interests for   video retrieval.



 
  * SURGE/SURGE+<cit.>: 
 SURGE is a basic graph-based model that takes graph convolution and graph pooling to extract and fuse users' interest from user  interaction for recommendation. 

 We also extend SURGE as SURGE+ to handle the dual graph. 
 The experimental setting is the same as MIND/MIND+.



 
  * GIFT<cit.>: It is a graph-based   model that utilizes diverse types of relations r  to guide the feature propagation between cold- and warm-domain for video  recommendation. 

Here we take   user-video interaction    as cold-domain  and   user-item interaction as warm-domain for experiment.




  * MGFN<cit.>: It is a multi-graph based video recommendation model which 
 encapsulates interaction patterns across scenarios via  multi-graph fusion network. 


 
  * CCDR<cit.>: It is a contrastive cross-domain model for retrieval, which  simultaneously utilizes both intra- and inter-domain contrastive learning. Here we take user-video and user-item as two domains.



 
  * GMN: It is our complete model, which performs both node- and preference-level graph matching on the dual graph for video retrieval.



   






















 §.§ Implementation.

For all models, we random initialize parameters with  Xavier initializer and select Adam as the optimizer.

For a fair comparison, we use

the following hyper-parameters for all models:


the dimension of  node embedding is 128, 

the learning rate is 0.0015,

the L2 regularizer is 0.01, and the dropout rate is 0.75. 
For MIND and SURGE, we use the multiple preferences version and set the maximum  number of preferences to 5.
For YoutubeDNN, we use 3 layer DNN (1024-512-256)  for prediction.

For offline evaluation,  we  select  AUC, Precision, Recall, and Loss as evaluation metrics.
We implement all models with TensorFlow 1.15 and run them on  Nvidia  A100 Cluster.
The  code of GMN  and the anoymous version of the  industrial dataset will be released after acceptance.

























































 §.§ Performance Comparison
 
 
The comparison results are shown in Table <ref>, where we have the following observations:


•  The proposed GMN consistently performs better than all baselines  with significant gaps.



Compared to the best performance of baselines, the improvements  of GMN on AUC metric is up to 1.2%-1.9% on  large-scale video-driven e-commerce, which indicates the effectiveness of both node- and preference-level  modeling in the proposed GMN for   video  retrieval. 



•  Multiple interactions   based methods (i.e., MIND+ and CCDR) outperform single graph methods (i.e., MIND and SURGE). It makes sense because both  user-video and user-item interactions are able to provide valuable information for understanding user preferences.






By capturing the correlation  between two types of interaction,

GIFT, MGFN, CCDR, and GMN 
always show their superiority  over 
concatenate based models (i.e., MIND+ and SURGE+).





• GNN-based models, including SURGE, GIFT, MGFN, CCDR, and GMN, perform better than traditional DNN-based models (i.e., MIND and YoutubeDNN), indicating the effectiveness of graph structure modeling in the recommendation.









































 §.§ Ablation Study


In this section, we conduct both model-level and graph-level ablation studies for further verification, shown in Table <ref>.


•  Model-level.













 
We first conduct a model-level ablation study to show how  the  delicate designs in GMN affect   performance. 

As shown in Table <ref>, we test two variants of GMN (GMN_\  NL and GMN_\ PL), which remove node- and preference-level graph matching, respectively. 





Obviously,  GMN performs better than  both GMN_\  PL and GMN_\ NL), which clearly verifies  the   superiority of node- and graph-level  matching.


Note that the degradation of GMN_\ NL is   more significant than GMN_\ PL,   implying that the node-level graph matching contributes more significantly.


The reason is that node-level graph matching is able to capture the potential relevance between videos and items and integrate them seamlessly, serving as the basis of preference extraction and preference-level graph matching.




•  Graph-level.







We also conduct graph-level ablation study    to verify the effectiveness of   dual graphs modeling, where 



GMN_\   U-V and GMN_\ U-I    remove the user-video graph and user-item graph, respectively.

Note that since graph-level ablation studies
are conducted only on
a single graph (user-video or user-item), so they naturally  build upon model-level ablation studies.

In Figure  <ref>, we find that  removing either the user-video graph or the  user-item graph fails to comprehensively capture user preference, leading to suboptimal performance.    



It suggests the necessity of dual graph modeling in video-driven e-commerce.

































































































 §.§ Case Study

We further present a case study to show the potential interpretability of GMN. 
Taking a young boy as an example who is interested in the Apple iPhone and the Casio watch, he has interacted with related items and videos.

Figure <ref> shows the heatmap on both node- and preference-level relevance  matrix, where darker color means higher relevance.
Since item i_9 and video v_8  are both related to iPhone,  the node-level relevance score  r^u_v_8, j_9 is high. A similar phenomenon can be observed at the  preference-level (e.g., p^u_1,1).






 §.§ Parameter Study









 In this section, we investigate the sensitivity of parameters and report the performance of GMN on the offline 7-days dataset with various parameters in Figure <ref>.
 
•  Dimension of Metric Matrix. As shown in Figure <ref>(a),  we  test the effect of the dimension  of the metric matrix 𝐌. Obviously, with the growth of the  dimension, the performance of GMN continues to rise. The reason is that, a larger dimension enables the proposed GMN to properly measure the similarity between videos and items. 



•   Number of User Preferences.   Naturally, most   users have diverse  preferences, and one preference fails to comprehensively describe the characteristics of users. 
Figure <ref>(b) shows that  
  more  user preferences  (i.e., larger k_1) extracted from  preference-level modeling   improve the overall performance of GMN. 
And, GMN achieves the best performance when  the number of user preferences  is set to 4.
 
 




























§ ONLINE EVALUATIONS

 









 §.§ Online A/B Testing











We conduct A/B testing during 0907-0925  (19 days) for online evaluations, shown in Figure <ref>. 
Although MIND+ is not the strongest benchmark in offline evaluation, it is the previously deployed model,  affecting hundreds of millions of users every day. Therefore, we use MIND+ as  the   baseline in online A/B testing.


















 


Here, we select the widely used CTR[ CTR= Number  of  clicked  videos / Number  of  exposed  videos × 100(%)]  to evaluate   online performance.


The larger CTR means users are willing to click the exposed videos recommended  by our system and indicates better performance. 




As shown in Figure <ref>, the proposed GMN  consistently shows significant superiority over the previously  deployed MIND+ on all days.
 

More specifically,  the average  CTR of the proposed GMN is  3.74% while MIND+ achieves only 3.49%, resulting in a CTR improvement of up to 7.15%.


Following <cit.>, we also use  TP99[TP99 is a minimum time that 99% of queries have been served.] response time to 
evaluate the efficiency of our system, shown in Figure <ref>.











 









Full-day observation shows that the response time of the proposed  GMN system is only about 1 ms higher than the previously developed MIND+,
meeting response time requirements and ensuring user experience.




Currently, we have deployed GMN on a well-known video-driven e-commerce platform, affecting hundreds of millions of users every day.

































































 §.§ System Architecture and Online Serving.
 

Here we present how to deploy the proposed GMN system for real-world video-driven e-commerce, shown in Figure <ref>.
For offline training,  we collect real-world user clicks as positive samples and random sample several videos as negative samples.   Then, we sample user-centric sub-graphs and associate them with training samples.  as training data. 
The trained model will be divided into two parts and developed in different platforms: (1) The user part, which is developed to the model center, will generate user embedding based on real-time user behavior. (2) The video part, which is developed to the retrieval center, will retrieve candidate videos based on user embedding.








§ CONCLUSION


The thriving of e-commerce is accompanied by a newly emerging paradigm,  video-driven e-commerce,  which utilizes attractive videos to improve user experience and stimulate consumption.

In this paper, we propose the dual graph to represent video-driven e-commerce and innovatively reduce the user preference understanding problem to a graph matching problem. 

To solve it, we further propose a novel  Graph Matching Network (GMN),   which mainly consists of node- and preference-level graph matching.

By properly matching the heterogeneous interactions and preferences, the proposed GMN achieves significant improvements over the state-of-the-art models in both offline and online evaluation (e.g., AUC+1.9% and  CTR+7.15%).

Currently, the proposed GMN  has been developed in a well-known video-driven e-commerce platform, affecting hundreds of millions of users every day.


































ACM-Reference-Format





 Paper Title: Higher order quantum reservoir computing for non-intrusive reduced-order models.


MicroMIL: Graph-based Contextual Multiple Instance Learning for Patient Diagnosis Using Microscopy Images
    
JongWoo Kim1Equal contribution Bryan Wong1[1] YoungSin Ko2 MunYong Yi1Corresponding author 1KAIST and 2Seegene Medical Foundation  

1{gsds4885, bryan.wong, munyi}@kaist.ac.kr, 2noteasy@mf.seegene.com  
    
===============================================================================================================================================================================================================

§ ABSTRACT 
Forecasting dynamical systems is of importance to numerous real-world applications. When possible, dynamical systems forecasts are constructed based on first-principles-based models such as through the use of differential equations. When these equations are unknown, non-intrusive techniques must be utilized to build predictive models from data alone. Machine learning (ML) methods have recently been used for such tasks.  Moreover, ML methods provide the added advantage of significant reductions in time-to-solution for predictions in contrast with first-principle based models. However, many state-of-the-art ML-based methods for forecasting rely on neural networks, which may be expensive to train and necessitate requirements for large amounts of memory. In this work, we propose a quantum mechanics inspired ML modeling strategy for learning nonlinear dynamical systems that provides data-driven forecasts for complex dynamical systems with reduced training time and memory costs. This approach, denoted the quantum reservoir computing technique (QRC), is a hybrid quantum-classical framework employing an ensemble of interconnected small quantum systems via classical linear feedback connections. By mapping the dynamical state to a suitable quantum representation amenable to unitary operations, QRC is able to predict complex nonlinear dynamical systems in a stable and accurate manner. We demonstrate the efficacy of this framework through benchmark forecasts of the NOAA Optimal Interpolation Sea Surface Temperature dataset and compare the performance of QRC to other ML methods.









§ INTRODUCTION

Quantum computing has emerged as a promising frontier offering exponential speed-up potential for solving complex problems compared to classical computing methods. The distinctive ability of quantum computers to efficiently process large datasets, simulate quantum systems, and optimize solutions has spurred significant interest and research. Variational quantum algorithms have shown promise to be utilized for computational fluid dynamics applications <cit.>. Quantum machine learning (QML) has particularly garnered attention as an interdisciplinary research area, leveraging quantum mechanics' inherent properties to potentially surpass classical machine learning methods. 
The integration of quantum computing (QC) and machine learning (ML) in dynamical systems offers promising advancements in forecasting capabilities. Quantum algorithms can handle high-dimensional data more efficiently, improving the accuracy of dynamical systems models. Concurrently, ML techniques enhance these models' interpretability and predictive power by leveraging large datasets and complex patterns <cit.>. 
Quantum algorithms have demonstrated polynomial speedups for both linear and nonlinear differential equations. These speedups become exponentially larger as the problem dimensions increase, showing promise for applications in computational fluid dynamics (CFD) and other areas <cit.>. 
Adiabatic annealing-based quantum computing methods have been studied for solving fluid dynamics problems such as the Navier-Stokes (NS) equation <cit.>. However, Lewis et al. (2023) <cit.> suggests that simulating the classical dynamics of turbulent or chaotic systems may not offer an exponential quantum advantage over current classical algorithms. 
Succi et al. (2024) <cit.> investigated the feasibility of using the Liouville formulation for ensemble simulations of fluid flows on quantum computers. However, the Liouville approach is anticipated to necessitate several hundred logical qubits, which is limited by current hardware. 
The inherently non-local nature of quantum mechanics might be beneficial for accurately representing the classical non-locality observed in low-Reynolds number flows <cit.>. 
Sanavio et al. (2024) <cit.> propose a quantum computing algorithm for fluid flows based on the Carleman-linearization
of the Lattice Boltzmann (LB) method.

Freeman et al. (2024) <cit.> introduced a novel data-driven parameterization scheme for unresolved components of dynamical systems using quantum mechanics and Koopman operator theory, successfully applied to the Lorenz 63 and 96 systems.
However, the method's scalability and practicality in complex, higher-dimensional systems remain to be fully assessed. Giannakis et al. (2022) <cit.> developed a framework for simulating measure-preserving, ergodic dynamical systems on quantum computers by combining ergodic theory with quantum information science. The framework shows promise for developing data-driven quantum algorithms for modelling nonlinear dynamics and forecasting real-world systems, but further exploration is needed for applications such as climate dynamics and turbulent fluid flows.
Gourianov et al. (2022) <cit.> utilize interscale correlations in turbulent flows to develop a structure-resolving simulation algorithm based on Matrix Product States (MPS). This quantum-inspired method significantly reduces computational complexity and shows potential for future implementation on quantum computers. 

Pfeffer et al. (2022) explored the use of a hybrid quantum-classical gate-based reservoir computing model to predict and reconstruct the dynamics of classical, nonlinear thermal convection flows <cit.>. The study demonstrated that the quantum reservoir, utilizing entangled qubits, can match the performance of classical reservoirs that use thousands of perceptrons, even when implemented on a noisy intermediate-scale quantum (NISQ) device. However, it's important to note that the model was tested on relatively low-dimensional systems. Extending this approach to higher-dimensional and more complex systems, such as turbulent flows, and real-world applications, remains a challenge.


Pfeffer et al. (2023) <cit.> proposed hybrid quantum-classical gate-based reservoir computing models to replicate low-order statistical properties of a two-dimensional turbulent Rayleigh-Benard convection flow. These models aim to learn the nonlinear and chaotic dynamics of the turbulent flow in a lower-dimensional latent data space, using time-dependent expansion coefficients of the most energetic Proper Orthogonal Decomposition (POD) modes. The training data is generated by conducting a POD-based compression of the direct numerical simulations of the original turbulent flow. The hybrid quantum reservoir computing models operate in a reconstruction mode, taking 3 POD modes as input at each step and reconstructing the missing 13 modes. The quantum models perform similarly to the classical model with the added cost of sophisticated quantum hardware. We are inspired by this study to develop a non-intrusive reduced-order modeling strategy that may offer significant benefits when the underlying governing equations are unavailable - and therefore not amenable to classical physics-based modeling.



Machine learning (ML) methods, and neural networks in particular, are an attractive approach for the computationally efficient forecasting of various realistic time-series data offering a data-driven alternative to traditional numerical methods which struggle with rapid prediction and real-time control in applications like weather and climate among others. Typical ML baselines include the long short-term memory network (LSTMs) <cit.> and Gated Recurrent Units (GRUs) <cit.>. They are both types of recurrent neural networks designed for sequence data without the problem of vanishing gradients, but they differ in their architectures and how they handle information flow. LSTMs are equipped with a more complex architecture that includes three gates: input, output, and forget gates, along with a memory cell that allows for better long-term memory retention and control over information flow. GRUs simplify this structure by using only two gates: reset and update gates, merging the cell state and hidden state into one. While LSTMs offer more control and flexibility due to their more intricate gating mechanisms, GRUs are computationally more efficient and easier to implement, often providing comparable performance in practice.

In this study, we investigate the use of data-driven methods to perform forecasts for a geophysical dataset given by the sea-surface temperature (SST). SST prediction is a commonly chosen benchmark for assessing the quality of a data-driven forecasting model. For example, Yang et al. (2017) <cit.> use CFCC-LSTMs and a 3D grid to deal with spatiotemporal information to predict future SST values. Zhang, Qin et al. (2017) <cit.> formulated the prediction of SST as a time series regression problem and proposed an LSTM-based network to model the temporal relationship of SST in order to predict future values.
Zhang, Zhen (2020)<cit.> proposed a fully connected neural network model based on GRU is designed to predict SST over the medium and long term.
 Xie et al. (2019)<cit.> proposed a gate recurrent unit (GRU) encoder-decoder with
SST codes and dynamic influence links (DIL) to predict SST for long-term predictions.


Reservoir Computing (RC) is a class of data-driven forecasting frameworks in which a fixed, randomly connected network, known as a reservoir, processes input streams nonlinearly. The processed data is then fed into a trainable readout for pattern analysis and processing of temporal or sequential data. Reservoir Computing (RC) has demonstrated significant success in modeling the dynamics of high-dimensional chaotic systems <cit.>.

Quantum Reservoir Computing (QRC) extends this paradigm by implementing the reservoir using quantum many-body systems, leveraging quantum dynamics for information processing. This adaptation opens avenues for exploring quantum advantages in temporal machine-learning tasks. In this context, Higher Order Quantum Reservoir Computing (HQRC) introduces a novel approach by organizing quantum systems into an ensemble reservoir. Each system within this ensemble, akin to a computational node, receives inputs and signals from other systems, enhancing computational power through controllable linear feedback connections. This higher-order scheme offers increased expressiveness and scalability, potentially unlocking advanced capabilities for temporal forecasting tasks.

This paper focuses on demonstrating the efficacy of Higher Order Quantum Reservoir Computing (HQRC) framework proposed by Tran and Nakajima (2020)<cit.> in forecasting Sea Surface Temperature (SST) data. By leveraging the unique features of quantum systems within an ensemble reservoir framework, we aim to showcase the potential of HQRC for addressing real-world prediction challenges. Through empirical evaluation and comparative analysis, we highlight the advantages of HQRC for SST forecasting, contributing to the growing body of research on quantum-enhanced machine learning techniques.



§ METHODS AND DATA




 §.§ Quantum Reservoir Dynamics


For ease of exposition and without loss of generality, let us consider a one-dimensional input and output case with the input sequence of scalars u = {u_1, …, u_L} and the corresponding target sequence ŷ = {ŷ_1, …, ŷ_L}, where u_k is a continuous variable in [0, 1]. Quantum Reservoir Computing (QRC) emulates a nonlinear function Q to produce the output y_k = Q(w, ρ^(0), {u_l}_l=1^k) (k = 1, …, L). Here, ρ^(0) is the initial state of the quantum system, and w is the parameter that needs to be optimized.

A temporal learning task consists of three phases: a washout phase, a training phase, and an evaluation phase. In the washout phase, the system evolves for the first T transient steps to wash out the initial conditions from the dynamics. The training phase to optimize w is performed with training data {u_k}_k=T^L_1, {y_k}_k=T^L_1, where 1 ≤ T < L_1 < L, such that the mean-square error between y_k and ŷ_k over k = T, …, L_1 becomes minimum. The trained parameter w is used to generate outputs in the evaluation phase.

For an N-qubits system, at time t = (k-1)τ, the input u_k ∈ [0, 1] is fed to the system by setting the density matrix of the first spin to

    ρ_u_k = (1 - u_k)|0⟩⟨ 0| + u_k |1⟩⟨ 1| ∈𝕄_2 × 2.

Therefore, the density matrix ρ∈𝕄_2^N × 2^N of the entire system is mapped by a completely positive and trace preserving (CPTP) map

    ρ→𝒯_u_k(ρ) = ρ_u_k⊗Tr_1[ρ],

where Tr_1 denotes a partial trace with respect to the first qubit. After the input is set, the system continues evolving itself during the time interval τ. The dynamics are governed by the Schrödinger equation, and the information of the input sequence encoded in the first spin spreads through the system. It follows that the state of the system before the next input u_k+1 is

    ρ^(k) = e^-iHτ𝒯_u_k(ρ^(k-1))e^iHτ,


where ρ^(k) = ρ(kτ) is the density matrix at t = kτ.

If we employ the ordered basis {O_j} in the operator space, then the observed signals at time t are the first N_out elements

    s_j(t) = Tr[ρ(t)O_j],

where the selection of observables depends on the physical implementation of the system. In quantum mechanics, the ordered basis {O_j} in the operator space typically consists of a set of linearly independent operators that span the space of all observables for a given system. For example, in the case of a single qubit, a common ordered basis consists of the Pauli matrices {I, σ_x, σ_y, σ_z}, where I is the identity matrix, and σ_x, σ_y, σ_z are the Pauli matrices. These matrices form a complete basis for the space of 2 × 2 Hermitian operators, meaning any observable for a single qubit can be expressed as a linear combination of these basis operators. The choice of basis depends on the physical implementation and measurement capabilities of the system.




 §.§ Temporal Multiplexing

A temporal multiplexing scheme is introduced to enhance performance in extracting dynamics. In this scheme, signals are measured not only at time kτ but also at each of the subdivided V time intervals during the evolution in the interval τ to construct V virtual nodes. The density matrix is then updated by

    ρ((k - 1)τ + 1/Vτ) = U_(τ/V) T_u_k(ρ^(k-1)) U^†_(τ/V),


    ρ((k - 1)τ + v/Vτ) = U_(τ/V)ρ((k - 1)τ + v - 1/Vτ) U^†_(τ/V),    (v = 2, …, V),

where U_(τ/V) = e^-iH(τ/V). Therefore, temporal signals from N_outV nodes can be obtained.

The learning procedure is straightforward, parameterizing the linear readout function as y_k = ∑_i=0^N_outV w_i x_ki, where x_ki = s_j((k - 1)τ + v/Vτ) for i = (j - 1)V + v > 0 (1 ≤ v ≤ V, 1 ≤ j ≤ N_out). Here, x_k0 = 1.0 are introduced as constant bias terms, and w = [w_0, w_1, …, w_N_outV] represents the readout weight parameters. If we denote K as the number of time steps used in the training phase, w is optimized via linear regression, or Ridge regression in the matrix form

    ŵ^T = (X^T X + β I)^-1 X^T ŷ,

where ŷ = [y_1, …, y_K]^T is the target sequence, X = [x_ki] ∈ℝ^K × (N_outV + 1) is the training data matrix, and β is the parameter serving as the positive constant shifting the diagonals introduced to avoid the problem of the near-singular moment matrix.





 §.§ Higher Order Quantum Reservoir Computing model

In the paper, The fully connected transverse field Ising model, a standard framework for building Quantum Reservoirs (QR), is employed. The Hamiltonian is given by H = J ∑_i ≠ j h_i,jσ^x_iσ^x_j + J ∑_j g_jσ^z_j, where σ^γ_ j (γ∈{x, y, z}) is the operator measuring the spin j along the γ direction. This operator can be described as an N-tensor product of 2 × 2-matrices:

    σ^γ_j = I ⊗…⊗σ^γ_j⊗…⊗ I,

where I = [ 1 0; 0 1 ], σ_x = [ 0 1; 1 0 ], σ_y = [  0 -i;  i  0 ], and σ_z = [  1  0;  0 -1 ]. J is the coupling magnitude of the Hamiltonian, while the coupling parameter h_i,j and the transverse field parameter g_j are uniformly selected from the range [-1.0, 1.0]. N_out = N observables O_j = σ^z_j are chosen to produce the signals of the readout nodes. 
A Higher-Order Quantum Reservoir (HQR) comprises an ensemble of N_qr Quantum Reservoirs (QRs). The l-th system Q_l has the Hamiltonian H_l with N_l qubits (known as true nodes) and V_l virtual nodes. In order to interface between an input of dimension N_in, we tile the input to match the HQRC dimension (Hence, ensuring N_in divides N_qr).

To simplify, we set all parameters the same for each system, i.e. H_l = H, N_l = N and V_l = V. Each individual reservoir is then evolved independently under a temporal multiplexing scheme as described above and the final readouts at each time step t = kτ
are compiled as a VN dimensional vector x_k = [x_k1, x_k2…, x_kVN]. 

The reservoir states of Q_l at time t = kτ (k ≥ 0) are thus each represented by a V_lN_l-dimensional vector x_kl, initialized at k = 0 as the zero vector x_0. Then, the reservoir states of the HQR at t = kτ can be represented by an N_total-dimensional vector z_k = [x_k1^T, …, x_kN_qr^T]^T, where N_total = ∑_l=1^N_qr N_lV_l. The scaling z → (z + 1)/2 is employed, such that the elements of z are in [0, 1].

At t = (k - 1)τ (k ≥ 1), the mixed input u'_kl, injected into Q_l, is a linear combination of the external input and the reservoir states from all QRs. The mixed input u'_k∈ℝ^N_qr× 1 can be represented in matrix form as

    u'_k = (1 - α)W_inu_k + α W_conz_k-1,

where W_con∈ℝ^N_qr× N_total is randomly generated and fixed to represent the linear feedback connection between the QRs, and α (0 ≤α≤ 1) is defined as the connection strength parameter. Note that, each reservoir is fed one input dimension and the final construction of z_k represents information from all the dimensions.

After injecting the mixed input u'_kl into Q_l, the density matrix in Q_l is transformed by the CPTP map, T_u'_kl, and consequently evolves in each τ/V time.

For the state evolution of the system to be replicated without requiring feeding the output from the readout part into each QR’s input, Let W_out∈ℝ^N_out× N_total denote the trained readout weights. In the case of forecasting, N_out = N_in. We replicate W_out into W'_out∈ℝ^N_qr× N_total (this is possible since N_qr is a multiple of N_in) and represent the input in the closed system when the external input u_k is withdrawn as

    u'_k = (1 - α)W'_outz_k-1 + α W_conz_k-1,

where z_k-1∈ℝ^N_total× 1 is the reservoir states of HQR at t = (k - 1)τ. The predicted output is given by 

    u^pred_k = W'_outz_k-1



We need to ensure that u^pred_k does not fall out of the range [0,1] and so we perform clipping to restrict the prediction within this range.

    u_k = 
     
    1    if  u_k > 1 
     
    u_k    if  0 ≤ u_k ≤ 1 
     
    0    if  u_k < 0

The number of parameters is given by the size of the Readout matrix, shape (N_total+1)× N_qr, where N_total=∑_l=1^N_qrcN_lV_l. For generalizing to higher dimensional cases, the aforementioned procedure is performed for each scalar component of the state-vector and W_in is learned jointly to account for interactions across components. Note that W_con is not learnt and is fixed randomly.





 §.§ The sea-surface temperature dataset


In this study, we use an open-source repository for our training and test data. Specifically, we utilize the National Oceanic and Atmospheric Administration (NOAA) Optimum Interpolation sea surface temperature V2 data set (henceforth NOAA-SST).[https://www.esrl.noaa.gov/psd/]  This data set exhibits a periodic behavior due to seasonal cycles in addition to fine-scaled phenomena emerging from complex ocean dynamics. Weekly-averaged NOAA-SST data are available on a quarter-degree grid which is sub-sampled to a one-degree grid for the purpose of our demonstrations. This data set has previously been used in several data-driven studies (for instance, see <cit.>), particularly for forecasting, extracting seasonal and long-term trends, and for flow-field recovery <cit.>. Each “snapshot” of data corresponds to an array of size 360 × 180 (based on a one-degree resolution). However, a mask is used to remove missing locations in the array that corresponds to the land area. Furthermore,  forecasts are performed for those coordinates which correspond to oceanic regions alone (and not inland bodies of water which are also masked). Unmasked data points are flattened to obtain a column vector for each snapshot of our training and test data.

The NOAA-SST data is available from October 22, 1981, to June 30, 2018 (i.e., 1,914 snapshots for the weekly averaged temperature). We utilize the period of October 22, 1981, to December 31, 1989 for training our model. From January 1990 onwards, the rest of the data is utilized for testing our various model deployments. This train-test split of the data set is commonly used in literature <cit.> and the 8-year training period captures several important trends in the global sea surface temperature. Since this data set is produced by combining local and satellite temperature observations, it represents an attractive forecasting task for non-intrusive data-driven methods without requiring the physical modeling of underlying processes.

Each snapshot of the SST, representing the system, is a vector in ℝ^N, where N is the number of grid points. We calculate the mean:  D̅ = 1/T∑_t D_t, and hence define the mean-subtracted snapshots as:  y_t = D_t - D̅. Proper Orthogonal Decomposition (POD) is utilized to identify a set of orthonormal basis vectors (v_1, ..., v_M), where M < N, approximating spatial snapshots, such that y_t ≈∑_j=1^M a_j(t) v_j, t = 1, …, T. POD chooses the basis, v_j , to minimize the residual with respect to the L_2 norm, R = ∑_t=1^T ||y_t - ∑_j=1^M a_j(t)v_j ||^2 Defining the snapshot matrix,  S = [y_1| … |y_T], the optimal basis is given by the M  eigenvectors of  SS^T  with largest eigenvalues. Afterward, the coefficients are found by orthogonal projection: a(t) = ⟨ y_t, v ⟩., and models are trained to forecast future coefficients based on past ones. Predictions for coefficients are converted to physical space by adding the mean snapshot and a linear combination of coefficients and corresponding basis vectors, D̅ + ∑_jâ_j v_j. In this study, we use 5 POD modes to compress the original dataset for building an efficient time-series forecasting model.  We employ MinMaxZeroOne scaling to normalise the training input data POD modes to the reservoir into the range [0,1]. For the test POD modes, we reuse the same minimum and maximum feature values of the training modes to avoid data leakage.

We emphasize here that the proposed time-series forecasting technique makes predictions for the SST autoregressively – this is in comparison with past studies which have made windowed predictions (i.e., the ground truth input is always assumed to be observed). This mode of deployment is considerably more challenging as the trained model is expected to be stable to perturbations introduced by iterative prediction errors.



§ RESULTS


In this section, we outline the results from our trained quantum reservoir models for forecasting the POD-compressed sea-surface temperature. We also compare the performance of the proposed approach with benchmark data-driven regression models for time-series data to establish any gains. We evaluate metrics from our forecasts of POD coefficients by reconstructing the state on the original latitude-longitude grid. Predictions are made for autoregressive rollouts over 300 predicted timesteps over which time-averaged metrics are obtained. Ensemble estimates are also provided for the top-5 models based on validation performance. Here, we define validation performance as the autoregressive rollout for 300 timesteps given a starting point within the first 127 snapshots.

We average the predictions from five different models, each with the best performance on the training data. These models were trained after tuning a set of hyperparameters, the results of which can be found in the Appendix. To ensure fairness, we spent approximately 10 hours optimizing the hyperparameters for each model using an Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz 2.30 GHz. For consistency, we predicted 300 weeks (time steps) from a randomly selected starting point during the test duration. In all the figures displayed, we have started from the 116th week.
Figure <ref>, <ref>, <ref> and <ref> shows the target and average prediction and the corresponding standard deviation (shaded) over the ensemble of the top 5 best-performing models on training and test data modal coefficients. Figure <ref>, <ref>, <ref> and <ref> show the reconstruction and prediction error grids for training and test data for the last prediction week. The land region is masked. The RMSE errors over the entire grid and for the East Pacific region (corresponding to latitudes and longitudes in the region between -10 to +10 degrees latitude and 200-250 degrees longitude) for test and training data, averaged over 300 predicted timesteps, are all shown in Table <ref> along with computational performance metrics like training time, memory and number of parameters used. It is to be noted that the lowest-possible reconstruction error for training data is 0.458, and for test data, it is 0.491, so expecting a lower RMSE is impractical. This error corresponds to that left over from the process of projecting the original state on to reduced-set of POD basis vectors and is irreducible.

It is to be noted that the models with the best performance in training data need not be the ones with the best performance on test data as well (and were often observed to be different). This is usually attributed to overfitting and the usual strategies for mitigation have been applied here (such as early stopping on held-out validation data). It is observed that the HQRC method clearly outperforms GRUs, LSTMs and ESNs in terms of the RMSE error metrics. Moreover, computationally, it is significantly less expensive than LSTMs and GRUs to train and deploy, as it takes less time to train and requires fewer trainable parameters. However, the classical reservoir-based model i.e ESNs, although computationally cheaper to train, perform worse than the other methods. The HQRC model thus proves to be an effective balance of the efficiency of the ESN and the learning abilities of more complex neural network based models for forecasting. We go into the specific details of each model's performance in the following.

HQRC: The model is able to accurately capture the trend for all dimensions of the training data; It able to capture the trend for the test data fairly well for the first three dimensions, it is even able to capture the large scale trends for the last two dimensions of the test data, but it is unable to capture the small scale fluctuations well.
The model is particularly stable in its performance, and the shaded region, i.e. the standard deviation, only becomes prominent for the later timesteps for the test data. We perform several ablation studies for the HQRC, including the influence of the input-window length, the number of reservoirs, the effect of compression (and consequently dimensionality), as well as the sensitivity to initial condition perturbations in the Appendix. 







GRU: While GRU performs fairly well for training data, deviations are observed over the duration of the rollout. It can also be observed that the standard deviation between the models increases as the prediction proceeds autoregressively. For test data, the prediction becomes flat for all the dimensions, along with a huge standard deviation, indicating a disagreement between the models (this is another commonly observed mode of failure for autoregressively deployed time-series forecasting models). As observed from the prediction reconstruction for test data (Figure <ref>), a significant fraction of the domain shows high errors.






LSTM: LSTMs tend to perform worse on training data, but perform better on test data in comparison with GRUs. This is likely because GRU deployments overfit on training data. However, a similar gradual convergence to a fixed-point is obtained during autoregressively deployments for the LSTM as well (although this is less pronounced than the GRU). For a fair comparison of LSTMs and GRUs, we utilize approximately the same input window length. We note that for both LSTMs and GRUs this window length is given by SL + HSPL/4 where SL is the sequence length and HSPL is the hidden state propagation length. The former refers to the duration in time for which backpropagation is performed before encoding the input sequence and the latter refers to number of times the hidden state is recursively updated before the prediction commences. These values are set according to the parameters chosen in Vlachas et al., <cit.>.  Note that the prediction length (PL) is set to 1 for an autoregressive rollout.

 






ESN: Finally, we assess the performance of ESNs which are very similar in terms of computational expense and number of parameters to the HQRC. We observe large errors during testing when deploying the ESN, despite good performance during training (for the top 5 models by validation error) - which suggests a tendency to overfit. We note that the issue of flatlining during test deployments is not observed unlike the GRU and LSTM models. Specific details of the best hyperparameters for the QRC and other benchmark models are available in the Appendix. 










§ CONCLUSION


In this paper, we have introduced a novel time-series forecasting technique for compressed representations of dynamical systems using higher-order quantum reservoir computing. Our approach relies on the quantum embedding of the state-space into a density matrix that may be evolved by unitary operations performed by multiple quantum reservoirs. This leads to the emergence of nonlinearity from solely linear operations and is viable for forecasting low-dimensional representations of complex nonlinear dynamical systems. Our proposed approach is efficient in terms of training time and memory consumption and more accurate when compared to classical deep learning based methods for forecasting regression tasks. We demonstrate this on the task of forecasting a real-world complex dynamical system benchmark given by the NOAA optimum interpolation sea-surface temperature dataset. However, we also observe limitations of our method when increasing the dimensionality of the state-space which motivates future studies with respect to optimal selection of state-space (i.e., modal) components from our time-series data before the construction of the density matrix. Furthermore, future work may also benefit from examinations of HQRC performance on multiple modalities of data - for instance when time-series from different modalities of sensing may be incorporated into the state-vector evolution.



§ ACKNOWLEDGEMENTS


This research used resources of the Argonne Leadership Computing Facility, which is a U.S. Department of Energy Office of Science User Facility operated under contract DE-AC02-06CH11357. RM acknowledges funding support from ASCR for DOE-FOA-2493 “Data-intensive scientific machine learning”.







unsrt  
  





§ APPENDIX

For all the models, the NRMSE for the coefficients is averaged over 3 test cases selected randomly as starting points. Testing indices (Week from which the prediction starts ∈{116,40, 66}) for over 300 timesteps. The NRMSE is defined as √(⟨(Y_true - Y_pred)^2/σ^2⟩) where ⟨ . ⟩ denotes average over all timesteps.






















 §.§ Effect of HQRC washout length


In the above experiments, We used 40 warming-up steps, i.e. 40 timesteps were fed into the Quantum reservoir model before starting the prediction for 300 steps. This was done to ensure that the reservoir state z captured the trend of the time series and to wash out or forget the bias of the initial state <cit.>. We wanted to evaluate the performance of the QRC model for different dynamics lengths (or washout steps). Table <ref> shows the RMSE results for different dynamics lengths, keeping other hyperparameters constant (corresponding to the model with the best training data performance). As expected, the performance seems to improve as we increase the dynamics length, this is because the reservoir state is better able to washout the effect of the initial state and capture the dynamics of the data. 

























 §.§ Sensitivity to initial condition perturbations


Figure <ref> Shows predictions when slight perturbations added at the first timestep. The model is trained with V=10, α = 0.5, β = 10^-7 and rest is same as above. The mean over perturbations and the zero perturbation predictions tend to coincide for the most part for the first three dimensions, indicating the stability of the model. For the last two dimensions, the standard deviation over different perturbations tends to increase with time.




 §.§ The effect of modal compression


Instead of keeping just 5 modes, we decided to keep 10 modes to see to if the performance is affected. Figure <ref> shows the average prediction over the top 5 models based on training data (see appendix), for the 300 timesteps for the test data. We note that the predictions fluctuate a lot, and the standard deviation between the models is very high.

The final RMSE error for the training data over all 300 predicted timesteps (between 116^th and 415^th weeks) is 1.220, compared with the corresponding default reconstruction RMSE of 0.458. The RMSE restricted to the East pacific region for the HQRC prediction is 1.414.
The RMSE between 116^th and 415^th weeks of the test data is 1.095 and in the East pacific region is 1.305.
Figure <ref> shows the reconstruction error grid from the 10 predicted mode coefficients.








 §.§ Results for different Qubits/Reservoirs

Figure <ref> compares RMNSE values for different parameters while keeping other parameters fixed. This justifies our choice of chosen hyperparameters.







 Paper Title: Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks.


	Fluctuating Line-of-Sight Fading Distribution: Statistical Characterization and Applications 


    
        Thanh Luan Nguyen1, 
        Georges Kaddoum1^,2
        
    
    1 Department of Electrical Engineering, École de Technologie Supérieure (ÉTS), Montréal, QC, Canada 

    2 Artificial Intelligence & Cyber Systems Research Center, Lebanese American University
    

      Emails: 
        1thanh-luan.nguyen.1@ens.etsmtl.ca,
        1^,2georges.kaddoum@etsmtl.ca, 

    August 5, 2024
===========================================================================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
First-order methods, such as gradient descent (GD) and stochastic gradient descent (SGD) have been proven effective in training neural networks. In the setting of over-parameterization, there is a line of work demonstrating that randomly initialized (stochastic) gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. However, the learning rate of GD in training two-layer neural networks has a poor dependence on the sample size and the Gram matrix, resulting in a slow training process. In this paper, we show that for the L^2 regression problems, the learning rate can be improved from 𝒪(λ_0/n^2) to 𝒪(1/H^∞_2), which implies that GD enjoys a faster convergence rate. Moreover, we further generalize the method for GD in training two-layer Physics-Informed Neural Networks (PINNs), showing a similar improvement for the learning rate. Although the improved learning rate depends mildly on the Gram matrix, we still need to set it small enough in practice due to the agnostic eigenvalues of the Gram matrix. More importantly, the convergence rate relies on the least eigenvalue of the Gram matrix, leading to slow convergence. In this work, we provide the convergence analysis of natural gradient descent (NGD) in training two-layer PINNs. We show that the learning rate can be 𝒪(1) and at this time, the convergence rate is independent of the Gram matrix. 




§ INTRODUCTION


In recent years, neural networks have achieved remarkable breakthroughs in the fields of image recognition <cit.>, natural language procssing <cit.>, reinforcement learning <cit.> and so on. Moreover, due to the flexibility and scalability of neural networks, researchers are paying much attention in exloring new methods involving neural networks for handling problems in scientific computing, where one long-standing and essential problem is solving partial differntial equations (PDEs) numerically. Classical numerical methods, such as finite difference, finite volume and finite elements methods, suffer the curse of dimensionality in solving high-dimensional PDEs. Due to this drawback, various methods involving neural networks have been proposed for solving different type PDEs <cit.>. Among them, the most representative approach is Physics-Informed Neural Networks (PINNs) <cit.>. In the framework of PINNs, one incorporate PDE constraints into the loss function and train the neural network with it. With the use of automatic differentiation, the neural network can be efficiently trained by first-order or second-order methods.

In the applications of neural networks, one  inevitable issue is the selection of the optimization method. First-order methods, such as gradient descent (GD) and stochastic gradient descent (SGD), are widely used in optimizing neural networks as they only calculate the gradient and therefore are computationally efficient. Beside the first-order methods, there has been much interest in utilizing second-order optimization methods to speed up the training, not only for the regression problems <cit.> but also for the problems related to PDEs <cit.>.  

For the convergence part of the optimization method, it has been shown that gradient descent algorithm can even achieve zero training loss under the setting of over-parametrization, which refers to a situation where a model has more parameters than necessary to fit the data <cit.>. These works are based on the idea of neural tangent kernel (NTK), which shows that training multi-layer fully-connected neural networks via gradient descent is equivalent to performing a certain kernel method as the width of every layer goes to infinity. As for the finite width neural networks, with more refined analysis, it can be shown that the parameters are closed to the initialization throughout the entire training process when the width is large enough. It directly leads to the linear convergence for GD. Despite these beautiful convergence results, the learning rate depends on the sample size and the Gram matrix. Thus, it need to be sufficiently small to guarantee the convergence in practice. But doing so results in a slow training process. In contrast to the first-order methods, the second-order methods enjoy fast convergence due to the absence of requirements for the learning rate <cit.>.  



 §.§ Contributions

The main contributions of our work can are summarized as follows:

	
  * For the L^2 regression problems, we demonstrate that the learning rate η of gradient descent can be improved from 𝒪(λ_0/n^2), as shown in <cit.>, to 𝒪(1/H^∞_2), where H^∞ is the Gram matrix induced by the ReLU activation function and the random initialization, λ_0 is the least eigenvalue of H^∞. <cit.> has also shown the same improvement of the learning rate, but it requires that m=Ω(n^6/λ_0^4δ^3). Moreover in <cit.>,  the dependence on n, i.e. Ω(n^6), is necessary and can not be improved due to the requirement of the proof method. Different from the method in <cit.>, our method comes from a new recursion formula of gradient descent and it can be easily generalized to PINNs. Compared to <cit.>, we only require that m=Ω(n^4/λ_0^4(log(n/δ))^2). 
	
	
  * For the PINNs, we simultaneously improve both the learning rate η of gradient descent and the requirement of the width m. The improvements rely on a new recursion formula of gradient descent, which is similar to that for regression problems. Specifically, we can improve the learning rate η=𝒪(λ_0) required in <cit.> to η=𝒪(1/H^∞_2) and the requirement for the width m, i.e. m=Ω̃((n_1+n_2)^2/λ_0^4 δ^3), can be improved to m=Ω̃(1/λ_0^4(log(n_1+n_2/δ))). 
	
	
  * We provide the convergence results for natural gradient descent (NGD) in training over-parameterized two-layer PINNs with ReLU^3 activation functions and smooth activation functions. Compared to gradient descent, the convergence rate of NGD is independent of λ_0, which results in a faster convergence. Meanwhile, we show that the learning rate can be even 𝒪(1). When the activation function is smooth, NGD enjoys quadratic convergence rate.




 §.§ Related Works

First-order methods. There are mainly two ways to study the optimization of neural networks and try to understand why first-order methods can find a global minimum. One way is to analyse the optimization landscape <cit.>. It has been shown that gradient descent can find a global minimum in polynomial time if the optimization landscape has some good geometric properties. However, some unrealistic assumptions in these works make it difficult to generalize to practical neural networks. Another way to understand the optimization of neural networks is to analyze the optimization dynamics of first-order methods. For the two-layer ReLU neural networks, as shown in <cit.>, as long as the width m is large enough and no two inputs are parallel, randomly initialized gradient descent converges to a globally optimal solution at a linear rate. Later, the results have been generalized to deep neural networks with smooth activation functions <cit.>. Both rely on that the Gram matrices are stable throughout the training process, which leads to the convergence to the global minimum. Besides regression and classification problems, <cit.> demonstrated the convergence of the gradient descent for two-layer PINNs by the similar analysis of optimization dynamics. However, both <cit.> and <cit.> require that the learning rate be sufficiently small to converge. In this work, we conduct a refined analysis of gradient descent for L^2 regression problems and PINNs, leading to a milder requirement for the learning rate.

Second-order methods. Although second-order methods possess better convergence rate, they are rarely used in training deep neural networks due to the prohibitive computational cost. As a variant of the original Gauss-Newton method, natural gradient descent (NGD) is more efficient in practice. Meanwhile, as shown in <cit.> and <cit.>, it also enjoys faster convergence rate for the L^2 regression problems than of gradient descent. In this paper, we provide the convergence analysis for NGD in training two-layer PINNs. 




 §.§ Notations
 We denote [n]={1,2,⋯, n} for n∈ℕ. Given a set S, we denote the uniform distribution on S by Unif{S}. We use I{E} to denote the indicator function of the event E. For variables a and b, a≲ b is equivalent to a≤ Cb for some universal constant C. For two functions f_1(n) and f_2(n), the relation f_1(n)=𝒪(f_2(n)) is equivalent to f_1(n)≤ Cf_2(n) for some universal constant C. And similarly, f_1(n)=Ω(f_2(n)) is equivalent to f_1(n)≥ Cf_2(n). Throughout the paper, we use boldface denote vectors. Given x_1,⋯, x_d∈ℝ, we use (x_1,⋯, x_d) or [x_1,⋯, x_d] to denote a row vector with i-th component x_i for i∈ [d] and then (x_1,⋯, x_d)^T∈ℝ^d is a column vector.




 §.§ Organization of this Paper

We present the improvements of the learning rate of gradient descent for L^2 regression problems and PINNs in Section 2 and Section 3 respectively. In Section 4, we show the convergence of natural gradient descent for training PINNs with ReLU^3 activation functions and smooth activation functions. We conclude in Section 5 and the detailed proofs are put in the Appendix for readability and brevity.



§ IMPROVED LEARNING RATE OF GRADIENT DESCENT FOR L^2 REGRESSION PROBLEMS




 §.§ Problem Setup

In this section, we consider a two-layer neural network f with the following form.

    f(x;w,a)=1/√(m)∑_r=1^m a_r σ(w_r^T x̃),

where x∈ℝ^d is the input, x̃=(x^T, 1)^T ∈ℝ^d+1, w=(w_1^T, ⋯, w_m^T)^T, a=(a_1, ⋯, a_m)^T, for r∈ [m], w_r∈ℝ^d+1 is the weight vector of the first layer, a_r∈ℝ is the output weight and σ(·) is the ReLU activation function. Different from the setup in <cit.>, the two-layer neural network that we consider has bias term. In the following, we assume that each input vector x∈ℝ^d has been augmented to x̃=(x^T, 1)^T ∈ℝ^d+1. With a little abuse of notation, we write x for x̃ and write f as

    f(x;w,a)=1/√(m)∑_r=1^m a_r σ(w_r^T x).

For the L^2 regression problem, given training data set {(x_i, y_i)}_i=1^n, the aim is to minimize the loss function 

    L(w,a):=∑_i=1^n 1/2 (f(x_i;w,a)-y_i)^2.


Before training, we initialize the first layer vector w_r(0)∼𝒩(0, I) and output weight a_r ∼ Unif({-1,1}) for r∈ [m]. In the training process, we fix the second layer and optimize the first layer by gradient descent (GD), i.e., we update the weights by the following formulation.

    w(k+1)=w(k)-η∂ L(w(k))/∂w,

where η>0 is the learning rate, k∈ℕ and L(w(k)) is an abbreviation of L(w(k),a). From the form of L(w), we can deduce that

    ∂ L(w)/∂w_r=a_r/√(m)∑_i=1^n (f(x_i;w,a)-y_i)x_i I{w_r^T x_i ≥ 0}.


We denote u_i(k)=f(x_i;w(k),a) the prediction on input x_i at k-th iteration and we let u(k)=(u_1(k),⋯, u_n(k))^T ∈ℝ^n be the prediction vector at k-th iteration. For the labels, we define y:=(y_1,⋯, y_n)^T ∈ℝ^n. 

According to <cit.>, in the continuous setting, the dynamics of predictions can be written as 

    d/dtu(t)=H(t)(y-u(t)),

where H(t)∈ℝ^n× n is the Gram matrix at time t with (i,j)-entry

    H_ij(t)=1/mx_i^T x_j ∑_r=1^m I{w_r(t)^T x_i≥ 0, w_r(t)^T x_j≥ 0}.


In the setting of over-parameterization and randomly initialization, <cit.> has shown that (1) H(0)-H^∞_2 =𝒪(√(1/m)), where H^∞ is the Gram matrix induced by the initialization with (i,j)-th entry

    H_ij^∞=𝔼_w∼𝒩(0,I)[x_i^T x_j ∑_r=1^m I{w^T x_i≥ 0, w^T x_j≥ 0}]

and (2) H(t)-H(0)_2 =𝒪(√(1/m)) for all t>0. Therefore, as m→∞, the dynamics of the predictions are charactered by H^∞, leading to the linear convergence.



 §.§ Main Results

To simplify the analysis, we make the following assumptions on the training data.

For i∈ [n], x_i_2 ≤√(2) and |y_i|≤ 1, 
where x_i ∈ℝ^d+1 is the augmented input.



No two samples in {x_i}_i=1^n are parallel, i.e., for any x_t,x_s ∈{x_i}_i=1^n and any α∈ℝ, we have x_t ≠x_s. 


Since inputs are all augmented, Assumption 2 is equivalent to that no two samples in {x_i}_i=1^n are equal, which holds naturally. Under Assumption 2, Theorem 3.1 in <cit.> implies that λ_0:=λ_min(H^∞)>0, which is crucial in the convergence analysis. 

As stated before, there are two improtant facts that determine the optimization dynamics, one is that at initialization H(0) is closed to H^∞ and another is that H(k) does not go far away from the initialization H(0) for all k∈ℕ. These facts are supported by the following two lemmas.


If m=Ω(n^2/λ_0^2log(n/δ)), we have with probability at least 1-δ, H(0)-H^∞_2≤λ_0/4 and λ_min(H(0))≥3λ_0/4.



Let R∈ (0,1], if w_1(0),⋯,w_m(0) are i.i.d. generated from 𝒩(0,I), then with probability at least 1-n^2e^-mR/4, the following holds. For any set of weight vectors w_1,⋯, w_m∈ℝ^d+1 that satisfy for any r∈ [m], w_r-w_r(0)_2<R, then the matrix H(w)∈ℝ^n× n defined by

    H(w)_ij=1/mx_i^T x_j∑_r=1^m I{w_r^Tx_i≥ 0, w_r^Tx_j≥ 0}.

satisfies 	

    H(w)-H(0)_F < 4nR.




Although Lemma 2 looks almost same to Lemma 3.1 in <cit.>, the proof of Lemma 3.1 in <cit.> lacks an important part, i.e., the proof of the measurability for the related variable. In fact, Lemma 2 ensures us to bound the change of H(w) when w in a small ball. However, the set consisting of vectors in the small ball is uncountable, thus the related variable may not be measurable due to the discontinuity of the indicator function appeared in the definition of H(w). In the proof, we borrow the concept of pointwise measurability in the empirical process (see Chapter 8 in <cit.>) to bridge the gap. Then we can bound the supremum with a new random variable. By applying a concentration inequality to this new random variable, we can reach the conclusion.	

	
Abount the convergence of gradient descent, <cit.> has demonstrated that if the learning rate η=𝒪(λ_0/n^2), then randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate when m is large enough. The requirement of η is from the decomposition for the residual in the (k+1)-th iteration, i.e.,

    y-u(k+1)=y-u(k)-(u(k+1)-u(k)).

Instead of decomposing the residual into the two terms as above, we write it as follows, which serves as a recursion formula.


For all k∈ℕ, we have

    y-u(k+1)=(I-η H(k))(y-u(k))-I_1(k),

where I_1(k)=(I_1^1(k),⋯, I_1^n(k))^T∈ℝ^n with i-th 

    I_1^i(k):=u_i(k+1)-u_i(k)-⟨∂ u_i(k) /∂w, w(k+1)-w(k)⟩.



After <cit.>, <cit.> improved the network size m to m=𝒪(λ_0^-4n^4log^3(n/δ)log(m/δ)), but it still requires η=𝒪(λ_0/n^2). Another work <cit.> provided a refined analysis over <cit.>, which shows that η=1/C_1H^∞_2 is enough for the convergence rate 1-λ_0 C_2/C_1H∞_2, but the constants C_1,C_2 may depend on the parameters λ_0,n and δ. It requires the network size m to be Ω(λ_0^-4n^6δ^-3) to make C_1,C_2 independent of λ_0,n and δ. In this paper, we improve both the learning rate and network size and the main reslut is the following theorem.


Under Assumption 1 and Assumption 2, if we set the number of hidden nodes m=Ω(n^4/λ_0^4log( n/δ) ) and the learning rate η =𝒪(1/H^∞_2), then with probability at least 1-δ over the random initialization, the gradient descent algorithm satisfies 

    y-u(k)_2^2 ≤( 1-ηλ_0/2)^k y-u(0)_2^2

for all k∈ℕ.



Our proof method requires that I-ηH(k) is positive definite for all k∈ℕ. As H(k) is closed to H^∞, the requirement is satisfied if ηH^∞_2≲ 1. Note that H^∞_2≤ n, it is sufficient to set η=𝒪(1/n), which results in an improvement of 𝒪(λ_0/n).




§ IMPROVED LEARNING RATE OF GRADIENT DESCENT FOR TWO-LAYER PHYSICS-INFORMED NEURAL NETWORKS




 §.§ Problem Setup


In this section, we consider the same setup as <cit.>, i.e., we focus on the PDE with the following form.

    {   ∂ u/∂ x_0(x)-∑_i=1^d ∂^2 u/∂ x_i^2(x)=f(x),  x∈ (0,T)×Ω, 
       u(x)=g(x),  x∈{0}×Ω∪ [0,T]×∂Ω,
    .

where x=(x_0,x_1,⋯,x_d)^T∈ℝ^d+1 and x_0 ∈ [0,T] is the time variable. In the following, we assume that x_2≤ 1 for x∈ [0,T]×Ω̅ and f,g are bounded continuous functions. 

Moreover, we consider a two-layer neural network of the following form.

    ϕ(x;w,a)=1/√(m)∑_r=1^m a_rσ(w_r^T x̃),

where w=(w_1^T,⋯, w_m^T)^T∈ℝ^m(d+2), a=(a_1,⋯, a_m)^T ∈ℝ^m and for r∈ [m], w_r∈ℝ^d+2 is the weight vector of the first layer, a_r is the output weight and σ(·) is the ReLU^3 activation function. Similar as that in Section 2, x̃=(x^T,1)^T∈ℝ^d+2 is the augmented vector from x and in the following, we write x for x̃ for brevity. 

In the framework of PINNs, given training samples {x_p}_p=1^n_1 and {y_j}_j=1^n_2 that are from interior and boundary respectively, we aim to minimize the following empirical loss function.

    L(w,a)   := ∑_p=1^n_11/2n_1( ∂ϕ/∂ x_0(x_p;w,a)-∑_i=1^d ∂^2 ϕ/∂ x_i^2(x_p;w,a)-f(x_p))^2
          + ∑_j=1^n_21/2n_2(ϕ(y_j;w,a)-g(y_j))^2.


Similar as that for the L^2 regression problems, we initialize the first layer vector w_r(0)∼𝒩(0, I), output weight a_r ∼ Unif({-1,1}) for r∈ [m] and fix the output weights. Then the gradient descent updates the hidden weights by the following formulations:

    w_r(k+1)   =w_r(k)-η∂ L(w(k), a)  /∂w_r

for all r∈ [m] and k∈ℕ, where η>0 is the learning rate. For brevity, we write L(w) for L(w,a).

To simplify the notations, for the residuals of interior and boundary, we denote them by s_p(w) and h_j(w) respectively, i.e.,

    s_p(w)=1/√(n_1)( ∂ u/∂ x_0(x_p;w)-∑_i=1^d ∂^2 u/∂ x_i^2(x_p;w)-f(x_p) )

and 

    h_j(w)=1/√(n_2)(u(y_j;w)-g(y_j))	.

Then the empirical loss function can be written as

    L(w)=1/2(s(w)_2^2+ h(w)_2^2),

where

    s(w)=(s_1(w), ⋯, s_n_1(w))^T∈ℝ^n_1

and

    h(w)=(h_1(w), ⋯, h_n_2(w))^T ∈ℝ^n_2.


At this time, we have

    ∂ L(w)/∂w_r=∑_p=1^n_1s_p(w)∂ s_p(w)/∂w_r+∑_j=1^n_2h_j(w)∂ h_j(w)/∂w_r

and the Gram matrix H(w) is defined as

    H(w)=D^T D,  D:=[ ∂ s_1(w)/∂w,⋯, ∂ s_n_1(w)/∂w, ∂ h_1(w)/∂w, ⋯, ∂ h_n_2(w)/∂w].





 §.§ Main Results

First, we make the following assumptions about the training samples, which are similar to those for the L^2 regression problems.


For p∈ [n_1] and j∈ [n_2], x_p_2 ≤√(2), y_j_2 ≤√(2), where all inputs have been augmented.



No two samples in {x_p}_p=1^n_1∪{y_j}_j=1^n_2 are parallel.


Under Assumption 4, Lemma 3.3 in <cit.> implies that the Gram matrix H^∞:=𝔼_w∼𝒩(0,I) H(w) is strictly positive definite and we let λ_0=λ_min( H^∞). Similarly, H^∞ plays an important role in the optimization process. The following two lemmas indicate that H(0)-H^∞_2=𝒪(1/√(m)) and H(k)-H(0)_2=𝒪(1/√(m)), leading to the linear convergence.


If m=Ω(d^4/λ_0^2log(n_1+n_2/δ)), we have that with probability at least 1-δ, H(0)-H^∞_2≤λ_0/4 and λ_min(H(0))≥3/4λ_0.



Let R∈ (0,1], if w_1(0),⋯,w_m(0) are i.i.d. generated from 𝒩(0,I), then with probability at least 1-δ-(n_1+n_2)^2e^-mR, the following holds. For any set of weight vectors w_1,⋯, w_m∈ℝ^d+1 that satisfy for any r∈ [m], w_r-w_r(0)_2<R, then 

    H(w)-H(0)_F < CM^2R,

where M=2(d+2)log(2m(d+2)/δ) and C is a universal constant.


In <cit.>, the decomposition for the residual in the (k+1)-th iteration is same as the one in <cit.>, i.e., 

    [ s(k+1); h(k+1) ]=[ s(k); h(k) ] +([ s(k+1); h(k+1) ]-[ s(k); h(k) ]) ,

which leads to the requirement η=𝒪(λ_0).
Instead, we can generalize easily the method used in the L^2 regression problems to PINNs and obtain the following recursion formula.


For all k∈ℕ, we have

    [ s(k+1); h(k+1) ]=(I-ηH(k))[ s(k); h(k) ] +I_1(k),

where I_1(k)=(I_1^1(k), ⋯, I_1^n_1(k), I_1^n_1+1(k),⋯, I_1^n_1+n_2(k))^T ∈ℝ^n_1+n_2 and for p∈ [n_1],

    I_1^p(k)=s_p(k+1)-s_p(k)-⟨∂ s_p(k) /∂w, w(k+1)-w(k)⟩,

for j∈ [n_2],

    I_1^n_1+j(k)=h_j(k+1)-h_j(k)-⟨∂ h_j(k) /∂w, w(k+1)-w(k)⟩.



    m=Ω(d^12/λ_0^4log^6 (md/δ)  log(n_1+n_2/δ) ).

By using the recursion formula, we arrive at our main result.

Under Assumption 3 and Assumption 4, if we set the number of hidden nodes 

    m=Ω(d^12/λ_0^4log^6 (md/δ)  log(n_1+n_2/δ) )

and the learning rate η =𝒪(1/H^∞_2), then with probability at least 1-δ over the random initialization, the gradient descent algorithm satisfies 

    [ s(k); h(k) ]_2^2 ≤(1-ηλ_0/2)^k [ s(0); h(0) ]_2^2

for all k∈ℕ.


It may be confused that <cit.> has uesd the same method in <cit.>, but it only requires η=𝒪(λ_0). Actually, it is due to that the loss function of PINN has been normalized. If we let n_1=n_2=n and H^∞ be the Gram matrix induced by unnormalized loss function of PINN, then λ_min(H^∞)=λ_min(H^∞)/n, leading to the convergence rate similar to that of regression problem. Due to the normalization of loss function, H^∞_2 is independent of the sample size n, but may depends on the dimension d. 



§ CONVERGENCE OF NATURAL GRADIENT DESCENT FOR TWO-LAYER PHYSICS-INFORMED NEURAL NETWORKS




 §.§ Problem Setup


Although we have improved the learning rates of gradient descent for L^2 regression problems and PINNs, one needs to set the learning rates to be small enough due to the agnostic H^∞_2. For instance, we may need to set it to be 𝒪(1/n) for the L^2 regression problem. Moreover, the convergence rate 1-ηλ_0/2 also depends on λ_0, which may be slow with small λ_0. <cit.> and <cit.> have provided the convergence results for natural gradient descent (NGD) in training over-parameterized two-layer neural networks for L^2 regression problems. They showed that the maximal learning rate can be 𝒪(1) and the convergence rate is independent of λ_0, which result in a faster convergence rate. In the section, we conduct the convergence analysis of NGD for PINNs and show that NGD indeed leads to an even faster convergence rate.  

We consider the same setup as Section 3 and aim to minimize the following empirical loss function via NGD.

    L(w):=1/2(s(w)_2^2+ h(w)_2^2).


The NGD gives the following update rule:

    w(k+1)=w(k)-ηJ(k)^T(J(k)J(k)^T)^-1[ s(k); h(k) ],

where J(k)=( J_1(k)^T,⋯, J_n_1+n_2(k)^T )^T∈ℝ^(n_1+n_2)× m(d+2) is the Jacobian matrix for the whole dataset and η>0 is the learning rate. 
Specifically, for p∈ [n_1],

    J_p(k)=[(∂ s_p(k)/∂w_1)^T, ⋯, (∂ s_p(k)/∂w_m)^T]∈ℝ^1× m(d+2)

and for j∈ [n_2],

    J_n_1+j(k)=[(∂ h_j(k)/∂w_1)^T, ⋯, (∂ h_j(k)/∂w_m)^T]∈ℝ^1× m(d+2).



For the activation function of the two-layer neural network

    ϕ(x;w,a)=1/√(m)∑_r=1^m a_rσ(w_r^T x),

we consider both settings that σ(·) is the ReLU^3 activation function or a smooth activation function that satisfies the following assumption. From Lemma 3.3 in <cit.> and Lemma 2 in <cit.>, we know that H^∞ is strictly positive definite in both settings and we let λ_0=λ_min (H^∞).


There exists a constant c>0 such that |σ(0)|≤ c and for any z,z^'∈ℝ,

    |σ^(k)(z)-σ^(k)(z^')|≤ c|z-z^'|,

where k∈{0,1,2,3}. Moreover, σ(·) is analytic and is not a polynomial function.


Unlike the approach for gradient descent, <cit.> and <cit.> focus on the change of the Jacobian matrix for NGD, rather than focusing on the Gram matrix.  More precisely, they show that J(w) is stable with respect to w, where J(w) is the Jacobian matrix with weight vector w=(w_1^T,⋯, w_m^T)^T. However, this approach does not work for PINNs. Instead, we consider the stability of J(w) with respect to every single weight vector w_r.


Let R∈ (0,1], if w_1(0),⋯, w_m(0) are i.i.d. generated 𝒩(0,I), then with probability at least 1-P(δ,m,R) the following holds. For any set of weight vectors w_1,⋯,w_m ∈ℝ^d+2 that satsify for any r∈ [m], w_r-w_r(0)_2< R, then 

(1) when σ(·) is the ReLU^3 activation function, we have that

    J(w)-J(0)_2≤ CM√(R),
 
where C is a universal constant, M=2(d+2)log(2m(d+2)/δ) and 

    P(δ,m,R)=1-δ-n_1e^-mR;


(2) when σ(·) satisfies Assumption 5, we have that 

    J(w)-J(0)_2≤ CdR

for m≥log^2(1/δ), where C is a universal constant and P(δ,m,R)=δ.


With the stability of Jacobian matrix, we can derive the following main result.


Let L(k)=L(w(k)), then the following conclusions hold.

(1) When σ(·) is the ReLU^3 activation function, under Assumption 4, we set 

    m=Ω(d^12/λ_0^4log^6 (md/δ) log(n_1+n_2/δ))

 and η∈ (0,1), then with probability at least 1-δ over the random initialization for all k∈ℕ

    L(k)≤ (1-η)^k L(0).


(2) When σ(·) satisfies Assumption 5, under Assumption 4, we set

    m=Ω( d^6/λ_0^3log^2 (md/δ) log(n_1+n_2/δ) )

and η∈ (0,1), then with probability at least 1-δ over the random initialization for all k∈ℕ

    L(k)≤ (1-η)^k L(0).



In fact, NGD can enjoy a second-order convergence rate when σ(·) satisfies Assumption 5.

Under the setting of Theorem 3(2), in addition to (40), we also have that

    [ s(t+1); h(t+1) ]_2  ≤CB^4/√(mλ_0^3)[ s(t); h(t) ]_2^2,

where C is a universal constant and B=√(2(d+2)log(2m(d+2)/δ))+1.





§ CONCLUSION AND DISCUSSION

In this paper, we have improved the learning rate of gradient descent for both L^2 regression problems and PINNs, indicating that gradient descent enjoys better convergence. Furthermore, we demonstrate that natural gradient descent can find the global optima of two-layer PINNs with ReLU^3 or smooth activation functions for a class of second-order linear PDEs. Compared to gradient descent, natural gradient descent possesses faster convergence rate and the maximal learning rate is 𝒪(1). Despite this, natural gradient descent is quite expensive in terms of cumputation and memory in training neural networks. Therefore, several cost-effective variants have been proposed, such as K-FAC <cit.> and mini-batch natural gradient descent <cit.>. It would be interesting to investigate the convergence of these methods for PINNs. Additionally, generalizing the convergence analysis to deep neural networks is an important direction for future work.  


IEEEtran
	



§ APPENDIX
	



§ PROOF OF SECTION 2

Before the proofs, we first define the event 

    A_ir:={∃w: w-w_r(0)_2≤ R , I{w^T x_i≥ 0}≠ I{w_r(0)^T x_i≥ 0}}

for all i∈ [n]. 

Note that the event happens if and only if |w_r(0)^T x_i|<R, thus by the anti-concentration inequality of Gaussian distribution,  we have

    P(A_ir)=P_z∼𝒩(0,x_i_2^2) ( |z|<R)=P_z∼𝒩(0,1)( |z|<R/x_i_2)≤2R/√(2π)x_i_2≤2R/√(2π),

as x_i _2 ≥ 1 for all i∈ [n].



 §.§ Proof of Lemma 2


Recall that

    ∑_i=1^n∑_j=1^n |H(w)_ij-H(0)_ij|^2 
       =∑_i=1^n∑_j=1^n(x_i^Tx_j)^2(1/m∑_r=1^m(I{w_r^Tx_i≥ 0,w_r^Tx_j≥ 0}-I{w_r(0)^Tx_i≥ 0,w_r(0)^Tx_j≥ 0}) )^2.

To prove the measurability, we will show that I{w_r^Tx_i≥ 0,w_r^Tx_j≥ 0} can be approximated by the sequence {I{w̃_k^Tx_i≥ 0,w̃_k^Tx_j≥ 0}}_k ∈ℕ, where w̃_k is a rational vector in ℚ^d+1 for each k∈ℕ, with ℚ denoting the set of all rational numbers. Therefore, taking the supremum of w within the ball is equivalent to taking the supremum over all rational vectors within this ball, which implies that the random variable is measurable. 

We first focus on I{w_r^Tx_i≥ 0} and let w_r=(w_r1^T, w_r0)^T, x_i=(x_i1^T,1)^T with w_r0∈ℝ and w_r1,x_i1∈ℝ^d. For each k∈ℕ, we can take β_k ∈ℚ^d such that β_k-w_r1_2≤1/2k. Then choose t_k∈ℚ such that t∈ (w_r0+1/2k, w_r0+1/k] and let w̃_k=(β_k^T, t_k)^T. From the construction, we have

    I{w̃_k^Tx_i≥ 0}=I{β_k^Tx_i1+t_k≥ 0}=I{w_r^Tx_i+ t_k-w_r0+(β_k-w_r1)^Tx_i1≥ 0},

as w_r^Tx_i=w_r1^Tx_i1+w_r0. Define r_k=t_k-w_r0+(β_k-w_r1)^Tx_i1, then r_k>0 and r_k→ 0, as |(β_k-w_r1)^Tx_i1|≤β_k-w_r1_2≤1/2k and t-w_r0∈ (1/2k, 1/k]. Since the function u→ I{u≥ x} is right-continuous for any x∈ℝ. Thus I{w̃_k^Tx_i≥ 0}=I{w_k^Tx_i+r_k≥ 0}→ I{w_k^Tx_i≥ 0} as k→∞. Meanwhile, when k is large enough, w̃_k is in the small ball centered at w_r(0), i.e., w̃_k-w_r(0)_2<R. Similarly, we have that I{w̃_k^Tx_j≥ 0}→ I{w_k^Tx_j≥ 0} as k→∞. Thus, we can deduce that I{w̃_k^Tx_i≥ 0,w̃_k^Tx_j≥ 0}→ I{w_k^Tx_i≥ 0,w_k^Tx_j≥ 0} as k→∞. 

Therefore, 

    sup_c
    w_1-w_1(0)_2<R,
    ⋯
    w_m-w_m(0)_2<R ∑_i=1^n∑_j=1^n |H(w)_ij-H(0)_ij|^2=sup_c
    w_1-w_1(0)_2<R, w_1∈ℚ^d+1
    ⋯
    w_m-w_m(0)_2<R,w_m∈ℚ^d+1∑_i=1^n∑_j=1^n |H(w)_ij-H(0)_ij|^2,

which leads to the measurability.

From the definition of A_ir, i.e., (41), and the fact w_r-w_r(0)_2<R, we can deduce that 

    |I{w_r^Tx_i≥ 0}-I{w_r(0)^Tx_i≥ 0}|≤ I{A_ir}.

Thus, when both A_ir and A_jr do not happen, we have

    |I{w_r^Tx_i≥ 0,w_r^Tx_j≥ 0}-I{w_r(0)^Tx_i≥ 0,w_r(0)^Tx_j≥ 0}| 
       ≤ |I{w_r^Tx_i≥ 0}-I{w_r(0)^Tx_i≥ 0≥ 0}|+|I{w_r^Tx_j≥ 0}-I{w_r(0)^Tx_j≥ 0≥ 0}| 
       ≤ I{A_ir}+I{A_jr}=0,

which imples 

    |I{w_r^Tx_i≥ 0,w_r^Tx_j≥ 0}-I{w_r(0)^Tx_i≥ 0,w_r(0)^Tx_j≥ 0}|≤ I{A_ir A_jr}.

Therefore, combining (43) and (44) yields that

    sup_c
    w_1-w_1(0)_2<R,
    ⋯,
    w_m-w_m(0)_2<R ∑_i=1^n∑_j=1^n |H(w)_ij-H(0)_ij|^2 
       ≤ 4sup_c
    w_1-w_1(0)_2<R,
    ⋯,
    w_m-w_m(0)_2<R ∑_i=1^n∑_j=1^n (1/m∑_r=1^m(I{w_r^Tx_i≥ 0,w_r^Tx_j≥ 0}-I{w_r(0)^Tx_i≥ 0,w_r(0)^Tx_j≥ 0}) )^2
       ≤ 4∑_i=1^n∑_j=1^n (1/m∑_r=1^m I{A_ir A_jr})^2.

It remains only to bound the term 1/m∑_r=1^m I{A_ir A_jr}. Note that

    𝔼[I{A_ir A_jr}]≤ P(A_ir)+P(A_jr)≤4R/√(2π)

and then

    Var(I{A_ir A_jr}) ≤𝔼[I{A_ir A_jr}≤4R/√(2π).

Thus, applying Bernstein's inequality for the random variable I{A_ir A_jr}-𝔼[I{A_ir A_jr}] yields that with probability at least 1-e^-t, 

    1/m∑_r=1^m I{A_ir A_jr}≤4R/√(2π)+√(2t/m4R/√(2π))+ t/3m.

Choosing t=mR and taking a union bound for i,j∈ [n] yields that with probability at least 1-n^2e^-mR,

    1/m∑_r=1^m I{A_ir A_jr}≤ 4R

holds for all i,j∈ [n]. 

Plugging (46) into (45) leads to the conclusion.




 §.§ Proof of Lemma 3


First, we can decompose u_i(k+1)-u_i(k) as follows.

    u_i(k+1)-u_i(k)   =u_i(k+1)-u_i(k)-⟨∂ u_i(k) /∂w, w(k+1)-w(k)⟩ + ⟨∂ u_i(k) /∂w, w(k+1)-w(k)⟩
       := I_1^i(k)+I_2^i(k).

For I_2^i(k), we have

    I_2^i(k)   =⟨∂ u_i(k) /∂w, w(k+1)-w(k)⟩
       = ∑_r=1^m ⟨∂ u_i(k) /∂w_r, w_r(k+1)-w_r(k)⟩
       =∑_r=1^m ⟨∂ u_i(k) /∂w_r,-η∂ L(k)/∂w_r⟩
       = ∑_r=1^m (-η) ∑_j=1^n (u_j-y_j) ⟨∂ u_i(k) /∂w_r, ∂ u_j(k) /∂w_r⟩
       = η∑_j=1^n (∑_r=1^m  ⟨∂ u_i(k) /∂w_r, ∂ u_j(k) /∂w_r⟩) (y_j-u_j) 
       =η [H(k)]_i (y-u(k)),

where [H(k)]_i denotes the i-row of the matrix H(k).
	
Thus,

    u_i(k+1)-u_i(k)=η [H(k)]_i (y-u(k))+I_1^i(k)

and then

    (y_i-u_i(k))-(y_i-u_i(k+1)) =η [H(k)]_i (y-u(k))+I_1^i(k) ,

which yields

    (y-u(k))-(y-u(k+1)) =ηH(k) (y-u(k))+I_1(k).

A simple transformation leads to the conclusion

    y-u(k+1)=(I-ηH(k))(y-u(k))-I_1(k).





 §.§ Proof of Theorem 1


The proof follows a similar induction to that in <cit.>. Our induction hypothesis is the following condition.

At the t-th iteration, we have 

    y-u(t)_2^2≤(1-ηλ_0/2)^t y-u(0)_2^2.



Suppose that Condition 1 holds for t=0,⋯, k, then Corollary 4.1 in <cit.> implies that for every r∈ [m] and t=0,⋯,k

    w_r(t+1)-w_r(0)_2≤4√(n)y-u(0)_2 /√(m)λ_0 :=R^'.

In the following, we aim to show that Condition 1 also holds for t=k+1, thus the conclusion of Theorem 1 holds directly.

Let S_i={r∈ [m]: I{A_ir}=0} and S_i^⊥=[m]\ S_i.

Recall that 

    y-u(k+1)=(I-ηH(k))(y-u(k))-I_1(k).


As Condition 1 holds for t=0,⋯,k, from (48), we have that for any r∈ [m],

    w_r(k)-w_r(0)_2≤ R^'.


By taking R=λ_0/32n in Lemma 2 and R^'≤ R, we have that 

    H(k)-H(0)_2 ≤λ_0/4,

which implies that λ_min(H(k))≥λ_min(H(0))-λ_0/4≥λ_0/2 and 

    H(k)_2≤H(0)_2+λ_0/4≤H^∞_2+λ_0/2≤3/2H^∞_2.


Thus, when η≤2/3H^∞_2, I-ηH(k) is positive definite and meanwhile I-ηH(k)_2≤ 1-ηλ_0/2. Combining with the recursion formula (10) yields that

    y-u(k+1)_2^2    = (I-ηH(k))(y-u(k))-I_1(k)_2^2 
       = (I-ηH(k))(y-u(k))_2^2 + I_1(k)_2^2 -2⟨ ( I-ηH(k))(y-u(k)),   I_1(k) ⟩
       ≤ (1-ηλ_0/2)^2 y-u(k)_2^2 + I_1(k)_2^2 +2 I_1(k)_2  (I-ηH(k))(y-u(k))_2 
       ≤ (1-ηλ_0/2)^2 y-u(k)_2^2 + I_1(k)_2^2 +2(1-ηλ_0/2) I_1(k)_2 y-u(k)_2 .

Then, it remains only to bound the term I_1(k)_2. 

Recall that for i∈ [n],

    I_1^i(k)=∑_r=1^m a_r/√(m)[ σ(w_r(k+1)^T x_i)-σ(w_r(k)^T x_i)- (w_r(k+1)-w_r(k))^T x_i I{w_r(k)^Tx_i ≥ 0}].

Note that for r ∈ S_i, we can deduce that I{w_r(k+1)^T x_i ≥ 0}= I{w_r(k)^T x_i ≥ 0}, as w_r(k)-w_r(0)_2≤ R^'≤ R and w_r(k)-w_r(0)_2≤ R^'≤ R. Thus, for r ∈ S_i, we have 

    σ(w_r(k+1)^T x_i)-σ(w_r(k)^T x_i)- (w_r(k+1)-w_r(k))^T x_i I{w_r(k)^Tx_i ≥ 0}
       =[(w_r(k+1)^T xi)I{w_r(k+1)^T x_i≥ 0} -(w_r(k)^T x_i)I{w_r(k)^T x_i≥ 0}] 
          - (w_r(k+1)-w_r(k))^T x_i I{w_r(k)^Tx_i ≥ 0}
       = [(w_r(k+1)^T x_i)I{w_r(k)^T x_i≥ 0} -(w_r(k)^T x_i)I{w_r(k)^T x_i≥ 0}] 
          - (w_r(k+1)-w_r(k))^T x_i I{w_r(k)^Tx_i ≥ 0}
       =0.

It implies that 

    I_1^i(k)=∑_r∈ S_i^⊥a_r/√(m)[ σ(w_r(k+1)^T x_i)-σ(w_r(k)^T x_i)- (w_r(k+1)-w_r(k))^T x_i I{w_r(k)^Tx_i ≥ 0}].


Therefore,

    |I_1^i(k)|   ≤1/√(m)∑_r∈ S_i^⊥ 2√(2)w_r(k+1)-w_r(k)_2 
       =2√(2)/√(m)∑_r∈ S_i^⊥-η∂ L(k)/∂w_r_2 
       ≤4 /√(m)∑_r∈ S_i^⊥η√(n)y-u(k)_2/√(m)
       = 4η√(n)y-u(k)_2/m∑_r=1^m I{r∈ S_i^⊥},

where the second inequality follows from that

    ∂ L(k)/∂w_r   = ∑_i=1^n(u_i(k)-y_i)∂ u_i(k)/∂w_r
       = ∑_i=1^n(u_i(k)-y_i) a_r/√(m)I{w_r(k)^Tx_i≥ 0}x_i
	
and then 

    ∂ L(k)/∂w_r_2≤√(n)y-u(k)_2/√(m),

as x_i_2≤√(2).

Since S_i={r∈ [m]: I{A_ir}=0}, it follows that I{r∈ S_i^⊥}=I{A_ir}, then applying Berstein's inequality yields that with probability at least 1-ne^-mR,

    1/m∑_r=1^m I{r∈ S_i^⊥}≤ 4R,

holds for all i∈ [n].

Thus from (50) and (51), we have  

    I_1(k)_2    = √(∑_i=1^n  |I_1^i(k)|^2 )
       ≤ 4η√(n)y-u(k)_2 √(∑_i=1^n (1/m∑_r=1^m I{r∈ S_i^⊥})^2 )
       ≤ 8η n Ry-u(k)_2
       = ηλ_0/8y-u(k)_2,

where the last inequality is due to R=λ_0/64n.

Plugging (52) into (49) yields that

    y-u(k+1)_2^2    ≤(1-ηλ_0/2)^2 y-u(k)_2^2 + I_1(k)_2^2 +2(1-ηλ_0/2) I_1(k)_2 y-u(k)_2 
       ≤[(1-ηλ_0/2)^2+η^2λ_0^2/64 +2(1-ηλ_0/2)ηλ_0/8] y-u(k)_2^2 
       ≤[1-ηλ_0+η^2λ_0^2/4+η^2λ_0^2/64 +ηλ_0/4] y-u(k)_2^2 
       ≤ (1-ηλ_0/2)y-u(k)_2^2,

where the last inequality follows from the fact that

    η≤2/3H^∞_2≤2/3λ_0,

which imples that 

    η^2λ_0^2/4+η^2λ_0^2/64≤ηλ_0/4.


Finally, we need consider the requirement for m. First, Lemma 1 shows that m=Ω( n^2/λ_0^2log(n/δ) ). Second, from R^'≤ R=λ_0/32n and R^'=4√(n)y-u(0)_2/√(m)λ_0, we know

    m=Ω(    n^3 y-u(0)_2^2 /λ_0^4).

Then from the estimation in Lemma 10 for the initial prediction y-u(0)_2, we obtain that

    m=Ω(    n^4  /λ_0^4log(n/δ) ).





§ PROOF OF SECTION 3

Before the proofs, we first recall that 

    ∂ s_p(w)/∂w_r=a_r/√(mn_1)(σ^”(w_r^Tx_p)w_r0x_p+σ^'(w_r^T x_p) [     1; 0_d+1 ]-σ^”'(w_r^T x_p)w_r1_2^2 x_p-2σ^”(w_r^T x_p) [    0; w_r1 ])

and

    ∂ h_j(w)/∂w_r=a_r /√(mn_2)σ^'(w_r^T y_j)y_j.





 §.§ Proof of Lemma 4


In the following, we aim to bound H(0)-H^∞_F, as H(0)-H^∞_2≤H(0)-H^∞_F. Note that the entries of H(0)-H^∞ have three forms as follows. 

    ∑_r=1^m ⟨∂ s_i(w)/∂w_r, ∂ s_j(w)/∂w_r⟩ -𝔼_w∑_r=1^m ⟨∂ s_i(w)/∂w_r, ∂ s_j(w)/∂w_r⟩,


    ∑_r=1^m ⟨∂ s_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩ -𝔼_w∑_r=1^m ⟨∂ s_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩

and

    ∑_r=1^m ⟨∂ h_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩ -𝔼_w∑_r=1^m ⟨∂ h_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩.

For the first form (55), to simplify the analysis, we let 

    Z_r(i)   =σ^”(w_r(0)^Tx_i)w_r0(0) x_i+σ^'(w_r(0)^T x_i) [     1; 0_d+1 ]
          	-σ^”'(w_r(0)^T x_p)w_r1(0)_2^2 x_p-2σ^”(w_r(0)^T x_i) [       0; w_r1(0) ]

and

    X_r(ij)=⟨Z_r(i),Z_r(j) ⟩ ,

then 

    ∑_r=1^m ⟨∂ s_p(w)/∂w_r, ∂ s_j(w)/∂w_r⟩ -𝔼_w∑_r=1^m ⟨∂ s_p(w)/∂w_r, ∂ s_j(w)/∂w_r⟩=1/n_1 m∑_r=1^m (X_r(ij)-𝔼X_r(ij)).

Note that |X_r(ij)|≲ 1+w_r(0)_2^4, thus

    X_r(ij)_ψ_1/2≲ 1+ w_r(0)_2^4 _ψ_1/2≲ 1+ w_r(0)_2^2_ψ_1^2 ≲ d^2.

For the centered random variable, the property of ψ_1/2 quasi-norm implies that

    X_r(ij)-𝔼[X_r(ij)]_ψ_1/2≲X_r(ij)_ψ_1/2+𝔼[X_r(ij)]_ψ_1/2≲ d^2.
 
Therefore, applying Lemma 8 yields that with probability at least 1-δ,

    |∑_r=1^m 1/m(X_r(ij)-𝔼[X_r(ij)]) | ≲d^2/√(m)√(log(1/δ)) +d^2/m(log(1/δ))^2,

which means

    |∑_r=1^m ⟨∂ s_i(w)/∂w_r, ∂ s_j(w)/∂w_r⟩ -𝔼_w∑_r=1^m ⟨∂ s_i(w)/∂w_r, ∂ s_j(w)/∂ww_r⟩|≲d^2/n_1√(m)√(log(1/δ)) +d^2/n_1m(log(1/δ))^2.



Similarly, for the second form (56) and third form (57), we can deduce that

    ⟨∂ s_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩ -𝔼_w⟨∂ s_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩_ψ_1/2≲d^2/√(n_1n_2)m

and 

    ⟨∂ h_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩ -𝔼_w⟨∂ h_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩_ψ_1/2≲d^2/n_2m.



Thus applying Lemma 8 yields that with probability at least 1-δ,

    |∑_r=1^m ⟨∂ s_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩ -𝔼_w∑_r=1^m ⟨∂ s_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩|≲d^2/√(n_1n_2)√(m)√(log(1/δ)) +d^2/√(n_1n_2)mlog(1/δ)

and with probability at least 1-δ,

    |∑_r=1^m ⟨∂ h_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩ -𝔼_w∑_r=1^m ⟨∂ h_i(w)/∂w_r, ∂ h_j(w)/∂w_r⟩|≲d^2/n_2√(m)√(log(1/δ)) +d^2/n_2mlog(1/δ) .



Combining (58), (59) and (60), we can deduce that with probability at least 1-δ,

    H(0)-H^∞_2^2 
       ≤H(0)-H^∞_F^2 
       ≲d^4/mlog(n_1+n_2/δ)+d^4/m^2(log(n_1+n_2/δ))^4 
       ≲d^4/mlog(n_1+n_2/δ).

Thus when √(d^4/mlog(n_1+n_2/δ))≲λ_0/4, i.e.,

    m=Ω(d^4/λ_0^2log(n_1+n_2/δ)),

we have λ_min(H(0))≥3/4λ_0.





 §.§ Proof of Lemma 5


We first reformulate the term ∂ s_p(k)/∂w_r in (53) as follows.

    ∂ s_p(k)/∂w_r
    = σ^”(w_r(k)^Tx_p) [          w_r0(k)x_p0; w_r0(k)x_p1-2w_r1(k) ]+σ^'(w_r(k)^Tx_p) [     1; 0_d+1 ]-σ^”'(w_r(k)^Tx_p)w_r1(k)_2^2 x_p.

Similar to Lemma 4, it suffices to bound H(w)-H(0)_F, which can in turn allows us to bound each entry of H(w)-H(0). 

For i∈ [n_1] and j∈ [n_1], we have

    H_ij(w)   =∑_r=1^m ⟨∂ s_i/∂w_r, ∂ s_j/∂w_r⟩
       = 1/n_1m∑_r=1^m ⟨σ^”(w_r^Tx_i) [       w_r0x_i0; w_r0x_i1-2w_r1 ]+σ^'(w_r^Tx_i) [     1; 0_d+1 ]-σ^”'(w_r^Tx_i)w_r1_2^2 x_i, .
           . σ^”(w_r^Tx_j) [       w_r0x_j0; w_r0x_j1-2w_r1 ]+σ^'(w_r^Tx_j) [     1; 0_d+1 ]-σ^”'(w_r^Tx_j)w_r1_2^2 x_j⟩

After expanding the inner product term, we can find that although it has nine terms, it only consists of six classes. For simplicity, we use the following six symbols to represent the corresponding classes.

    σ^”σ^”,σ^”σ^', σ^'σ^', σ^”'σ^”, σ^”'σ^', σ^”'σ^”'.

For instance, σ^”σ^' represents

    ⟨σ^”(w_r^Tx_i) [       w_r0x_i0; w_r0x_i1-2w_r1 ] ,  σ^'(w_r^Tx_j) [     1; 0_d+1 ]⟩ , ⟨σ^'(w_r^Tx_i) [     1; 0_d+1 ], σ^”(w_r^Tx_j) [       w_r0x_j0; w_r0x_j1-2w_r1 ]⟩ .

In fact, when bounding the corresponding terms for H_ij(w)-H_ij(0), the first four classes can be grouped into one category. They are of the form f_1(w)f_2(w)f_3(w)f_4(w), where for each 1≤ i≤ 4, f_i(w) is Lipschitz continuous with respect to ·_2 and |f_i(w)|≲w_2 (Note that σ^'(·)=(σ^”(·))^2). On the other hand, when w_1-w_2_2≤ R≤ 1, we can deduce that 

    |f_1(w_1)f_2(w_1)f_3(w_1)f_4(w_1)-f_1(w_2)f_2(w_2)f_3(w_2)f_4(w_2)|≲ R (w_1_2^3+1).

Thus, for the terms in H_ij(w)-H_ij(0) that belong to the first four classes, we can deduce that they are less than CR(w_r(0)_2^3+1), where C is a universal constant.

For the classes σ^”'σ^” and σ^”'σ^', they are both involving σ^”' that is not Lipschitz continuous. To make it precise, we write the class σ^”'σ^” explicitly as follows.

    σ^”(w_r^Tx_i)σ^”'(w_r^Tx_j) w_r1_2^2 [       w_r0x_i0; w_r0x_i1-2w_r1 ] ^Tx_j.

Note that when w_r-w_r(0)_2<R, we have that

    |σ^”'(w_r^Tx_j)-σ^”'(w_r(0)^Tx_j)|=|I{w_r^Tx_j≥ 0}-I{w_r(0)^Tx_j≥ 0}|≤ I{A_jr},

where the definition of A_jr is same as (44) for the regression problems.

Thus, we can deduce that for the terms in H_ij(w)-H_ij(0) that belong to the classes σ^”'σ^” and σ^”'σ^', they are less than

    C[ (I{A_ir}+I{A_jr})w_r(0)_2^3 + R (w_r(0)_2^3+1)],

where C is a universal constant.

Similarly, for the last class σ^”'σ^”' that are of the form 
    σ^”'(w_r^Tx_i)σ^”'(w_r^Tx_j)w_r1_2^4x_i^Tx_j,

we can deduce that

    |σ^”'(w_r^Tx_i)σ^”'(w_r^Tx_j)w_r1_2^4x_i^Tx_j-σ^”'(w_r(0)^Tx_i)σ^”'(w_r(0)^Tx_j)w_r1(0)_2^4x_i^Tx_j| 
       ≲ I{A_ir A_jr}w_r(0)_2^4+Rw_r(0)_2^3+R.


Combining the upper bounds for the six classes, we have that

    |H_ij(w)-H_ij(0)|   ≲1/n_1[1/m (R∑_r=1^m w_r(0)_2^3)+1/m∑_r=1^m(I{A_ir}+I{A_jr}) w_r(0)_2^4 +R]
       ≲1/n_1[1/m (R∑_r=1^m w_r(0)_2^4)+1/m∑_r=1^m(I{A_ir}+I{A_jr}) w_r(0)_2^4 +R],

where the last inequality follows from that w_r(0)_2^3≲w_r(0)_2^4+1 due to Young's inequality for products.

Now, we focus on the term 1/m∑_r=1^m I{A_ir}w_r(0)_2^4. 

Since

    P(|w_ri(0)|^2≥ 2log (2/δ)) ≤δ

and then 

    P(w_r(0)_2^2≥ 2(d+2)log (2(d+2)/δ)) ≤δ,

we obtain that

    P(∃ r∈ [m],w_r(0)_2^2≥ 2(d+2)log2m(d+2)/δ) ≤δ.
 
Let M=2(d+2)log (2m(d+2)/δ), then 

    1/m∑_r=1^m I{A_ir}w_r(0)_2^4 
       =1/m∑_r=1^m I{A_ir}w_r(0)_2^4 I{w_r(0)_2^2 ≤ M} + 1/m∑_r=1^m I{A_ir}w_r(0)_2^4 I{w_r(0)_2^2 > M}
       ≤M^2/m∑_r=1^m I{A_ir}+1/m∑_r=1^m w_r(0)_2^4 I{w_r(0)_2^2 > M}.

Applying Bernstein's inequality for the first term yields that with probability at least 1-e^-mR,

    1/m∑_r=1^m I{A_ir}≤ 4R.

Moreover, with probability at least 1-δ, I{w_r(0)_2^2 > M}=0 holds for all r∈ [m].

Thus from (61), with probability at least 1-δ-n_1e^-mR, we have that for any i∈ [n_1] and j∈[n_1],

    |H_ij(w)-H_ij(0)|   ≲1/n_1[ RM^2+RM^2+R ] 
       ≲1/n_1 M^2R.


For i∈ [n_1], j∈ [n_1+2,n_2] and i∈ [n_1+1,n_2], j∈ [n_2], from the form of ∂ h_j/∂w_r, i.e.,

    ∂ h_j/∂w_r=a_r/√(n_2m)σ^'(w_r^Ty_j) y_j,

we can obtain similar results for the terms ⟨∂ s_i/∂w,∂ h_j/∂w⟩ and ⟨∂ h_i/∂w,∂ h_j/∂w⟩.

With all results above, we have that with probability at least 1-δ-(n_1+n_2)e^-mR,

    H(w)-H(0)_F≲ M^2R.





 §.§ Proof of Lemma 6


Similar to the proof of Lemma 3, we have

    s_p(k+1)-s_p(k)   =[s_p(k+1)-s_p(k)-⟨∂ s_p(k) /∂w, w(k+1)-w(k)⟩] + ⟨∂ s_p(k) /∂w, w(k+1)-w(k)⟩
       := I_1^p(k)+I_2^p(k).

For the second term I_2^p(k), we have that

    I_2^p(k)   = ⟨∂ s_p(k) /∂w, w(k+1)-w(k)⟩
       = ⟨∂ s_p(k) /∂w, -η∂ L(k)/∂w⟩
       = -∑_r=1^m η⟨∂ s_p(k) /∂w_r,  ∂ L(k)/∂w_r⟩
       = -∑_r=1^m η⟨∂ s_p(k) /∂w_r,  ∑_t=1^n_1s_t(k)∂ s_t(k) /∂w_r+ λ∑_j=1^n_2h_j(k)∂ h_j(k) /∂w_r⟩
       =-η [H(k)]_p [ s(k); h(k) ],

where [H(k)]_p denotes the p-row of H(k).

Similarly, for h(k), we have

    h_j(k+1)-h_j(k)   =[h_j(k+1)-h_j(k)-⟨∂ h_j(k) /∂w, w(k+1)-w(k)⟩] + ⟨∂ h_j(k) /∂w, w(k+1)-w(k)⟩
       := I_1^n_1+j(k)+I_2^n_1+j(k)

and

    I_2^n_1+j(k)=-η [H(k)]_n_1+j[ s(k); h(k) ].



Combining (63), (64), (65) and (66) yields that

    [ s(k+1); h(k+1) ]-[ s(k); h(k) ]   = I_1(k)+I_2(k) 
       =I_1(k) -ηH(k) [ s(k); h(k) ].

A simple transformation directly leads to

    [ s(k+1); h(k+1) ]=(I-ηH(k))[ s(k); h(k) ] +I_1(k).





 §.§ Proof of Theorem 2


The proof strategy is similar to that for Theorem 1. Our induction hypothesis is the following boundedness of the hidden weights and convergence rate of the empirical loss.


At the t-th iteration, we have that for each r∈ [m], w_r(t)_2≤ B and 

    L(t)≤(1-ηλ_0/2)^t L(0),

where B=√(2(d+2)log(2m(d+2)/δ) )+1.


From (62), we know that with probability at least 1-δ, w_r(0)_2≤√(2(d+2)log(2m(d+2)/δ) ) holds for any r∈ [m]. Thus, if we can prove that w_r(t) is closed enough to w_r(0), then w_r(t) _2 ≤ B. 


If Condition 2 holds for t=0,⋯, k, then we have for every r∈ [m],

    w_r(k+1)-w_r(0) _2≤CB^2 √(L(0))/√(m)λ_0:=R^',

where C is a universal constant.

Corollary 2 implies that when m is large enough, we have w_r(k+1)-w_r(0) _2≤ 1 and then w_r(k+1)_2 ≤ B. Thus, in induction, we only need to prove that (67) also holds for t=k+1, which relies on the recursion formula (26).

Recall that

    [ s(k+1); h(k+1) ]=(I-ηH(k))[ s(k); h(k) ] +I_1(k).

From Corollary 2 and Lemma 5, taking CM^2R < λ_0/4 in (24) and R^'≤ R in (68) yields that λ_min(H(k))≥λ_0/2 and

    H(k)_2 ≤H(0)_2+λ_0/4≤H^∞_2+λ_0/2≤3/2H^∞_2.

Therefore, if we take η≤2/31/H^∞_2, then I-ηH(k) is positive definite and I-ηH(k)_2 ≤ 1-ηλ_0/2.

Combining these facts with the recursion formula, we have that 

    [ s(k+1); h(k+1) ]_2^2 
       =(I-ηH(k))[ s(k); h(k) ]_2^2 +I_1(k) _2^2+2⟨ (I-ηH(k))[ s(k); h(k) ] ,I_1(k)⟩
       ≤(1-ηλ_0/2)^2 [ s(k); h(k) ]_2^2 + I_1(k) _2^2+2(1-ηλ_0/2) [ s(k); h(k) ]_2 I_1(k) _2.

Thus, it remains to bound I_1(k) _2.

For I_1(k), recall that  I_1(k)=(I_1^1(k), ⋯, I_1^n_1(k), I_1^n_1+1(k),⋯, I_1^n_1+n_2(k))^T ∈ℝ^n_1+n_2 and for p∈ [n_1],

    I_1^p(k)=s_p(k+1)-s_p(k)-⟨∂ s_p(k) /∂w, w(k+1)-w(k)⟩,

for j∈ [n_2],

    I_1^n_1+j(k)=h_j(k+1)-h_j(k)-⟨∂ h_j(k) /∂w, w(k+1)-w(k)⟩.
 

Recall that

    s_p(k)=1/√(n_1)(1/√(m)∑_r=1^m a_rσ^'(w_r(k)^Tx_p)w_r0(k)-a_rσ^”(w_r(k)^Tx_p )w_r1(k)_2^2-f(x_p) )

and

    ∂ s_p(k)/∂w_r   = a_r/√(n_1m) [σ^”(w_r(k)^T x_p)w_r0(k)x_p+σ^'(w_r(k)^T x_p) [     1; 0_d+2 ] -σ^”'(w_r(k)^T x_p)w_r1(k)_2^2 x_p 
          -2σ^”(w_r(k)^T x_p)[       0; w_r1(k) ] ].


Define χ_pr^1(k):=σ^'(w_r(k)^Tx_p)w_r0(k) and χ_pr^2(k):=σ^”(w_r(k)^Tx_p )w_r1(k)_2^2, i.e.,  χ_pr^1(k) and χ_pr^2(k) are related to the operators ∂ u/∂ t and Δ u respectively. 

Then define 

    χ̂_pr^1(k)=χ_pr^1(k+1)-χ_pr^1(k)-⟨∂χ_pr^1(k) /∂w_r, w_r(k+1)-w_r(k)⟩

and

    χ̂_pr^2(k)=χ_pr^2(k+1)-χ_pr^2(k)-⟨∂χ_pr^2(k) /∂w_r, w_r(k+1)-w_r(k)⟩.

At this time, we have

    I_1^p(k)=1/√(n_1m)∑_r=1^m a_r[χ̂_pr^1(k)-χ̂_pr^2(k) ].

The purpose of defining them in this way is to enable us to handle the terms related to the operators ∂ u/∂ t and Δ u separately. 

Similar to that for regression problems with ReLU activation function, we first define some events. For p∈ [n_1], define 

    A_p,r={∃w: w-w_r(0)_2 ≤ R, I{w^T x_p≥ 0}≠ I{w_r(0)^T x_p≥ 0}}

and let S_p^⊥=[n_1]\ S_p.

In the following, we are going to show that |χ̂_pr^1(k)|=𝒪(w_r(k+1)-w_r(k)_2^2) for every r∈ [m] and |χ̂_pr^2(k)|=𝒪(w_r(k+1)-w_r(k)_2^2) for r∈ S_p, |χ̂_pr^2(k)|=𝒪(w_r(k+1)-w_r(k)_2) for r∈ S_p^⊥. Thus, we can prove that I_1_2=𝒪(√(L(k))/√(m)). Then combining with (69) leads to a milder requirement for η.

For χ̂_pr^1(k), we have that

    χ̂_pr^1(k)   = σ^'(w_r(k+1)^T x_p)w_r0(k+1)-σ^'(w_r(k)^Tx_p)w_r0(k)
          -⟨w_r(k+1)-w_r(k), x_p ⟩σ^”(w_r(k)^T x_p)w_r0(k)-(w_r0(k+1)-w_r0(k)) σ^'(w_r(k)^T x_p) 
       = (σ^'(w_r(k+1)^Tx_p)-σ^'(w_r(k)^Tx_p))w_r0(k+1)
    -⟨w_r(k+1)-w_r(k), x_p ⟩σ^”(w_r(k)^T x_p)w_r0(k).

From the mean value theorem, we can deduce that there exists ζ(k) ∈ℝ such that

    σ^'(w_r(k+1)^Tx_p)-σ^'(w_r(k)^Tx_p)=σ^”(ζ(k))⟨w_r(k+1)-w_r(k), x_p ⟩

and

    |σ^”(ζ(k))- σ^”(w_r(k)^Tx_p)|   ≤ |ζ(k)-w_r(k)^Tx_p| 
       ≤√(2)w_r(k+1)-w_r(k)_2.

Then, for χ̂_pr^1(k), we can rewrite it as follows.

    χ̂_pr^1(k)   = σ^”(ζ(k))⟨w_r(k+1)-w_r(k), x_p ⟩ w_r0(k+1)
    -⟨w_r(k+1)-w_r(k), x_p ⟩σ^”(w_r(k)^T x_p)w_r0(k)
       =[(σ^”(ζ(k))-σ^”(w_r(k)^Tx_p) )⟨w_r(k+1)-w_r(k), x_p ⟩ w_r0(k+1) ] 
          +[⟨w_r(k+1)-w_r(k), x_p ⟩σ^”(w_r(k)^T x_p)(w_r0(k+1)-w_r0(k)) ].

It implies that 

    |χ̂_pr^1(k)|≲ Bw_r(k+1)-w_r(k)_2^2.


For χ̂_pr^2(k), we write it as follows explicitly.

    χ̂_pr^2(k)   = σ^”(w_r(k+1)^T x_p)w_r1(k+1)_2^2 -σ^”(w_r(k)^T x_p)w_r1(k)_2^2 
          -⟨w_r(k+1)-w_r(k), x_p ⟩σ^”'(w_r(k)^T x_p)w_r1(k)_2^2 
          -2⟨w_r1(k+1)-w_r1(k),w_r1(k) ⟩σ^”(w_r(k)^T x_p) .

Note that for the term σ^”(w_r(k)^T w_p)w_r1(k)_2^2, we can rewrite it as follows.

    σ^”(w_r(k)^T x_p)w_r1(k)_2^2  
       = σ^”(w_r(k)^T x_p)w_r1(k)-w_r1(k+1)+w_r1(k+1)_2^2 
       = σ^”(w_r(k)^T x_p)[w_r1(k)-w_r1(k+1)_2^2+w_r1(k+1)_2^2-2⟨w_r1(k+1)-w_r1(k),w_r1(k+1) ⟩ ],

where the first term σ^”(w_r(k)^T x_p)w_r1(k)-w_r1(k+1)_2^2=𝒪(Bw_r(k+1)-w_r(k)_2^2).  

plugging (71) into (70) yields that 

    χ̂_pr^2(k)   = [σ^”(w_r(k+1)^T x_p)-σ^”(w_r(k)^T x_p)]w_r1(k+1)_2^2 
          -⟨w_r(k+1)-w_r(k), x_p ⟩σ^”'(w_r(k)^T x_p)w_r1(k)_2^2 
          +2⟨w_r1(k+1)-w_r1(k),w_r1(k+1)-w_r1(k) ⟩σ^”(w_r(k)^T x_p) +𝒪(Bw_r(k+1)-w_r(k)_2^2)
       = [σ^”(w_r(k+1)^T x_p)-σ^”(w_r(k)^T x_p)-⟨w_r(k+1)-w_r(k), x_p ⟩σ^”'(w_r(k)^T x_p)]w_r1(k+1)_2^2 
          +⟨w_r(k+1)-w_r(k), x_p ⟩σ^”'(w_r(k)^T x_p)(w_r1(k+1)_2^2-w_r1(k)_2^2) 
          +𝒪(Bw_r(k+1)-w_r(k)_2^2)
       = [σ^”(w_r(k+1)^T x_p)-σ^”(w_r(k)^T x_p)-⟨w_r(k+1)-w_r(k), x_p ⟩σ^”'(w_r(k)^T x_p)]w_r1(k+1)_2^2 
          +𝒪(Bw_r(k+1)-w_r(k)_2^2).

Thus, we only need to consider the term 

    σ^”(w_r(k+1)^T x_p)-σ^”(w_r(k)^T x_p)-⟨w_r(k+1)-w_r(k), x_p ⟩σ^”'(w_r(k)^T x_p).


For r∈ S_p, as w_r(k+1)-w_r(0)_2≤ R, w_r(k)-w_r(0)_2≤ R, we have that  I{w_r(k+1)^T x_p≥ 0}=I{w_r(k)^T x_p≥ 0}, which yields that 

    σ^”(w_r(k+1)^T x_p)-σ^”(w_r(k)^T x_p)-⟨w_r(k+1)-w_r(k), x_p ⟩σ^”'(w_r(k)^T x_p) 
       = [(w_r(k+1)^T x_p)I{w_r(k+1)^T x_p≥ 0}-(w_r(k)^T x_p)I{w_r(k)^T x_p≥ 0}] 
          -⟨w_r(k+1)-w_r(k), x_p ⟩ I{w_r(k)^T x_p≥ 0}
       =[(w_r(k+1)^T x_p)I{w_r(k)^T x_p≥ 0}-(w_r(k)^T x_p)I{w_r(k)^T x_p≥ 0}]
          -⟨w_r(k+1)-w_r(k), x_p ⟩ I{w_r(k)^T x_p≥ 0}
       =0.


For r∈ S_p^⊥, the Lipschitz continuity of σ^” implies that 

    σ^”(w_r(k+1)^T x_p)-σ^”(w_r(k)^T x_p)-⟨w_r(k+1)-w_r(k), x_p ⟩σ^”'(w_r(k)^T x_p)=𝒪(w_r(k+1)-w_r(k)_2).


Combining (72), (73) and (74), we can deduce that for r∈ S_p,

    |χ̂_pr^2(k)|≲ Bw_r(k+1)-w_r(k)_2^2,

for r∈ S_p^⊥,

    |χ̂_pr^2(k)|≲ Bw_r(k+1)-w_r(k)_2^2+B^2w_r(k+1)-w_r(k)_2.


With the estimations for χ̂_pr^1(k) and χ̂_pr^2(k),  we have 

    |I_1^p(k)|   ≤1/√(n_1m)∑_r=1^m (|χ̂_pr^1(k)|+|χ̂_pr^2(k)|) 
       ≲1/√(n_1m)∑_r=1^m Bw_r(k+1)-w_r(k)_2^2 +1/√(n_1m)∑_r∈ S_p^⊥ B^2w_r(k+1)-w_r(k)_2.


For j∈ [n_2], we consider I_1^n_1+j(k), which can be written as follows.

    I_1^n_1+j(k)	   =h_j(k+1)-h_j(k)-⟨w(k+1)-w(k), ∂ h_j(k)/∂w⟩
       = ∑_r=1^m a_r/√(n_2m)[σ(w_r(k+1)^Ty_j)-σ(w_r(k)^T y_j) -⟨w_r(k+1)-w_r(k), y_j⟩σ^'(w_r(k)^Ty_j)].


From the mean value theorem, we have that there exists ζ(k)∈ℝ such that

    σ(w_r(k+1)^Ty_j)-σ(w_r(k)^T y_j)=σ^'(ζ(k))⟨w_r(k+1)-w_r(k), y_j⟩

and

    |σ^'(ζ(k))-σ^'(w_r(k)^Ty_j)|   ≤ 2B|ζ(k)-w_r(k)^Ty_j| 
       ≤ 2√(2) B w_r(k+1)-w_r(k)_2.

Thus, 

    |σ(w_r(k+1)^Ty_j)-σ(w_r(k)^T y_j) -⟨w_r(k+1)-w_r(k), y_j⟩σ^'(w_r(k)^Ty_j)| 
       =| σ^'(ζ(k))⟨w_r(k+1)-w_r(k), y_j⟩ -σ(w_r(k)^T y_j) -⟨w_r(k+1)-w_r(k), y_j⟩σ^'(w_r(k)^Ty_j)| 
       =|(σ^'(ζ(k))-σ^'(w_r(k)^Ty_j))⟨w_r(k+1)-w_r(k), y_j⟩|
       ≲ Bw_r(k+1)-w_r(k)_2.

Therefore, for j∈ [n_2],

    |I_1^n_1+j(k)|≲B/√(n_2m)∑_r=1^m w_r(k+1)-w_r(k)_2^2.

From the updating rule of gradient descent, we can deduce that for every r∈ [m],

    w_r(k+1)-w_r(k)_2= -η∂ L(k)/∂w_r _2 ≲η B^2/√(m)√(L(k)).

Plugging (77) into (75) and (76), we can deduce that

    |I_1^p(k)|   ≲B/√(n_1m)∑_r=1^m w_r(k+1)-w_r(k)_2^2 + B^2/√(n_1m)∑_r∈ S_p^⊥w_r(k+1)-w_r(k)_2 
       ≲B/√(n_1m)∑_r=1^m η^2B^4/m L(k) +B^2/√(n_1m)∑_r∈ S_p^⊥η B^2/√(m)√(L(k))
       = η^2B^5 L(k)/√(n_1m) +η B^4√(L(k))/√(n_1)1/m∑_r=1^m I{r∈ S_p^⊥}
       ≤η^2B^5 √(L(0))√(L(k))/√(n_1m) +η B^4√(L(k))/√(n_1)1/m∑_r=1^m I{r∈ S_p^⊥}

and

    |I_1^n_1+j(k)|   ≲B/√(n_2m)∑_r=1^m w_r(k+1)-w_r(k)_2^2 
       ≲B/√(n_2m)∑_r=1^m η^2B^4/m L(k) 
       ≤η^2B^5 √(L(0))√(L(k))/√(n_2m).

Recall that

    A_pr={∃w: w-w_r_2 ≤ R, I{w^Tx_p≥ 0}≠ I{w_r^Tx_p≥ 0}}

and

    P(A_p,r) ≤R/√(2π),  S_p={r∈ [m]:I{A_p,r}=0}.

Thus, from Bernstein's inequality, we have that with probability at least 1-e^-mR,

    1/m∑_r=1^m I{r∈ S_p^⊥} =1/m∑_r=1^m I{A_pr}≲ 4R.

Then the inequality holds for all p∈[n_1] with probability at least 1-n_1e^-mR. Thus from (78), we have that for every p∈ [n_1]

    |I_1^p(k)|≲η^2B^5 √(L(0))√(L(k))/√(n_1m)+η B^4√(L(k))/√(n_1)R.

Combining (79) and (80), we have that

    I_1(k)_2   =√(∑_p=1^n_1 |I_1^p(k)|^2 +∑_j=1^n_2 |I_1^n_1+j(k)|^2  )
       ≲η^2B^5 √(L(0))√(L(k))/√(m)+η B^4√(L(k))R.

Plugging it into (69) yields that

    [ s(k+1); h(k+1) ]_2^2 
       ≤ (1-ηλ_0/2)^2 [ s(k); h(k) ]_2^2 + I_1(k) _2^2+2(1-ηλ_0/2) [ s(k); h(k) ]_2 I_1(k) _2 
       ≤ [(1-ηλ_0/2)^2 +  C^2(η^2B^5 √(L(0))/√(m)+η B^4R)^2 +2C(η^2B^5 √(L(0))/√(m)+η B^4R)][ s(k); h(k) ]_2^2
       ≤ (1-ηλ_0/2)[ s(k); h(k) ]_2^2,

where the last inequality requires that

    η^2B^5 √(L(0))/√(m)≲ηλ_0 ,  η B^4R ≲ηλ_0.

Recall that we also require that  CM^2R<λ_0/4 and

    R^'=CB^2√(L(0))/√(m)λ_0<R.
 
Finally, with R=𝒪(λ_0/B^4) and Lemma 11 for the upper bound of L(0), m needs to satisfies that 

    m=Ω( B^12L(0)/λ_0^4)=Ω(d^12/λ_0^4log^6 (md/δ)  log(n_1+n_2/δ) ).






§ PROOF OF SECTION 4



 §.§ Proof of Lemma 7



Recall that 

    ∂ s_p(w)/∂w_r   = a_r/√(n_1m) [σ^”(w_r^T x_p)w_r0x_p+σ^'(w_r^T x_p) [   1; 0_d ] -σ^”'(w_r^T x_p)w_r1_2^2 x_p 
          -2σ^”(w_r^T x_p)[    0; w_r1 ] ]

and

    ∂ h_j(w)/∂w_r=a_r/√(n_2m)σ^'(w_r^Ty_j)y_j.

(1) For p∈ [n_1], define the event

    A_p,r={∃w: w-w_r(0)_2 ≤ R, I{w^T x_p≥ 0}≠ I{w_r(0)^T x_p≥ 0}}.

Similar to (44), we can deduce that for any p∈ [n_1],

    P(A_p,r)≤2R/√(2π) .


From the form of ∂ s_p(w)/∂w_r, we can deduce that

    ∂ s_p(w)/∂w_r-∂ s_p(0)/∂w_r_2 
       ≲1/√(n_1m)[ R(w_r(0)_2+1) +|I{w_r^T x_p≥ 0}-I{w_r(0)^T x_p≥ 0}|(w_r(0)_2^2+1)] 
       ≤1/√(n_1m)[ R(w_r(0)_2+1) +I{A_p,r}(w_r(0)_2^2+1)],

where the second inequality follows from the fact w-w_r(0)_2<R≤ 1 and the definition of A_p,r.

Similarly, we have that

    ∂ h_j(w)/∂w_r-∂ h_j(0)/∂w_r_2≲1/√(n_2m)R(w_r(0)_2+1).

Combining (81) and (82), we can deduce that

    J(w)-J(0)_2^2 
       ≤J(w)-J(0)_F^2 
       =∑_i=1^n_1+n_2J_i(w)-J_i(0)_2^2
       = ∑_r=1^m ( ∑_p=1^n_1∂ s_p(w)/∂w_r - ∂ s_p(0)/∂w_r_2^2  + ∑_j=1^n_2∂ h_j(w)/∂w_r - ∂ h_j(0)/∂w_r_2^2) 
       ≲∑_r=1^m ( ∑_p=1^n_11/n_1m(R(w_r(0)_2+1)+I{A_p,r}(w_r(0)_2^2+1))^2 +∑_j=1^n_21/n_2m(Rw_r(0)_2+R)^2 )
       ≲R^2/m∑_r=1^m (w_r(0)_2^2+1) +1/n_1m∑_p=1^n_1∑_r=1^m I{A_p,r}(w_r(0)_2^4+1) 
       = R^2/m∑_r=1^m (w_r(0)_2^2+1)
          +1/n_1m∑_p=1^n_1∑_r=1^m I{A_p,r}(w_r(0)_2^4 I{w_r(0)_2^2≤ M} +w_r(0)_2^4 I{w_r(0)_2^2> M} +1)
       ≲R^2/m∑_r=1^m (w_r(0)_2^2+1)+ M^2/n_1m∑_p=1^n_1∑_r=1^m I{A_p,r}+1/m∑_r=1^m w_r(0)_2^4 I{w_r(0)_2^2> M},

where M=2(d+2)log(2m(d+2)/δ). Note that from (62), we have

    P(∃ r∈ [m], w_r(0)_2^2≥2 (d+2)log(2m(d+2)/δ))≤δ.

On the other hand, applying Bernstein's inequality yields that with probability at least 1-n_1e^-mR,

    1/m∑_r=1^m I{A_p,r} <4R

holds for all p∈ [n_1]. 

Therefore, we have that

    J(w)-J(0)_2^2 ≲ MR^2+R^2+M^2R ≲ M^2R

holds with probability at least 1-δ-n_1e^-mR.

(2) Note that when σ satisfies the assumption 5, σ^',σ^” and σ^”' are all Lipschitz continuous. Thus we can obtain that

    ∂ s_p(w)/∂w_r-∂ s_p(0)/∂w_r_2 ≲1/√(n_1m) R(w_r(0)_2^2+w_r(0)_2+1)≲1/√(n_1m) R(w_r(0)_2^2+1),

where the second inequality is from Young's inequality.

Similarly, we have

    ∂ h_j(w)/∂w_r-∂ h_j(0)/∂w_r_2 ≲1/√(n_2m) R(w_r(0)_2+1).


Combining (83) and (84) yields that

    J(w)-J(0)_2^2 
       ≤∑_r=1^m ( ∑_p=1^n_1∂ s_p(w)/∂w_r - ∂ s_p(0)/∂w_r_2^2  + ∑_j=1^n_2∂ h_j(w)/∂w_r - ∂ h_j(0)/∂w_r_2^2) 
       ≲∑_r=1^m ( ∑_p=1^n_11/n_1m(Rw_r(0)_2^2+R)^2 +∑_j=1^n_21/n_2m(Rw_r(0)_2+R)^2 )
       ≲R^2/m∑_r=1^m (w_r(0)_2^4+1) 
       ≲ R^2[d^2+ d^2/√(m)√(log(1/δ))+d^2/m(log(1/δ))^2 ],

where the last inequality follows from the fact that w_r(0)_2^4_ψ_1/2≲ d^2 and Lemma 8.




 §.§ Proof of Theorem 3

We prove Theorem 3 by induction. Our induction is the following condition for the hidden weights.


At the t-th iteration, we have w_r(t)_2 ≤ B and

    w_r(t)-w_r(0)_2 ≤CB^2√(L(0))/√(m)λ_0:=R^'
 
for all r∈ [m], where C is a universal constant and B=√(2(d+2)log(2m(d+2)/δ) )+1.


Instead of inducing on the convergence rate of the empirical loss function, as shown in Condition 2, we perform induction on the movements of the hidden weights. Because with Condition 3, we can directly derive the following convergence rate of the empirical loss function.


If Condition 3 holds for t=0,⋯, k and R^'≤ R and R^”≲√(λ_0), then  

    L(t)≤ (1-η)^t L(0),

holds for t=0,⋯, k, where R is the constant in Lemma 7 and 
R^”=CM√(R) in (36) when σ is the ReLU^3 activation function, R^”=CdR in (38) when σ satisfies Asssumption 5.


Thanks to Corollary 3, it is sufficient to prove that Condition 3 also holds for t=k+1. For readability, we defer the proof of Corollary 3 to the end of this section. In the following, we are going to show that the Condition 3 also holds for t=k+1, thus combining Condition 3 and Corollary 3 leads to Theorem 3.  

We let R^”=CM√(R) in (36) when σ is the ReLU^3 activation function and let R^”=CdR in (37) when σ satisfies Assumption 5.  	
	
First, set R^'≤ R and R^”≤√(λ_0)/3, then from Lemma 7 we have J(t)-J(0)_2≤√(λ_0)/3, thus λ_min(J(t))≥2√(λ_0)/3 and  λ_min(H(t)) ≥4λ_0/9 for t=0,⋯, k.

From the updating role of NGD, we have

    w_r(t+1)=w_r(t)-η[J(t)^T ]_r (H(t))^-1[ s(t); h(t) ],
 
where 

    [J(t)^T ]_r=(∂ s_1(t)/∂w_r,⋯, ∂ s_n_1(t)/∂w_r, ∂ h_1(t)/∂w_r,⋯, ∂ h_n_2(t)/∂w_r).
	
Therefore, for t=0,⋯,k and any r∈ [m], we have

    w_r(t+1)-w_r(t)_2 
       ≤η[J(t)^T ]_r_2 H(t)^-1_2  √(L(t))
       ≤9η/4λ_0[J(t)^T ]_r_2 √(L(t))
       ≤9η/4λ_0[J(t)^T ]_r_F √(L(t))
       =  9η/4λ_0√(∑_p=1^n_1∂ s_p(t)/∂w_r_2^2 + ∑_j=1^n_2∂ h_j(t)/∂w_r_2^2 )√(L(t))
       ≲4η/9λ_0√(B^4+1/m)√(L(t))
       ≲η  B^2/√(m)λ_0√(L(t))
       ≤η  B^2/√(m)λ_0 (1-η)^t/2√(L(0)),

where the last inequality is due to Corollary 3.

Summing t from 0 to k yields that

    w_r(k+1)-w_r(0)_2 
       ≤∑_t=0^kw_r(t+1)-w_r(t)_2
       ≤ Cη  B^2/√(m)λ_0∑_t=0^k(1-η)^t/2√(L(0))
       ≤ C B^2√(L(0))/√(m)λ_0.

Note that although C is a universal constant, it still depends on the activation function. 

Now, when R^'≤ 1, we can deduce that Condition 3 also holds for t=k+1. Thus, it remains only to derive the requirement for m.

Recall that we need m to satisfy that R^'=CB^2√(L(0))/√(m)λ_0≤ R and R^”≤√(λ_0)/3.

(1) When σ is the ReLU^3 activation function, R^”=CM√(R)≤√(λ_0)/3, wihch implies that R≲λ_0/M^2. Then R^'=CB^2√(L(0))/√(m)λ_0≤ R imples that

    m=Ω(M^4 B^4 L(0)/λ_0^4).

From Lemma 11, we can deduce that 

    m=Ω(d^12/λ_0^4log^6 (md/δ) log(n_1+n_2/δ)) .


(2) When σ satisfies Assumption 5, we have that 

    R≲√(λ_0)/d, R^'=CB^2√(L(0))/√(m)λ_0≤ R.

From Lemma 5 in <cit.>, we know

    L(0)≲ d^2log( n_1+n_2/δ).

Thus, we can deduce that

    m=Ω( d^6/λ_0^3log^2 (md/δ) log(n_1+n_2/δ) ).





Similar as before, when R^'≤ R and R^”≤√(λ_0)/3 , we have λ_min(J(t)) ≥2√(λ_0)/3 and then λ_min(H(t))≥4/9λ_0 for t=0,⋯, k.

Let u(t)=[ s(t); h(t) ], then

    u(t+1)-u(t)
       =u(w(t)-ηJ(t)^T H(t)^-1u(w(t))) -u(w(t)) 
       = -∫_0^1⟨∂u(w(s))/∂w , ηJ(t)^T H(t)^-1u(w(t))⟩ ds 
       = -∫_0^1⟨∂u(w(t))/∂w , ηJ(t)^T H(t)^-1u(w(t))⟩ ds 
         + ∫_0^1⟨∂u(w(t))/∂w-∂u(w(s))/∂w , ηJ(t)^T H(t)^-1u(w(t))⟩ ds 
       := I_1(t)+I_2(t),

where the second equality is from fundamental theorem of calculus and w(s)=sw(t+1)+(1-s)w(t)=w(t)-sηJ(t)^T H(t)^-1u(t).

Note that ∂u(w(t))/∂w=J(t), thus I_1(t)=ηu(t). Plugging it into (85) yields that

    u(t+1)=(1-η)u(t)+I_2(t).



Therefore, it remains only to bound I_2(t)_2.

    I_2(t)_2   = ∫_0^1⟨∂u(w(t))/∂w-∂u(w(s))/∂w , ηJ(t)^T H(t)^-1u(w(t))⟩ ds_2 
       ≤∫_0^1J(w(t))-J(w(s))_2 ηJ(t)^T H(t)^-1u(w(t)) _2ds 
       ≤ηJ(t)^T H(t)^-1_2 u(w(t)) _2∫_0^1J(w(t))-J(w(s))_2ds  
       ≤3η√(L(t))/2√(λ_0)∫_0^1J(w(t))-J(w(s))_2ds 
       ≤3η√(L(t))/2√(λ_0)∫_0^1 (J(w(t))-J(0)_2+J(w(s))-J(0)_2)ds 
       ≲η√(L(t))/√(λ_0) R^”

where the last inequality follows from the fact that

    w_r(s)-w_r(0)_2≤ sw_r(t+1)-w_r(0) _2+(1-s)w_r(t)-w_r(0) _2 ≤ R^'≤ R

and Lemma 7.  

Plugging (88) into the recursion formula (87) yields that

    u(t+1)_2^2    =(1-η)u(t)+I_2(t)_2^2 
       = (1-η)^2 u(t)_2^2 + I_2(t)_2^2 + 2⟨(1-η)u(t), I_2(t) ⟩
       ≤  (1-η)^2 u(t)_2^2 + I_2(t)_2^2 + 2 (1-η) u(t)_2 I_2(t)_2 
       ≤[ (1-η)^2+ C^2η^2(R^”)^2/λ_0+2 (1-η)Cη R^”/√(λ_0)] u(t)_2^2 
       ≤ (1-η)u(t)_2^2,

where the last inequality follows from the requirement

    2Cη R^”/√(λ_0)≤η/4,
 
i.e., R^”≲√(λ_0).


Therefore, we can deduce that u(t)_2^2 ≤ (1-η)^tu(0)_2^2 holds for t=0,⋯, k.






 §.§ Proof of Corollary 1


Setting η=1 in (85) yields that

    u(t+1)=I_2(t).
 
From (88), we have that

    I_2(t)_2   ≤3√(L(t))/2√(λ_0)∫_0^1J(w(t))-J(w(s))_2ds.

As w(s)=sw(t+1)+(1-s)w(t), then for any r∈ [m], we have w_r(s)_2 ≤ sw_r(t+1)_2+(1-s)w_r(t)_2 ≤ B. 

When σ is smooth, we can deduce that for any r∈ [m],

    ∂ s_p(w(s))/∂w_r-∂ s_p(w(t))/∂w_r_2 ≲1/√(n_1 m) (B^2+1) w_r(s)-w_r(t)_2 ≤1/√(n_1 m) (B^2+1) w_r(t+1)-w_r(t)_2

and

    ∂ h_j(w(s))/∂w_r-∂ h_j(w(t))/∂w_r_2 ≲1/√(n_1 m) (B+1) w_r(s)-w_r(t)_2 ≤1/√(n_1 m) (B+1) w_r(t+1)-w_r(t)_2.

From (85), we know that for any r∈ [m],

    w_r(t+1)-w_r(t)_2 ≲ B^2/√(m)λ_0√(L(t)) .

Thus for any s∈ [0,1], we have

    J(w(s))-J(w(t))_2^2 
       ≤∑_r=1^m ( ∑_p=1^n_1∂ s_p(w(s))/∂w_r-∂ s_p(w(t))/∂w_r_2^2+ ∂ h_j(w(s))/∂w_r-∂ h_j(w(t))/∂w_r_2^2) 
       ≲1/m∑_r=1^m ( (B^4+1)w_r(t+1)-w_r(t)_2^2+(B^2+1)w_r(t+1)-w_r(t)_2^2 ) 
       ≲ B^4  ( B^2/√(m)λ_0√(L(t)))^2.

Plugging it into (89), we have 

    I_2(t)_2   ≤3η√(L(t))/2√(λ_0)∫_0^1J(w(t))-J(w(s))_2ds
       ≲√(L(t))/√(λ_0)η B^4/√(m)λ_0√(L(t))
       =B^4/√(mλ_0^3)L(t).

Combining with the fact u(t+1)=I_2(t) yields that

    [ s(t+1); h(t+1) ]_2  ≤CB^4/√(mλ_0^3)[ s(t); h(t) ]_2^2

holds for t∈ℕ, where C is a universal constant.




§ AUXILIARY LEMMAS



If X_1,⋯, X_n are independent mean zero random variables with X_i_ψ_α<∞ for all 1≤ i≤ n and some α>0, then for any vector a=(a_1,⋯,a_n)∈ℝ^n, the following holds ture:

    P(|∑_i=1^n a_i X_i |≥ 2eC(α)b_2√(t)+2eL_n^*(α)t^1/αb_β(α)  )≤ 2e^-t,  for  all  t≥ 0,

where b=(a_1X_1_ψ_α,⋯, a_nX_n_ψ_α) ∈ℝ^n,

    C(α):=max{√(2),2^1/α}{√(8)(2π)^1/4e^1/24(e^2/e/α)^1/α    ,     if  α<1, 
    
    4e+2(log 2)^1/α    ,     if  α≥ 1.
    .

and for β(α)=∞ when α≥ 1 and β(α)=α/(α-1) when α>1,

    L_n(α):=4^1/α/√(2)b_2×{   b_β(α) ,     if  α<1, 
       4e b_β(α)/C(α),     if α≥ 1.
    .

and L^*_n(α)=L_n(α)C(α)b_2/b_β(α).



Let X_i, 1≤ i ≤ n be independent centered random variables a.s. bounded by c<∞ in absolute value. Set σ^2=1/n∑_i=1^n 𝔼X_i^2 and S_n=1/n ∑_i=1^n X_i. Then, for all t≥ 0,

    P(S_n≥√(2σ^2 t/n) +ct/3n)≤ e^-u.




For 0<δ <1, with probability at least 1-δ, we have

    y-u(0)_2^2 =𝒪(n log( n/δ)  ).




First, from Cauchy's inequality, we have

    y-u(0)_2^2   =∑_i=1^n (y_i-1/√(m)∑_r=1^m a_rσ(w_r(0)^T x_i) )^2 
       ≤∑_i=1^n 2y_i^2 +2 (1/√(m)∑_r=1^m a_rσ(w_r(0)^T x_i) )^2 
       = 2∑_i=1^n y_i^2+2 ∑_i=1^n (1/√(m)∑_r=1^m a_rσ(w_r(0)^T x_i) )^2.

Since w_r(0)^T x_i ∼𝒩(0,x_i_2) and x_i_2≤√(2), we can deduce that

    a_rσ(w_r(0)^T x_i) _ψ_2≤w_r(0)^T x_i _ψ_2 =𝒪(1)
 
holds for all r∈ [m] and i∈ [n].  

Thus for any fixed i∈ [n], applying Lemma 8 yields that with probability at least 1-2e^-t,

    |1/√(m)∑_r=1^m a_rσ(w_r(0)^T x_i)| ≤ C√(t),

where C is a universal constant. By taking union bound on all i∈ [n], the above inequality holds for all i∈ [n] with probability at least 1-2ne^-t. 

Let 2ne^-t=δ, i.e., t=log(2n/δ), then we have

    y-u(0)_2^2≲ n+nt ≲ nlog(n/δ).





For 0<δ <1, with probability at least 1-δ, we have that when m≥log^2 (n_1+n_2/δ),

    L(0)=[ s(0); h(0) ]_2^2 =𝒪(d^6 log(n_1+n_2/δ)).




Recall that for p∈ [n_1],

    s_p(0)=1/√(n_1)[1/√(m)∑_r=1^m a_r( σ^'(w_r(0)^T x_p)w_r0(0)-σ^”(w_r(0)^T x_p) w_r1(0)_2^2  )-f(x_p)]
	
and for j∈ [n_2],

    h_j(0)=1/√(n_2)[1/√(m)∑_r=1^m a_rσ(w_r(0)^T y_j) -g(y_j)].
	
Then

    L(0)   =∑_p=1^n_11/2(s_p(0))^2+∑_j=1^n_21/2(h_j(0))^2 
       ≤1/n_1∑_p=1^n_1(1/√(m)∑_r=1^m a_r( σ^'(w_r(0)^T x_p)w_r0(0)-σ^”(w_r(0)^T x_p) w_r1(0)_2^2  ) )+1/n_1∑_p=1^n_1f^2(x_p)
          +1/n_2∑_j=1^n_2( 1/√(m)∑_r=1^m a_rσ(w_r(0)^T y_j))^2+1/n_2∑_j=1^n_2g^2(y_j).
	
Note that 

    |a_r( σ^'(w_r(0)^T x_p)w_r0-σ^”(w_r(0)^T x_p) w_r1(0)_2^2  )|≲w_r(0)_2^3

and |a_rσ(w_r(0)^T y_j) |≲w_r(0)_2^3.

Since w_r(0)_2^3 _ψ_2/3≤ (w_r(0)_2^2 _ψ_1)^3/2≲ d^3, from Lemma 8, we have that for fixed i∈ [n_1] and j∈ [n_2] with probability at least 1-2e^-t,

    |1/√(m)∑_r=1^m a_r( σ^'(w_r(0)^T x_p)w_r0(0)-σ^”(w_r(0)^T x_p) w_r1(0)_2^2 )| ≲ d^3√(t)+d^3/√(m) t^3/2

and with probability at least 1-2e^-t,

    | 1/√(m)∑_r=1^m a_rσ(w_r(0)^T y_j)|≲ d^3√(t)+d^3/√(m) t^3/2.


Then taking a union bound for all i∈ [n_1] and j∈ [n_2] with 2(n_1+n_2)e^-t=δ yields that 

    L(0)   ≲ (d^3√(t)+d^3/√(m) t^3/2)^2 
       ≲ d^6t +d^6 t^3/m
       = d^6 ( log(n_1+n_2/δ) +1/mlog^3 (n_1+n_2/δ) )
       ≲ d^6 log(n_1+n_2/δ),
 
as m≥log^2 (n_1+n_2/δ).









